2.0.0: Rewrite CLI parsing onto CLI11 + a JSON settings pipeline

CHANGELIST.md

@@ -1,5 +1,16 @@
 # pluginval Change List
 
+### 2.0.0
+- Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline
+- Added `--config <file.json>` to load settings from JSON (repeatable; later files win per key)
+- Settings precedence is now (lowest to highest): defaults, environment variables, `--config`, command-line options
+- Environment variable support is now automatic for every option (including `DISABLED_TESTS`)
+- **Breaking:** unrecognised options now produce an error instead of being ignored
+- **Breaking:** invalid option values now error instead of silently falling back (e.g. `--rtcheck banana`, `--strictness-level abc`)
+- **Breaking:** falsey flag environment variables now mean off (e.g. `SKIP_GUI_TESTS=0` no longer enables skipping)
+- `--help` output is now generated by CLI11
+- Enabled the editor stress and extreme (real-time allocation / oversized block) tests that were present in the source tree but had not been compiled into the build
+
 ### 1.0.5
 - Added drag-and-drop of plug-in files onto the main window, with two drop zones to either validate the plug-in or add it to the plugin list [#170]
 - Added static linking to the Windows runtime so it should run on more Windows systems (particularly non-dev machines)

CLAUDE.md

@@ -4,7 +4,7 @@
 
 **pluginval** is a cross-platform audio plugin validator and tester application developed by Tracktion Corporation. It tests VST, VST3, AU (Audio Unit), LV2, and LADSPA plugins for compatibility and stability with host applications.
 
-- **Version**: 1.0.4 (see `VERSION` file)
+- **Version**: 1.0.4 (see `VERSION` file; a 2.0.0 entry is staged in `CHANGELIST.md` but `VERSION` is not yet bumped)
 - **License**: GPLv3
 - **Framework**: Built on JUCE (v8.0.x)
 - **Language**: C++20
@@ -80,7 +80,10 @@ pluginval/
 │   ├── MainComponent.cpp/h   # GUI main window component
 │   ├── Validator.cpp/h       # Core validation orchestration
 │   ├── PluginTests.cpp/h     # Test framework and base classes
-│   ├── CommandLine.cpp/h     # CLI argument parsing
+│   ├── CommandLine.cpp/h     # Thin CLI adapter (delegates to SettingsParser)
+│   ├── PluginvalSettings.h   # Unified settings struct + JSON mapping + toPluginTestOptions()
+│   ├── SettingsParser.cpp/h  # CLI/env/config -> merged JSON -> settings; child-process handoff
+│   ├── SettingsSerializer.cpp/h # JSON load/save + value coercions (comma lists, hex seed)
 │   ├── CrashHandler.cpp/h    # Crash reporting utilities
 │   ├── TestUtilities.cpp/h   # Helper functions for tests
 │   ├── RTCheck.h             # Real-time safety checking macros
@@ -188,6 +191,41 @@ VST2_SDK_DIR=/path/to/vst2sdk cmake -B Builds/Debug .
    - Auto-registers via static instance pattern
    - Defines requirements (thread, GUI needs)
 
+### CLI Settings Pipeline
+
+Command-line parsing centres on one plain settings struct (`PluginvalSettings`)
+that CLI11 binds to directly. A single instance is filled by successive layers,
+**lowest to highest precedence: defaults → environment → `--config` → CLI**
+(in `SettingsParser::parseTokens`):
+
+1. **preprocess** the raw command line — rewrite the deprecated `strictnessLevel`,
+   strip the macOS `-NSDocumentRevisionsDebugMode YES` flag, and insert an
+   implicit `--validate` when the last argument is a bare plugin path.
+2. **Environment layer.** Env-var names are *derived* from the registered
+   options (`--strictness-level` → `STRICTNESS_LEVEL`), so there is no separate
+   env table. A synthetic `--name=value` argv is built from the environment and
+   parsed by CLI11, reusing all its coercion.
+3. **`--config` layer.** Repeatable; each JSON file is `merge_patch`-ed in
+   command-line order (later files win per key). Beats the environment.
+4. **CLI layer.** The real arguments are parsed last and beat everything;
+   CLI11 only overwrites a member when its option was actually provided.
+
+`configureApp()` registers every option (bound to the struct) and is used for
+both the env pass and the CLI pass. Comma lists use `->delimiter(',')`, the enum
+uses a `CheckedTransformer`, and the hex/int seed is a small callback.
+`PluginvalSettings::toPluginTestOptions()` converts to the JUCE-flavoured
+`PluginTests::Options` at the boundary.
+
+Adding a new option is three edits: a struct member, an entry in the nlohmann
+macro list, and one `add_option(...)` line — its environment variable then works
+automatically. `SettingsSerializer` handles JSON load/save plus the two
+remaining conversions (hex seed, disabled-tests file).
+
+The child validation process receives a fully-resolved, **authoritative**
+settings set via a base64-encoded JSON argument (`--config-base64`), avoiding
+per-flag re-serialisation and command-line quoting hazards. `--help`/`--version`
+are handled by CLI11 (auto usage + a footer with the env-var/commands notes).
+
 ### Test Framework
 
 Tests are self-registering. To find all tests, look for static instances:
@@ -327,9 +365,9 @@ Basic usage:
 
 Key options:
 - `--validate [path]` - Validate plugin at path
+- `--config [file.json]` - Load a full settings set from JSON (overridden by env vars and CLI options)
 - `--strictness-level [1-10]` - Test thoroughness (default: 5)
 - `--skip-gui-tests` - Skip GUI tests (for headless CI)
-- `--validate-in-process` - Don't use child process (for debugging)
 - `--timeout-ms [ms]` - Test timeout (default: 30000, -1 for none)
 - `--verbose` - Enable verbose logging
 - `--output-dir [dir]` - Directory for log files
@@ -374,6 +412,8 @@ add_pluginval_tests(MyPluginTarget
 ### External
 - **JUCE** (v8.0.x) - Audio application framework (git submodule)
 - **magic_enum** (v0.9.7) - Enum reflection (fetched via CPM)
+- **CLI11** (v2.6.2) - CLI argument parsing, header-only (fetched via CPM)
+- **nlohmann/json** (3.12.0) - JSON settings layering/serialisation (fetched via CPM)
 - **rtcheck** (optional, macOS) - Real-time safety checking (fetched via CPM)
 - **VST3 SDK** (v3.7.x) - Steinberg VST3 SDK for embedded validator (fetched via CPM, optional)
 
@@ -439,6 +479,6 @@ Run internal tests via CLI:
 
 - Always test changes on multiple platforms when possible
 - VST3 plugins have specific threading requirements - use the `*OnMessageThreadIfVST3` helpers
-- Child process validation is the default and recommended for production use
-- In-process validation (`--validate-in-process`) is useful for debugging but a crashing plugin will crash pluginval
+- The GUI runs each validation in a separate child process for crash isolation (the default)
+- The CLI `--validate` path runs in-process; a crashing plugin will terminate pluginval, and the signal handler reports it as a failure rather than a pass
 - Real-time safety checking is only available on macOS currently (uses rtcheck library)

CMakeLists.txt

@@ -61,6 +61,8 @@ option(JUCE_ENABLE_MODULE_SOURCE_GROUPS "Enable Module Source Groups" ON)
 
 include(cmake/CPM.cmake)
 CPMAddPackage("gh:Neargye/magic_enum#v0.9.7")
+CPMAddPackage("gh:CLIUtils/CLI11@2.6.2")
+CPMAddPackage("gh:nlohmann/json@3.12.0")
 
 if(PLUGINVAL_ENABLE_RTCHECK)
     CPMAddPackage("gh:Tracktion/rtcheck#main")
@@ -120,16 +122,23 @@ set(SourceFiles
     Source/CrashHandler.h
     Source/MainComponent.h
     Source/PluginTests.h
+    Source/PluginvalSettings.h
+    Source/SettingsParser.h
+    Source/SettingsSerializer.h
     Source/TestUtilities.h
     Source/Validator.h
     Source/CommandLine.cpp
     Source/CrashHandler.cpp
     Source/Main.cpp
     Source/MainComponent.cpp
     Source/PluginTests.cpp
+    Source/SettingsParser.cpp
+    Source/SettingsSerializer.cpp
     Source/tests/BasicTests.cpp
     Source/tests/LocaleTest.cpp
     Source/tests/BusTests.cpp
+    Source/tests/EditorTests.cpp
+    Source/tests/ExtremeTests.cpp
     Source/tests/ParameterFuzzTests.cpp
     Source/TestUtilities.cpp
     Source/Validator.cpp)
@@ -176,7 +185,9 @@ target_link_libraries(pluginval PRIVATE
     juce::juce_audio_processors
     juce::juce_audio_utils
     juce::juce_recommended_warning_flags
-    magic_enum)
+    magic_enum
+    CLI11::CLI11
+    nlohmann_json::nlohmann_json)
 
 if (${CMAKE_SYSTEM_NAME} MATCHES "Linux")
     target_link_libraries(pluginval PRIVATE

SUBCOMMANDS_HANDOFF.md

@@ -0,0 +1,89 @@
+# HANDOFF: Restructure pluginval's CLI into subcommands
+
+## Goal
+Turn the current "mode" flags into proper subcommands, each with its own
+argument set, while keeping the old flat flags working as **deprecated aliases
+for one release**:
+
+| New (target) | Old (keep as deprecated alias) |
+|---|---|
+| `pluginval validate [options] <plugin>` (the **default**) | `pluginval --validate <plugin>` / `pluginval <plugin>` |
+| `pluginval run-tests` | `pluginval --run-tests` |
+| `pluginval strictness-help [level]` | `pluginval --strictness-help [level]` |
+| `pluginval --version` / `pluginval --help` | (unchanged) |
+
+All settings options (`--strictness-level`, `--config`, `--rtcheck`, …),
+environment variables, and the JSON precedence pipeline belong to the `validate`
+subcommand and are otherwise unchanged.
+
+This was deliberately deferred from the CLI11 refactor PR (#174). The pipeline
+and `PluginvalSettings` were designed to be reused unchanged.
+
+## Read these first (don't trust this summary — verify against the files)
+- `Source/SettingsParser.cpp/.h` — the parser. Key pieces to reuse:
+  - `preprocess()` — deprecation rewrite, macOS flag strip, implicit `--validate`, tokenise.
+  - `parseTokens(tokens, env)` — env → `--config` → CLI layering into one `PluginvalSettings`. **This is the `validate` body.**
+  - `configureApp(app, s)` — registers every option on a `CLI::App`, used for both the env and CLI passes.
+  - `createChildProcessCommandLine()` — the parent→child base64 handoff (`--config-base64 <b64> --validate <path>`).
+  - `isCommandLine(tokens)` — what makes `shouldPerformCommandLine` return true.
+- `Source/CommandLine.cpp` — `performCommandLine()` is the dispatcher today:
+  token-scans for `--run-tests` / `--strictness-help`, otherwise runs `parseTokens` for validate; `--help`/`--version` go through CLI11. Also `runUnitTests()`, `printStrictnessHelp()`.
+- `Source/Main.cpp` — calls `shouldPerformCommandLine()` then `performCommandLine()` with `getCommandLineParameters()` (a single `juce::String`).
+- `Source/CommandLineTests.cpp` — the test contract.
+
+## Recommended approach: a thin `argv[1]` verb dispatcher (not CLI11 subcommands)
+The existing `parseTokens` pipeline (env/config/CLI layering via two `CLI::App`
+passes) is the hard part and already works. Rather than re-express it inside
+CLI11's native `add_subcommand` machinery (which complicates the env/config
+layering because the options live on the subcommand), peel the verb off the
+front and route:
+
+1. In a new `dispatch()` step (in `SettingsParser` or `CommandLine.cpp`):
+   - Tokenise the command line (reuse `preprocess` minus the implicit-validate step, or add a pre-step).
+   - Look at the first non-option token:
+     - `validate` → strip it, run the existing validate pipeline on the rest.
+     - `run-tests` → `runUnitTests()`.
+     - `strictness-help` → `printStrictnessHelp(level)`.
+     - otherwise → **default to validate** (this preserves `pluginval <plugin>` and `pluginval --strictness-level 5 <plugin>`).
+2. Each verb keeps its own small set of expected args. `validate` reuses
+   `configureApp`/`parseTokens` verbatim. `run-tests` takes none.
+   `strictness-help` takes an optional level.
+
+This keeps `parseTokens` and the precedence layering untouched — the subcommand
+work is purely a routing layer in front of it.
+
+(If you prefer CLI11-native subcommands instead: put `configureApp` options on a
+`validate` subcommand, and make `buildEnvArgv`/the env pass target that
+subcommand's options. Doable, but more invasive for no functional gain here.)
+
+## Deprecated-alias behaviour (one release)
+Keep the old flat forms working, but print a one-line notice to stderr pointing
+at the new syntax, e.g.:
+- `pluginval --validate x` → run validate; warn `"--validate is deprecated; use 'pluginval validate x'"`.
+- `pluginval --run-tests` → warn `"use 'pluginval run-tests'"`.
+- `pluginval --strictness-help` → warn `"use 'pluginval strictness-help'"`.
+- `pluginval <plugin>` (bare path) → **no warning** (still the documented shorthand for `validate`).
+
+Gate the warnings behind detection of the old flag so the new subcommand form is
+silent. Remove the aliases in the release after next; note it in `CHANGELIST.md`.
+
+## Files to touch
+- `Source/SettingsParser.{h,cpp}` — add the verb dispatch + a `Command` result (validate/run-tests/strictness-help/help/version), or expose a `dispatch()` that returns which verb + the remaining tokens. Keep `parseTokens` as the validate body.
+- `Source/CommandLine.cpp` — `performCommandLine()` routes on the verb; emit the deprecation notices for old flat flags. `shouldPerformCommandLine()` must also recognise the bare verbs (`validate`/`run-tests`/`strictness-help`) in addition to the old flags.
+- `Source/CommandLineTests.cpp` — add tests: each subcommand; default-to-validate; bare-path shorthand; every deprecated alias still works (and warns); `run-tests`/`strictness-help` arg handling.
+- `docs/Command line options.md` — regenerate (`pluginval --help`); CLI11 can show per-subcommand help if you go native, otherwise hand-format the verb list.
+- `CHANGELIST.md` — note the subcommand syntax + the deprecation.
+- `CLAUDE.md` — update the "CLI Settings Pipeline" section to mention the verb layer.
+
+## Gotchas / decisions to make
+- **Child-process handoff.** `createChildProcessCommandLine()` emits `--config-base64 <b64> --validate <path>`. Decide whether the child invocation becomes `validate --config-base64 …` or stays flat. Simplest: keep it flat and have the dispatcher treat a leading `--config-base64`/`--validate` as the (deprecated, unwarned-for-internal) validate path. Note `--config-base64` is now hardened to reject being combined with non-`--validate` options — keep that working under whichever form you choose.
+- **`shouldPerformCommandLine`** is what flips pluginval into CLI (vs GUI) mode in `Main.cpp`. It must return true for `pluginval run-tests` etc., not just the old flags.
+- **`--help` scope.** With the dispatcher, `pluginval --help` is the top-level help (list verbs + the validate options). Consider `pluginval validate --help` for the full option list. CLI11-native subcommands give this for free.
+- **Implicit validate** currently lives in `preprocess`. With an explicit `validate` verb, make sure `pluginval <plugin>` (no verb) still resolves to validate, and `pluginval validate <plugin>` doesn't double-insert `--validate`.
+- **Reconcile** a positional plugin path under `validate` (e.g. `pluginval validate <plugin>`) with the existing `--validate <plugin>` option — pick one canonical form (recommend the positional for the new syntax, mapping it onto `s.validatePath`).
+
+## Verify
+- `pluginval run-tests` passes the full unit suite (it must, it's how CI runs tests).
+- `pluginval validate --strictness-level 10 <plugin>` and `pluginval <plugin>` both validate.
+- Every deprecated alias produces identical behaviour to before (plus a notice).
+- CI matrix green (Linux/macOS/Windows build + dependency). Remember `.github/workflows/build.yaml` uses `--run-tests` and `--strictness-level 10 --validate …`; update those to the new syntax **and** keep an alias test, or the deprecation will fire in CI.