v2.0.0-beta3

pluginval Change List

2.0.0

• Updated JUCE to 9.0.3
• Fixed the crash handler so a backtrace is shown again when a plugin crashes, and made it async-signal-safe [#144]
• Added --junit-xml <file> and --json-results <file> to both pluginval validate and pluginval test to write machine-readable reports for CI (also settable as JUNIT_XML / JSON_RESULTS). A report that can't be written fails the run. See docs/Adding pluginval to CI.md
• Added acceptance testing: pluginval test <config.json> renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting 0 on a match and 1 on a mismatch (with a diff WAV). Use --record-missing to record new references. See docs/Acceptance testing.md
• Added --plugin <plugin> to pluginval test to run a config against a given plugin instead of the one it names (the config's plugin can then be omitted)
• Added two GitHub Actions for use from other repositories: Tracktion/pluginval/validate (validate a list of plugins) and Tracktion/pluginval/acceptance (run every acceptance config in a directory against a built plugin). See .github/actions/README.md
• Restructured the command line into subcommands: pluginval validate [options] <plugin> (the default), pluginval run-tests and pluginval strictness-help [level]. The plugin path is now a positional argument of validate
• Deprecated: the flat flags --validate <plugin>, --run-tests and --strictness-help [level] still work as aliases (with a one-line notice) but will be removed in a future version. pluginval <plugin> remains a silent shorthand for validate
• Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline
• Added --config <file.json> to load settings from JSON (repeatable; later files win per key)
• Settings precedence is now (lowest to highest): defaults, environment variables, --config, command-line options
• Environment variable support is now automatic for every option (including DISABLED_TESTS)
• Breaking: unrecognised options now produce an error instead of being ignored
• Breaking: invalid option values now error instead of silently falling back (e.g. --rtcheck banana, --strictness-level abc)
• Breaking: falsey flag environment variables now mean off (e.g. SKIP_GUI_TESTS=0 no longer enables skipping)
• --help output is now generated by CLI11
• Added a "Discrete parameter state restoration" test (strictness 8) which checks a discrete parameter has the same value after restoring a state, whatever value it held beforehand. This fails for parameters that keep the raw normalised value from the host instead of mapping it to a step (e.g. juce::AudioParameterBool)
• Fixed false failures in the "Plugin state restoration" test where the first parameter was compared against the last value sent by a previous test rather than the plugin's own value
• Below strictness level 8, two-step (on/off) parameters are now only sent 0 or 1 when tests randomise parameter values. Arbitrary values are still sent to them at level 8 and above, and to all other parameters at every level
• Fixed the "Plugin state restoration" test's exact binary state comparison never running. It was documented as running at strictness level 8 and above but checked the test's own level instead of the run's
• Fixed the macOS app failing to start on any machine other than the one that built it ("Library not loaded: @rpath/librtcheck.dylib"); this affected the v2.0.0-beta1 and v2.0.0-beta2 releases
• Enabled the editor stress and extreme (real-time allocation / oversized block) tests that were present in the source tree but had not been compiled into the build

1.0.5

• Added drag-and-drop of plug-in files onto the main window, with two drop zones to either validate the plug-in or add it to the plugin list [#170]
• Added static linking to the Windows runtime so it should run on more Windows systems (particularly non-dev machines)
• Made PluginInfoTest run on the message thread as the functions it calls aren't thread-safe

1.0.4

• Limit auval's stress test to 20 seconds (vs 600) [#135]
• Fixed incorrect "ALL TESTS PASSED" message when validating out of process [#125]
• Updated juce to 8.0.3 [#133]
• Added LV2 support [#25]
• Provide CMake module for registering pluginval tests with CTest [#123]
• Changed minimum Linux version to Ubuntu 22.04

1.0.3

• Fix VST3 host bus issue

1.0.2

• Added com.apple.security.get-task-allow to allow debugging on pluginval [macOS]

1.0.1

• Fixed support for Ubuntu 18.04

1.0.0

• Added running auval as a test for Audio Units on macOS (levels 5 and up)
• Added running VST3's validator as a test for VST3s (levels 5 and up)
• Added better conformance to the VST3 standard when running at levels 5 and below
• Refactored internals for better stability

0.2.9

• Fixed support for macOS from 10.11 and Windows without the runtime DLLs

0.2.8

• Added a test to process audio with different sample rate and block sizes but not call releaseResources in between

0.2.6

• Avoided a deadlock when opening plugin windows for the second time on Linux

0.2.5

• Fixed Linux architecture for pre-built binaries

0.2.4

• Added tests for plugin programs
• Added a new CLI option to pass a file with a list of tests to disable
• Added notarisation to macOS builds for Catalina compatibility

0.2.3

• Fixed some failing tests on startup

0.2.2

• Added a test for disabled HiDPI awareness on Windows
• Updated JUCE to v5.4.3

0.2.1

• Added an EditorAutomationTest to adjust parameter values whilst showing the plugin editor

0.2.0

• Removed built-in support for VST2 testing, this can be built with by setting the VST2_SDK_DIR environment variable before running the tests/ scripts
• Added the option to repeat tests a specified number of times
• Added the option to randomise the test order each repeat
• Added some message loop running to editor tests to ensure the UI is actually displayed
• Added a test for processing audio whilst showing the editor
• Added an option to automatically save log files with a sensible name (see the "--output-dir" option)
• Disabled the LargerThanPreparedBlockSizeTest for AU, VST & VST3 formats
• Added a "Test File" button to the UI for validating plugins that won't scan (this improves the stack trace information in the log file)

0.1.5

• Fixed a problem catching allocations in clang release builds
• Avoided accidently setting the bypass parameter for VST3 plugins
• Simplified some parameter iteration methods to avoid returning bypass or non-automatable parameters
• Added MIDI note on and off messages for synth processing tests
• Added a Parameter thread safety test which attempts to call setValue on parameters from multiple threads concurrently mimicking automation and GUI interaction
• Added a PluginStateTestRestoration test which calculates some checksums before randomising and restoring plugin state to check state restoration is correct
• Added the ability to set command line arguments as environment variables e.g. "--skip-gui-tests" > "SKIP_GUI_TESTS=1", "--timeout-ms 30000" > "TIMEOUT_MS=30000"
• Added the ability to set a random seed to use for the tests

0.1.4

• Added stack backtraces to crashed validation output
• Added tests for detecting memory allocations and deallocations in the audio thread
• Added a test for creating an editor with an uninitialised plugin and a 0 sample rate and block size prepared plugin
• Added a test calling processBlock with more samples than initialised with

0.1.3

• Reduced the amount of default logging
• Added the "--verbose" command line flag
• Added some basic bus tests

0.1.2

• Added optional logging from the slave process
• Added an option to validate in the same process in GUI mode
• Added the "--validate-in-process" command line flag
• Replaced the "strictnessLevel" command line option with the more canonical "strictness-level" form
• Added a timeout option which can be set from the GUI "Options" menu or the command line "--timeoutMs [numMs]" flag