Issue running in headless mode vs gui

[cjappl]

Having a hard time running the validator in headless mode. Every plugin I try fails with the following output:

bash-3.2$ ./pluginval.app/Contents/MacOS/pluginval --validate /Library/Audio/Plug-Ins/Components/FabFilter\ Pro-Q\ 2.component/
Random seed: 0x74393be
-----------------------------------------------------------------
Starting test: CommandLineTests / Merge environment variables...
All tests completed successfully
-----------------------------------------------------------------
Starting test: CommandLineTests / Command line defaults...
All tests completed successfully
-----------------------------------------------------------------
Starting test: CommandLineTests / Command line parser...
All tests completed successfully
-----------------------------------------------------------------
Starting test: CommandLineTests / Command line random...
All tests completed successfully
-----------------------------------------------------------------
Starting test: AllocatorInterceptorTests / Ensure separate allocators are used for threads...
All tests completed successfully
-----------------------------------------------------------------
Starting test: AllocatorInterceptorTests / Ensure all allocations pass...
All tests completed successfully
-----------------------------------------------------------------
Starting test: AllocatorInterceptorTests / Ensure all allocations fail on MessageThread...
All tests completed successfully
-----------------------------------------------------------------
Starting test: AllocatorInterceptorTests / Ensure allocation violation status is reset...
All tests completed successfully
-----------------------------------------------------------------
Starting test: AllocatorInterceptorTests / Ensure all allocations pass on MessageThread, fail on background thread...
All tests completed successfully
-----------------------------------------------------------------
Starting test: AllocatorInterceptorTests / Ensure array allocations are caught...
!!! Test 3 failed
!!! Test 4 failed: Expected value greater than: 0, Actual value: 0

FAILED!!  2 tests failed, out of a total of 4

-----------------------------------------------------------------
Starting test: AllocatorInterceptorTests / Ensure allocations are thrown...
All tests completed successfully
pluginval v0.2.3 - JUCE v5.4.3
Started validating: /Library/Audio/Plug-Ins/Components/FabFilter Pro-Q 2.component/
Random seed: 0x751ad42
Validation started: 22 Jul 2019 9:48:26am

Strictness level: 5
-----------------------------------------------------------------
Starting test: pluginval / Scan for known types: /Library/Audio/Plug-Ins/Components/FabFilter Pro-Q 2.component/...
Num types found: 0
!!! Test 1 failed: No types found
FAILED!!  1 test failed, out of a total of 1

Finished validating: /Library/Audio/Plug-Ins/Components/FabFilter Pro-Q 2.component/
*** FAILED: 1 TESTS
*** FAILED: 1 TESTS`

This is with the invocation:

./pluginval.app/Contents/MacOS/pluginval --validate /Library/Audio/Plug-Ins/Components/FabFilter\ Pro-Q\ 2.component/

This plugin validates perfectly using the gui, output below:

pluginval v0.2.3 - JUCE v5.4.3
Started validating: AudioUnit-FF Pro-Q 2-95550150-61453d50
Random seed: 0x4305123
Validation started: 22 Jul 2019 9:49:46am

Strictness level: 5

Testing plugin: AudioUnit-FF Pro-Q 2-95550150-61453d50
FabFilter: FF Pro-Q 2 v2.1.2
-----------------------------------------------------------------
Starting test: pluginval / Open plugin (cold)...

Time taken to open plugin (cold): 180 ms
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Open plugin (warm)...

Time taken to open plugin (warm): 11 ms
Running tests 1 times
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Plugin info...

Plugin name: FF Pro-Q 2
Alternative names: FF Pro-Q 2
SupportsDoublePrecision: no
Reported latency: 0
Reported taillength: 0.18576

Time taken to run test: 0
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Editor...

Time taken to open editor (cold): 308 ms
Time taken to open editor (warm): 289 ms

Time taken to run test: 599 ms
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Open editor whilst processing...

Time taken to run test: 294 ms
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Audio processing...
Testing with sample rate [44100] and block size [64]
Testing with sample rate [44100] and block size [128]
Testing with sample rate [44100] and block size [256]
Testing with sample rate [44100] and block size [512]
Testing with sample rate [44100] and block size [1024]
Testing with sample rate [48000] and block size [64]
Testing with sample rate [48000] and block size [128]
Testing with sample rate [48000] and block size [256]
Testing with sample rate [48000] and block size [512]
Testing with sample rate [48000] and block size [1024]
Testing with sample rate [96000] and block size [64]
Testing with sample rate [96000] and block size [128]
Testing with sample rate [96000] and block size [256]
Testing with sample rate [96000] and block size [512]
Testing with sample rate [96000] and block size [1024]

Time taken to run test: 2 ms
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Plugin state...

Time taken to run test: 0
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Automation...
Testing with sample rate [44100] and block size [64] and sub-block size [32]
Testing with sample rate [44100] and block size [128] and sub-block size [32]
Testing with sample rate [44100] and block size [256] and sub-block size [32]
Testing with sample rate [44100] and block size [512] and sub-block size [32]
Testing with sample rate [44100] and block size [1024] and sub-block size [32]
Testing with sample rate [48000] and block size [64] and sub-block size [32]
Testing with sample rate [48000] and block size [128] and sub-block size [32]
Testing with sample rate [48000] and block size [256] and sub-block size [32]
Testing with sample rate [48000] and block size [512] and sub-block size [32]
Testing with sample rate [48000] and block size [1024] and sub-block size [32]
Testing with sample rate [96000] and block size [64] and sub-block size [32]
Testing with sample rate [96000] and block size [128] and sub-block size [32]
Testing with sample rate [96000] and block size [256] and sub-block size [32]
Testing with sample rate [96000] and block size [512] and sub-block size [32]
Testing with sample rate [96000] and block size [1024] and sub-block size [32]

Time taken to run test: 30 ms
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Editor Automation...

Time taken to run test: 1 sec
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Automatable Parameters...

Time taken to run test: 4 ms
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Basic bus...
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Listing available buses...
Inputs:
	Named layouts: Mono, Stereo, Mono, Stereo
	Discrete layouts: Discrete #1, Discrete #2, Discrete #1, Discrete #2
Outputs:
	Named layouts: Mono, Stereo
	Discrete layouts: Discrete #1, Discrete #2
Main bus num input channels: 2
Main bus num output channels: 2
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Enabling all buses...
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Disabling non-main busses...
!!! WARNING: Disabling non-main buses failed
All tests completed successfully
-----------------------------------------------------------------
Starting test: pluginval / Restoring default layout...
Main bus num input channels: 2
Main bus num output channels: 2

Time taken to run test: 264 ms

Time taken to run all tests: 3 secs
All tests completed successfully

Finished validating: AudioUnit-FF Pro-Q 2-95550150-61453d50
ALL TESTS PASSED

[drowaudio]

Does it only fail with AU types?
If you try VST3 or VST2 does it work ok?

[cjappl]

Just double checked. VST3 fails in the exact same way, but passes in the exact same way using the gui.

I don't have VST2 support right now, but I can check that if you think it's worthwhile to!

[drowaudio]

No, leaving VST2 is fine. I just wanted to check it wasn't a problem with the macOS Component manager lookup.
I'll try and have a look at this tomorrow but if you need an answer quicker, building pluginval in Xcode and setting the "Arguments passed on launch" setting to --validate /Library/Audio/Plug-Ins/Components/FabFilter\ Pro-Q\ 2.component/ and then stepping in to the Scan for known types test will probably find the culprit.

I'm wondering if it's some new security measure in macOS and the binary will need to be notarised now...

[cjappl]

Ok, I'll try that and report back. For the record this is OSX Mojave 10.14.5

[cjappl]

It seems like it's gotta be something in scanAndAddDragAndDroppedFiles.

It seems like it's never finding a match for the type of plugin that is there. I looked further into the juce code in a function getComponentDescFromIdentifier and it looks like it wants the file passed in to be prefixed with "AudioUnit:"

I'll keep digging if I have time but I'd be interested to hear what you think if you take a look

[cjappl]

So basically line 97 where you call knownPluginList.scanAndAddDragAndDroppedFiles in PluginTests.cpp. This is what fails and says there is no proper type found for the plugin.

[drowaudio]

That was where I was fearing the problem was.
The problem is that I don't think there's anything I can do to change that. The JUCE code looks in to the plist of the file provided and then tries to load that via the Apple component manager. If that's not returning the plugin for the description I don't think there's any way to fix that.

What's odd is that it works in GUI come though. I'm struggling to figure out why that might be...

[cjappl]

In my brief exploration, it seemed as if the gui code never went into that block at all. Maybe there is another execution path that the GUI takes for this purpose?

Let me know if you find anything. In my down time I'll try to continue investigating! We would love to use this for our CI, as it seems awesome.

Would you confirm when you get a chance that you can repro? I just want to make sure that I'm not the only crazy one seeing this

[cjappl]

Ok, good news. First issue solved, although you all may want to solve it in the code to prevent something like this in the future.

The command:
/Applications/pluginval.app/Contents/MacOS/pluginval --validate /Library/Audio/Plug-Ins/VST3/FabFilter\ Pro-Q\ 2.vst3

Will run as expected

The command:
/Applications/pluginval.app/Contents/MacOS/pluginval --validate /Library/Audio/Plug-Ins/VST3/FabFilter\ Pro-Q\ 2.vst3/

will fail (notice the trailing slash. This is something that my shell auto corrects to so it's a fairly easy mistake to make. I'm going to write up a separate issue for that, and dig in more to the AU. You all can decide what to do with that new issue, if it's worth fixing or not!

I think just rstripping any slashes would be something reasonable to do?

[cjappl]

That issue is here #20

[cjappl]

Annnnnd it's the issue with the AU as well. I'm going to write a PR for that. Closing this issue for now.

Note to any future archeologists, this issue exists too #7, so if you use tmux that may throw a wrench in your debugging of this issue.

Thanks for the support. See you over on #20