macOS: working around AVAudioEngine device selection limitations

[wtholliday]

AudioKit is built on AVAudioEngine, which is limited to a single audio device for both input and output. The system (or possibly AVAudioEngine, I'm not sure) creates a hidden Aggregate Audio device for the default system input and output, allowing the user to select whatever devices they want in System Preferences while still being compatible with AVAudioEngine.

On macOS, the current implementation of the AudioKit singleton (AKManager on the V5 branch) changes the input device by changing the device selected in System Preferences. If a user expects to change the input device within an app, they may be surprised to see their default device changed.

Further confusing things, setting the output device via AudioKit will change AVAudioEngine's device for both input and output. This will fail in many cases since AVAudioEngine only works with devices that have both input and output channels.

I believe the best possible behavior is for AudioKit to publish single list of devices for both input and output. If the user doesn't see the device they want, they can always create an aggregate device in Audio MIDI Setup. Ideally, I'd like AudioKit to support separate input and output devices, but AVAudioEngine doesn't support it.

It seems fairly clear that AVAudioEngine wasn't designed for us to reach around it down to CoreAudio and change the audio devices. Consider putting in a feature request for device selection in AVAudioEngine if you would like better support.

I'm happy to start working on this change, probably within the V5 develop branch, but I'd like to know if anyone has thoughts on this first, or if I've got any of this wrong.

[wtholliday]

I just checked that the following will switch the audio device selected in System Preferences:

try! AKManager.setInputDevice(AKManager.inputDevices![0])

and if I try to set the audio output device when my graph requires audio input:

try! AKManager.setOutputDevice(AKManager.outputDevices[2]) // built-in speakers on my machine
let mic = AKMicrophone()
AKManager.output = mic
try! AKManager.start()
mic?.start()

I get:

2020-07-01 14:56:51.937240-0700 AudioKitDevices[18428:2594751] [avae]            AVAEInternal.h:76    required condition is false: [AVAudioIONodeImpl.mm:1092:SetOutputFormat: (IsFormatSampleRateAndChannelCountValid(hwFormat))]
2020-07-01 14:56:51.937474-0700 AudioKitDevices[18428:2594751] required condition is false: IsFormatSampleRateAndChannelCountValid(hwFormat)

[0xmtn]

Hi @wtholliday,

In my case, interestingly enough, neither AudioKit singleton from the latest release nor AKManager from v5-develop branch changes the system preferred device. Here is the code (as mentioned in #2129):

    NSArray *outputDevices = [EZAudioDevice outputDevices];
    NSArray *AKOutputDevices = [AudioKit outputDevices];
    NSArray *AKInputDevices = [AudioKit inputDevices];
    NSError *setOutputDeviceError = nil;
    [AudioKit setOutputDevice:[AKOutputDevices objectAtIndex:1]; error: &setOutputDeviceError];
    NSError *setInputDeviceError = nil;
    [AudioKit setInputDevice:[AKInputDevices objectAtIndex:1] error:&setInputDeviceError];

Also let me ask you this: Shoud I still use AKMicrophone after changing the input device via setInputDevice? I mean, in the chain below, should I do / use something else (like AKInput) in order to get the input from my preferred device and push it to my mixer? What do you think?

    NSError *setInputDeviceError = nil;
    [AudioKit setInputDevice:[AKInputDevices objectAtIndex:1] error:&setInputDeviceError]; // this device is my external sound interface with 2 input channels (mic input and instrument input)
   
    AVAudioFormat *audioFormat = [AudioKit.engine.inputNode inputFormatForBus:0];
    AKSettings.sampleRate = audioFormat.sampleRate;
    mic = [[AKMicrophone alloc] initWith:audioFormat];
    
    frequencyTracker = [[AKFrequencyTracker alloc] init:mic hopSize:512.0 peakCount:20.0];
    booster = [[AKBooster alloc] init:frequencyTracker gain:1];
    
    chainMixer = [[AKMixer alloc] init: @[booster]];
    
    myDSPNode = [[AKCustomDSPNode alloc] init:chainMixer];
    [myDSPNode setAudioUnitEnableWithIsEnable:true];
    
    mainMixer = [[AKMixer alloc] init: @[myDSPNode]];
    mainMixer.volume = 0.5;

    AudioKit.output = mainMixer;
    [AudioKit startAndReturnError:nil];

Is the default or preferred method of getting an input into AudioKit is to use AKMicrophone? Or is it just for getting input from a microphone device (either headphones or built-in microphone)?

[0xmtn]

As for the required condition is false: IsFormatSampleRateAndChannelCountValid(hwFormat) error:

I got rid of this error via setting AKSettings.sampleRate to inputNode's sampleRate, like below:

    AVAudioFormat *audioFormat = [AudioKit.engine.inputNode inputFormatForBus:0];
    AKSettings.sampleRate = audioFormat.sampleRate;
    mic = [[AKMicrophone alloc] initWith:audioFormat];

I don't know exactly and I cannot reprodruce this error in your case but you might wanna try setting AKSettings.sampleRate to outputNode's sampleRate.

   AVAudioFormat *audioFormat = [AudioKit.engine.outputNode outputFormatForBus:0];   
   AKSettings.sampleRate = audioFormat.sampleRate; 

I have very little experience on audio I/O programming, so what I've just told you may sound silly. Forgive me if it does 🙏

[wtholliday]

@0xmtn If I set my output device to my built-in speakers, the sample rate on the output node is still 44.1kHz (the sample rate of the input node is 0, since the device has no input channels). It still fails with the same error.

This question reinforces my findings:

https://stackoverflow.com/questions/61827898/how-to-set-avaudioengine-input-and-output-devices-swift-macos

in particular:

"So I filed a support request with apple on this and another issue and the response confirms that an AVAudioEngine can only be assigned to a single Aggregate device (that is, a device with both input and output channels) - the system default units create effectively an aggregate device internally which is why they work, although I've found an additional issue in that if the input device also has output capabilities (and you activate the inputNode) then that device has to be both the input and output device as otherwise the output appears not to work."

I think the only thing wrong in the above quote is that any device with both inputs and output will do, not just aggregate devices. Most audio interfaces have both inputs and outputs, so it makes sense to include them in the list of devices that the app sees.

[wtholliday]

@0xmtn

Shoud I still use AKMicrophone after changing the input device via setInputDevice? I mean, in the chain below, should I do / use something else (like AKInput) in order to get the input from my preferred device and push it to my mixer? What do you think?

AFAIK AKMicrophone is fine for input. (@aure correct me if I'm wrong)

Is the default or preferred method of getting an input into AudioKit is to use AKMicrophone? Or is it just for getting input from a microphone device (either headphones or built-in microphone)?

AKMicrophone just connects AVAudioEngine's input node to a mixer node. There's nothing special going on there.

[0xmtn]

AFAIK AKMicrophone is fine for input. (@aure correct me if I'm wrong)

Is the default or preferred method of getting an input into AudioKit is to use AKMicrophone? Or is it just for getting input from a microphone device (either headphones or built-in microphone)?

AKMicrophone just connects AVAudioEngine's input node to a mixer node. There's nothing special going on there.

Sorry for the late reply. This actually clears the fog for me a bit. Thanks. And I'll try to execute on the paragraph below based on my understanding. I'll update here with the results asap.

"So I filed a support request with apple on this and another issue and the response confirms that an AVAudioEngine can only be assigned to a single Aggregate device (that is, a device with both input and output channels) - the system default units create effectively an aggregate device internally which is why they work, although I've found an additional issue in that if the input device also has output capabilities (and you activate the inputNode) then that device has to be both the input and output device as otherwise the output appears not to work."

I think the only thing wrong in the above quote is that any device with both inputs and output will do, not just aggregate devices. Most audio interfaces have both inputs and outputs, so it makes sense to include them in the list of devices that the app sees.