Duplex mode not working with AAudio and engines

[mbpictures]

Hi!

I'm using miniaudio on Android together with the AAudio backend and engines. I used the advanced_engine example as a base and modified it just a little:

ma_result result;
    ma_resource_manager_config resourceManagerConfig;

    resourceManagerConfig = ma_resource_manager_config_init();
    resourceManagerConfig.decodedFormat     = ma_format_f32;    /* ma_format_f32 should almost always be used as that's what the engine (and most everything else) uses for mixing. */
    resourceManagerConfig.decodedChannels   = 0;                /* Setting the channel count to 0 will cause sounds to use their native channel count. */
    resourceManagerConfig.decodedSampleRate = 48000;            /* Using a consistent sample rate is useful for avoiding expensive resampling in the audio thread. This will result in resampling being performed by the loading thread(s). */

    result = ma_resource_manager_init(&resourceManagerConfig, &resourceManager);
    if (result != MA_SUCCESS) {
        logMsg("Failed to initialize resource manager.");
        return;
    }

    /* We're going to want a context so we can enumerate our playback devices. */
    result = ma_context_init(NULL, 0, NULL, &context);
    if (result != MA_SUCCESS) {
        logMsg("Failed to initialize context.");
        return;
    }

    ma_device_config deviceConfig;
    ma_engine_config engineConfig;

    deviceConfig = ma_device_config_init(ma_device_type_playback);
    deviceConfig.playback.pDeviceID = NULL;
    deviceConfig.playback.format    = resourceManager.config.decodedFormat;
    deviceConfig.playback.channels  = 0;
    deviceConfig.capture.format = resourceManager.config.decodedFormat;
    deviceConfig.capture.channels = 2;
    deviceConfig.capture.pDeviceID  = NULL;
    deviceConfig.sampleRate         = resourceManager.config.decodedSampleRate;
    deviceConfig.dataCallback       = NativeMultiTrackPlayerTurboModule::playbackCallback;
    deviceConfig.pUserData          = this;

    result = ma_device_init(&context, &deviceConfig, &device);
    if (result != MA_SUCCESS) {
        logMsg("Failed to initialize device. %d", 1);
        return;
    }

    engineConfig = ma_engine_config_init();
    engineConfig.pDevice          = &device;
    engineConfig.pResourceManager = &resourceManager;
    engineConfig.noAutoStart      = MA_TRUE;    /* Don't start the engine by default - we'll do that manually below. */

    result = ma_engine_init(&engineConfig, &engine);
    if (result != MA_SUCCESS) {
        logMsg("Failed to initialize engine.");
        ma_device_uninit(&device);
        return;
    }

    result = ma_engine_start(&engine);

But I always see this log: Failed to initialize device. -1.
Is duplex mode together with engines generally not allowed or does AAudio/the Device (using an android emulator) not support duplex mode? Thanks already in advance for your help!

[mackron]

Are you able to define MA_DEBUG_OUTPUT and post the output here? That'll print some logs to stdout.

If you pass in NULL for the context in ma_device_init(), does that make it work? Your device config looks fine, but you're using ma_device_type_playback instead of ma_device_type_duplex which I'm assuming is just you testing stuff.

[mbpictures]

Oh wow, I just got it working by setting the sampleRate to 0! But I have another question: I try to sync playback of some audio tracks and recording (like in a basic DAW). I tried it with two devices (one playback, one capture), but the playback was delayed by around 400-500ms (so the capturing started that time before the playback started). How can I properly sync up the playback and the capturing? I now tried the duplex device and set a flag once all tracks have been started (using ma_sound_start()). In the data_callback of the capture device, I check for that flag to become true and start writing the input buffer to a WAV file. Is that the way to do it or do you have an idea for a better implementation? Thank you already very much!

[mackron]

It slightly concerning that specifying a sample rate would result in an error. miniaudio should just convert it under the hood. That might be worthy of an investigation at some point.

The best thing for lowering the delay is to reduce your latency as much as possible. On Android the most realistic way to do this would probably be with the AAudio backend, so confirming that AAudio is indeed being used would be the first thing to check. Another thing to do is set the performance profile to low-latency in the device config: deviceConfig.performanceProfile = ma_performance_profile_low_latency;.

When getting back in sync, make sure you don't have a huge backlog of audio in your cache. miniaudio's duplex device is actually very basic - it just routes directly from capture to playback without any real syncing, and I it doesn't use a huge cache either. If it's delayed by ~400ms with ma_device_type_duplex then it might actually be something on the Android side because I would have thought miniaudio's duplex system would be glitching at that point. Though it's hard to know for sure off the top of my head.

The MA_DEBUG_OUTPUT thing would be helpful in determining whether or not you're using AAudio, and it'll also show how big your internal buffers are which is an indication of your latency.

[mbpictures]

Yes, I can confirm that the AAudio is used. I saw that the low_latency profile is the default for the performanceProfile value, but I set it deviceConfig.performanceProfile = ma_performance_profile_low_latency; with no difference.

[mackron]

I tried the sample rate thing on my side and I wasn't able to replicate that issue.

Concerning the latency stuff, here's a couple more ideas:

• deviceConfig.aaudio.usage = ma_aaudio_usage_game;
• deviceConfig.aaudio.allowSetBufferCapacity = MA_TRUE; - This has been known to cause some compatibility issues with some devices so be careful with this.

If you do deviceConfig.aaudio.allowSetBufferCapacity = MA_TRUE; then you could also try setting the period count to 1 with this: deviceConfig.periods = 1.

If you are able to get the latency reduced I'd be interested to hear what works for you.

[mbpictures]

Thanks a lot for your suggestions! After some fiddling with the duplex type, I first tested using this config:

ma_device_config deviceConfig = ma_device_config_init(ma_device_type_duplex);
deviceConfig.capture.format   = ma_format_f32;
deviceConfig.capture.channels = 2;
deviceConfig.playback.format    = ma_format_f32;
deviceConfig.playback.channels  = 2;
deviceConfig.sampleRate         = 0;
deviceConfig.dataCallback       = playbackCallback;
deviceConfig.pUserData          = this;
deviceConfig.notificationCallback = notificationCallback;
deviceConfig.performanceProfile = ma_performance_profile_low_latency;
deviceConfig.aaudio.usage = ma_aaudio_usage_game;

But it had around 200ms-300ms latency, when directly feeding the input into the output with a callback like:

memcpy(pOutput, pInput, frameCount * sizeof(float) * channels);

When setting deviceConfig.periods = 1;, there is no real noticeable difference. But when combining with deviceConfig.aaudio.allowSetBufferCapacity = MA_TRUE;, the latency drops to something like~50ms? So it's noticeable, but waaaay better. Despite still not being close to making something like singing with loop-through in ears possible. I also used wired USB headphones for my testing. When playing back music and recording at the same time with the duplex device I also discovered that there is some sound crackling, so I increased the periods to 2, which adds a little bit of delay but reduces sound crackling.

So to wrap this up: Is there anything more I can do to further reduce latency? I checked the google docs and they mention 45ms for the low_latency and even down to 20ms for a "pro" configuration. However, is there something like a "pro" config available for AAudio/miniaudio? Thanks in advance for your time! You also mentioned some compatibility issues around devices. Is there a safe method to set the periods to the lowest value possible? If the latency can't be reduced further, is there a mechanism you can think of to measure the latency? So we can delay the recording and therefor sync the recording to the playback?

[mackron]

I don't claim to be an expert with Android, but reading the documentation it sounds like the pro audio stuff is just something that is reported by the hardware and not something you explicitly enable via an API. So I don't think there's anything miniaudio can do to "enable" a pro audio configuration. If I'm mistaken on that I'm happy to listen to some advice on how miniaudio can make use of it.

[mbpictures]

Thanks a lot for your help! Regarding the latency issue, I solved it by measuring the latency using a loopback test (e.g., playing back some noise and measuring the time until this sound is recorded) and delaying the recorded audio by this time. Works like a charm. Thanks again!