Latency issues

[yanostulens]

Dear @mackron ,

For a project I am using the miniaudio library for file playback and decoding. The goal is to make an application for an embedded Linux system with less powerful hardware (ARM single core, 500 MHz CLK). When testing on the less powerful hardware I noticed there is some latency when trying to playback larger files (in this case a 160MB mp3 file). So I switched back to a simple example you once provided on this forum and added a timer to check the latency. I used the code below:

#include <stdio.h>
#include <iostream>
#include <memory>
#include <chrono>

#define MINIAUDIO_IMPLEMENTATION
#include "miniaudio.h"


void data_callback(ma_device* pDevice, void* pOutput, const void* pInput, ma_uint32 frameCount)
{
    ma_data_source_read_pcm_frames((ma_data_source*)pDevice->pUserData, pOutput, frameCount, NULL);

    (void)pInput;
}

using namespace std::chrono;

int main(int argc, char** argv)
{
    //Timer init
    high_resolution_clock::time_point t1 = high_resolution_clock::now();

    ma_result result;
    ma_device_config deviceConfig;
    ma_device device;
    ma_resource_manager_config resourceManagerConfig;
    ma_resource_manager resourceManager;
    ma_resource_manager_data_source dataSource;

    /* We'll initialize the device first. */
    deviceConfig = ma_device_config_init(ma_device_type_playback);
    deviceConfig.dataCallback = data_callback;
    deviceConfig.pUserData    = &dataSource;    /* <-- We'll be reading from this in the data callback. */

    result = ma_device_init(NULL, &deviceConfig, &device);
    if (result != MA_SUCCESS) {
        printf("Failed to initialize device.");
        return -1;
    }

    /*
    We have the device so now we want to initialize the resource manager. We'll use the resource manager to load a
    sound based on the command line.
    */
    resourceManagerConfig = ma_resource_manager_config_init();
    resourceManagerConfig.decodedFormat     = device.playback.format;
    resourceManagerConfig.decodedChannels   = device.playback.channels;
    resourceManagerConfig.decodedSampleRate = device.sampleRate;

    result = ma_resource_manager_init(&resourceManagerConfig, &resourceManager);
    if (result != MA_SUCCESS) {
        ma_device_uninit(&device);
        printf("Failed to initialize the resource manager.");
        return -1;
    }

    /* Now that we have a resource manager we can load a sound. */
    result = ma_resource_manager_data_source_init(&resourceManager, "/home/yano/files/mix.mp3", MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_STREAM | MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_DECODE, NULL, &dataSource);
    if (result != MA_SUCCESS) {
        printf("Failed to load sound");
        return -1;
    }

    /* Now that we have a sound we can start the device. */
    result = ma_device_start(&device);
    if (result != MA_SUCCESS) {
        ma_device_uninit(&device);
        printf("Failed to start device.");
        return -1;
    }

    high_resolution_clock::time_point t2 = high_resolution_clock::now();
    duration<double> time_span = duration_cast<duration<double>>(t2 - t1);

    std::cout << "Time spend: " << time_span.count() << " seconds" << std::endl;
    printf("Press Enter to quit...\n");
    getchar();

    /* Teardown. */

    /* Uninitialize the device first to ensure the data callback is stopped and doesn't try to access any data. */
    ma_device_uninit(&device);

    /*
    Before uninitializing the resource manager we need to uninitialize every data source. The data source is owned by
    the caller which means you're responsible for uninitializing it.
    */
    ma_resource_manager_data_source_uninit(&dataSource);

    /* Uninitialize the resource manager after each data source. */
    ma_resource_manager_uninit(&resourceManager);

    return 0;
}

I ran this test program twice using the 160MB mp3, once on my desktop running Ubuntu and once on the ARM processor running a light weight debian distro and got the following results (note: the binaries have different names because they need different compilers, but they are both compiled from the code above):

So on my PC it took 0.26 seconds and on the ARM processor it took over 6 seconds which I believe is quite a number. Am I doing something wrong or do you have some advice in order to decrease these numbers?

Thank you in advance!
Kind regards

[mackron]

Both of those seem quite slow to me. My suggestion is to move your timers around to try and isolate the specific function call that's taking so long. Currently your timer encompasses device initialization and startup in addition to loading the resource. If you can narrow down your timer a bit it might give us some better intelligence as to what might be causing the slow down.

For a stream, the size of the file shouldn't make any difference to load times. I'm also suspecting that it could be device initialization that's taking so long.

[yanostulens]

Dear @mackron ,

Thank you for your answer. I narrowed the latency down to the ma_resource_manager_data_source_init() function. Also to elaborate on the filesizes and how they influence latency is redid two tests both using the following code (just the file that is loaded is different in size):

#include <stdio.h>
#include <iostream>
#include <memory>
#include <chrono>
#include <filesystem>

#define MINIAUDIO_IMPLEMENTATION
#include "miniaudio.h"


void data_callback(ma_device* pDevice, void* pOutput, const void* pInput, ma_uint32 frameCount)
{
    ma_data_source_read_pcm_frames((ma_data_source*)pDevice->pUserData, pOutput, frameCount, NULL);

    (void)pInput;
}

using namespace std::chrono;

int main(int argc, char** argv)
{
    ma_result result;
    ma_device_config deviceConfig;
    ma_device device;
    ma_resource_manager_config resourceManagerConfig;
    ma_resource_manager resourceManager;
    ma_resource_manager_data_source dataSource;

    /* We'll initialize the device first. */
    deviceConfig = ma_device_config_init(ma_device_type_playback);
    deviceConfig.dataCallback = data_callback;
    deviceConfig.pUserData    = &dataSource;    /* <-- We'll be reading from this in the data callback. */

    result = ma_device_init(NULL, &deviceConfig, &device);
    if (result != MA_SUCCESS) {
        printf("Failed to initialize device.");
        return -1;
    }

    /*
    We have the device so now we want to initialize the resource manager. We'll use the resource manager to load a
    sound based on the command line.
    */
    resourceManagerConfig = ma_resource_manager_config_init();
    resourceManagerConfig.decodedFormat     = device.playback.format;
    resourceManagerConfig.decodedChannels   = device.playback.channels;
    resourceManagerConfig.decodedSampleRate = device.sampleRate;

    result = ma_resource_manager_init(&resourceManagerConfig, &resourceManager);
    if (result != MA_SUCCESS) {
        ma_device_uninit(&device);
        printf("Failed to initialize the resource manager.");
        return -1;
    }

    std::cout << "Filesize is: " << std::filesystem::file_size("/home/yano/Downloads/files/california.mp3") << " bytes" << std::endl;

     //Timer init
    high_resolution_clock::time_point t1 = high_resolution_clock::now();

    /* Now that we have a resource manager we can load a sound. */
    result = ma_resource_manager_data_source_init(&resourceManager, "/home/yano/Downloads/files/california.mp3", MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_STREAM | MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_DECODE, NULL, &dataSource);
    if (result != MA_SUCCESS) {
        printf("Failed to load sound");
        return -1;
    }

    high_resolution_clock::time_point t2 = high_resolution_clock::now();

    /* Now that we have a sound we can start the device. */
    result = ma_device_start(&device);
    if (result != MA_SUCCESS) {
        ma_device_uninit(&device);
        printf("Failed to start device.");
        return -1;
    }

    duration<double> time_span = duration_cast<duration<double>>(t2 - t1);

    std::cout << "Time spend: " << time_span.count() << " seconds" << std::endl;
    printf("Press Enter to quit...\n");
    getchar();

    /* Teardown. */

    /* Uninitialize the device first to ensure the data callback is stopped and doesn't try to access any data. */
    ma_device_uninit(&device);

    /*
    Before uninitializing the resource manager we need to uninitialize every data source. The data source is owned by
    the caller which means you're responsible for uninitializing it.
    */
    ma_resource_manager_data_source_uninit(&dataSource);

    /* Uninitialize the resource manager after each data source. */
    ma_resource_manager_uninit(&resourceManager);

    return 0;
}

The first test uses a 160MB file resulting in a latency of 0.26 seconds, in the second test a file of only 5MB is used resulting in a latency of 0.04 seconds:

[mackron]

I have an idea on what might be happening, but not 100% sure. Are you able to do an experiment? In miniaudio.h, there's this function:

MA_API ma_result ma_mp3_get_length_in_pcm_frames(ma_mp3* pMP3, ma_uint64* pLength)
{
    if (pLength == NULL) {
        return MA_INVALID_ARGS;
    }

    *pLength = 0;   /* Safety. */

    if (pMP3 == NULL) {
        return MA_INVALID_ARGS;
    }

    #if !defined(MA_NO_MP3)
    {
        *pLength = drmp3_get_pcm_frame_count(&pMP3->dr);

        return MA_SUCCESS;
    }
    #else
    {
        /* mp3 is disabled. Should never hit this since initialization would have failed. */
        MA_ASSERT(MA_FALSE);
        return MA_NOT_IMPLEMENTED;
    }
    #endif
}

Are you able to comment out this line: *pLength = drmp3_get_pcm_frame_count(&pMP3->dr);? MP3 can take a bit of time to calculate the length of the MP3 file and I'm wondering if that's causing the problem. If so, I might be able to optimize streams to not call that function (though, off the top of my head I can't think of where I'd be calling that function for streams).

[yanostulens]

@mackron Yes!!!

Your answer above seems to be fixing the problem. Here are the results, both with the 160MB mp3 file again, once with the line commented out and once not:

Thank you so so much! :-)

[mackron]

OK, I'll review the code for stream initialization and avoid calling that function. I don't think it should be necessary off the top of my head. Thanks for checking that for me.

[mackron]

I just had a look now and it is indeed necessary. It's used by ma_resource_manager_data_stream_get_length_in_pcm_frames() and also for some cursor management.

I'm thinking that you may need to use use the MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC flag.

[yanostulens]

@mackron

I just did some more experiments and it is hard to prove but adding the MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC flag is not really solving the latency issue. Adding the MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC flag just makes the ma_resource_manager_data_source_init() call non-blocking, the audio however still starts playing with a lot of latency (around 6 seconds on my low power ARM processor).

I would love to do some more tests with the MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_ASYNC flag in order to prove this to you. Do you now in which part of the miniaudio lib or in which function the first actual pcm frames are sent to the soundcard? This way I can let the timer run from the start of the ma_resource_manager_data_source_init() function until the first samples are played.

[mackron]

What you're explaining sounds exactly what I would expect. There's no real way around it so long as the length is getting retrieved at initialization time. In the dev branch I've added a new flag called MA_RESOURCE_MANAGER_DATA_SOURCE_FLAG_UNKNOWN_LENGTH. When this is set, the length will not be retrieved, but it comes at the expense of ma_resource_manager_data_source_get_length_in_pcm_frames() always returning 0. This should make the performance consistent with that experimental change I suggested earlier.

[yanostulens]

Dear @mackron,

Just a suggestion but isn't there a way to run the ma_resource_manager_data_source_get_length_in_pcm_frames() function in another thread and let it return the length asynchronously?

[mackron]

No, that makes the implementation too complicated because you'll have one thread reading from the decoder while the other is scanning over it to calculate the length. It won't be as simple as wrapping it in a mutex because the internal state of the decoder will be in conflicting states. Then you have the issue of ma_resource_manager_data_source_get_length_in_pcm_frames() being called before the length calculation process is still in progress. It's just too messy all round.