[WIP] Threaded SLES audio with time stretching

[fzurita]

I've been trying implement a "Dual Async Audio" plugin option that does time stretching when a device is unable to run a game quickly enough. This remove crackling audio when a game slows down for whatever reason. Instead of letting the audio buffers run dry, it instead "time stretches" audio until the game catches up again. This is at least the intention.

I'm using the SoundTouch library to achieve this, and from my limited testing so far, it seems to work pretty well.

There are still a couple of WIP:

• For, example, since audio is now running in a separate thread, choosing to sync the game to the audio causes the game to run too fast. Instead, one has to rely fully on the core frame limiter. I'm think of adding a sleep to simulate the amount of time it should take to playback the sample if sync game to audio is chosen. [DONE]
• Also, the core frame limiter seems to be pretty inaccurate, probably because they are only limited to sleeping in 1ms boundaries due to the SDL library. I'm getting around this by speeding up or slowing down the core depending on the amount of audio buffers left in my queue.[Not needed now that sync game to audio works, although, I have ideas on how to improve the core situation]

@littleguy77 @Gillou68310 @paulscode Now here comes a question. Should I make this into a separate audio plugin or should the SLES audio plugin support the original operation and the dual sync audio operation depending on what the user selects?

I'm personally leaning towards making it into a different audio plugin due to the amount of change that was needed to make this work. It would be hard to reintegrate this back into the original SLES audio plugin. The other option would be replace the SLES audio plugin with this plugin and drop original functionality. And if the user wants, they can drop back to the SDL audio plugin.

[Gillou68310]

My main motivation for writing the sles plugin was to reduce the audio latency compared to the SDL plugin. If your new implementation still have less latency than SDL I think we can remove the original sles plugin, otherwise I think we should keep it.

I'll try to take a look at your new plugin this weekend.

[fzurita]

Thanks. I thought that it was originally Paul who wrote it for some reason, so I included him.

This does add some audio latency so that we can slow down the audio before buffers runs dry. It's not that much though. At least I don't hear unless something bad happens and we end up putting too much in there (which I have had issues with but fixed).

[Gillou68310]

Thanks. I thought that it was originally Paul who wrote it for some reason, so I included him

That's because he's the audio expert here ;-)

This does add some audio latency so that we can slow down the audio before buffers runs dry.

Ok I'll test it this weekend and report back.

[Gillou68310]

BTW Richard is ok for the configurable VI refresh rate option but it has to be a per-game option
mupen64plus/mupen64plus-core#139

[Gillou68310]

Don't hesitate to send him a pull request if you're interested.

[fzurita]

Cool, yeah, I'll just have to find the time.

[Gillou68310]

Sure ;-)

[Gillou68310]

Just a few remarks:

queueCallback is running on another thread (at least that's what the opensles doc says)
that's why I locked the mutex before accessing lock.value.

100 secondary buffers? It sounds enormous to me!

Why are you building as c++?

[fzurita]

I believe it's safe to access lock.value from multiple threads because of the volatile keyword use. Writing and reading to that is an atomic operation. I also only check if the value is zero, I don't care if when I access the variable is one or two off. If we really reached 100 secondary buffers, something went wrong.

100 buffers assuming 512 samples a buffer is only 204,800 bytes, nothing now a days :)

I'm using C++ because the SoundTouch library is a C++ library. You can't instantiate C++ objects from C files.

[fzurita]

When you try this out, for best experience, make sure sync audio and video are enabled.

This will make the game timing be driven off the audio plugin instead of the core. The core frame limiter is very inaccurate.

[Gillou68310]

I believe it's safe to access lock.value from multiple threads because of the volatile keyword use. Writing and reading to that is an atomic operation. I also only check if the value is zero, I don't care if when I access the variable is one or two off. If we really reached 100 secondary buffers, something went wrong.

The volatile keyword only notify the compiler to disable optimization on a variable, mutex must be locked in order to perform atomic operation. However considering the current design I agree that atomic operation isn't really necessary.

100 buffers assuming 512 samples a buffer is only 204,800 bytes, nothing now a days :)

Well it depends which memory is being used when enqueuing buffers ;-)
I cannot find anything in the opensles doc concerning enqueue limits.
Anyway I guess we should see a "buffer overrun" warning if anything goes wrong, I just hope it's not device specific!

I'm using C++ because the SoundTouch library is a C++ library. You can't instantiate C++ objects from C files.

Arf, is it possible to build SoundTouch as a shared library?

[fzurita]

Yes, I built the SoundTouch librart as a shared library. Maybe I could write a wrapper library to the SoundTouch library that exposes methods that can be accessed using osal_dynlib_getproc. Is it worth the trouble though to keep it as C instead of C++?

[Gillou68310]

Don't waste your time writing a wrapper, building as C++ is just fine. I'm just not confortable with those static_cast<>

[fzurita]

Oh, lol, since I was building C++ now, I took advantage of some C++ features like bool and static_cast

[fzurita]

Ok, I fixed the speed limiter in the core. Now it's no longer drifting from audio at 100% speed. I also made an upstream pull request.

Desync audio and video should now work correctly. I also notice that frame rate is now a lot smoother when running with the speed limiter.

[fzurita]

Number buffers and buffer size settings now actually do something. To start, I would do about 20 buffers and 512 buffer size

[fzurita]

20 buffers and 512 buffer size seems to result in a latency in audio being too high for my taste. From testing, it seems that 15 buffers at 256 buffer size seems more ideal. I'll probably change the default to that.

The bigger the number of buffers, the bigger the chance the audio stretching algorithm has to react to the slowed game speed. If the game is constantly running slow, then low buffer sizes seem fine. It's the quick changes in game speed that make the low buffer sizes trip up.

[fzurita]

I cleaned up the commit history a good amount, it should be much easier to follow.