Supporting WASAPI's new AudioClient3 (Windows 10) mode: Real Time Low-Latency Audio

[Arcitec]

I was looking at the WASAPI driver:

https://github.com/WeAreROLI/JUCE/blob/master/modules/juce_audio_devices/native/juce_win32_WASAPI.cpp

It uses AudioClient1 (IAudioClient), the original WASAPI format. It allows low latency in exclusive mode, but is stuck at a minimum of 10ms latency in shared mode.

As of Windows 10, Microsoft introduced IAudioClient3 for "real time audio" which allows as little as 2.67ms of latency at 48 kHz even in SHARED DEVICE MODE. That's incredible. Having low latency in "shared mode" (so that multiple applications can output audio at the same time) is the holy grail!

IAudioClient3 is a subclass of IAudioClient2 is a subclass of IAudioClient. So in theory, all you will have to do is use IAudioClient3 with your existing code to take advantage of it, possibly with some new flags, as well as some OS checks to only use IAudioClient3 if the user is on Windows 10.

I've collected some links:

1. Reading more about AudioClient3 and what is needed for low latency audio: http://blog.nirbheek.in/2018/03/low-latency-audio-on-windows-with.html
2. Microsoft Documentation: https://msdn.microsoft.com/library/windows/desktop/dn911487
3. Stackoverflow Discussion: https://stackoverflow.com/questions/37468283/how-to-get-below-10ms-latency-using-wasapi-shared-mode
4. References for how slow AudioClient1 latency is: https://forum.juce.com/t/wasapi-excluive-mode/6213 and https://forum.juce.com/t/windows-buffer-size-wont-go-below-132/24806 and https://forum.juce.com/t/winrt-audio/12869

NOTE:

To get ultra low latency, the audio driver MUST support the "WaveRT" protocol. Here are the descriptions of all audio driver protocols, taken from the Virtual Audio Cable (VAC) manual (which supports all 3). I have copied chunks of text from all over the VAC manual, so this is a bit jumbled, but generally provides a great overview of the differences between the audio driver protocols:

WaveCyclic, WavePci and WaveRT. In general, WaveCyclic is most simple and reliable, WaveRT is newest and most efficient, and WavePci has less latency than WaveCyclic.

WaveCyclic and WavePci both implement "old", "standard" or "legacy" KS streaming protocol between KS driver (filter) and KS client, while WaveRT implements "RT Audio", "looped" or "realtime" streaming protocol.

In Vista and later, WaveRT port/miniport are fully supported by PortCls and RT Audio protocol is fully supported by System Audio Engine.

WaveCyclic - intended for legacy audio adapters with a single circular hardware buffer common for all clients. It is the simplest (and usually most stable) interface but also the slowest one.
WavePci - intended for adapters with multiple bus mastering buffers individual for each client and internal hardware mixing support. Can provide less latency than WaveCyclic but port/miniport communication is much more complex and may cause problems in some cases.
WaveRT - intended for modern adapters having one or more circular hardware buffers directly accessible to user-mode clients. It is the most efficient interface having almost no overhead.

WaveCyclic and WavePci exist in all Kernel Streaming implementations. WaveRT was introduced in Windows Vista so it is not available in XP and older versions.

For a user-mode Kernel Streaming client (including System Audio Engine), audio drivers that use WaveCyclic or WavePci port interfaces are indistinguishable. In Windows terms, they support a "standard streaming protocol". Kernel Streaming version of Audio Repeater application calls such drivers "legacy". On the contrary, drivers using WaveRT port interface support "looped streaming protocol" and are considered "realtime". Audio Repeater calls them "RT Audio".

Most modern audio drivers for embedded (hidden under the cover) hardware support RT Audio protocol. USB audio drivers usually support legacy one.

Applications that work via WASAPI, MME, DS and other high-level interfaces, generally use the following rules to choose PortCls interfaces: 1) In Vista and later OSes, they prefer WaveRT for best performance. 2) If some applications cannot access the endpoints, they try WavePci as a balanced solution. 3) Otherwise they try WaveCyclic which is the simplest and most compatible interface.

And here are Microsoft resources describing WaveRT:

• https://docs.microsoft.com/en-us/windows-hardware/drivers/audio/introducing-the-wavert-port-driver
• https://docs.microsoft.com/en-us/windows-hardware/drivers/audio/understanding-the-wavert-port-driver
• https://docs.microsoft.com/en-us/windows-hardware/drivers/audio/developing-a-wavert-miniport-driver

It seems like WASAPI may automatically select WaveRT mode if available, but some research needs to be done in the JUCE project:

• Does it need the new AudioClient3 subclass? It seems like it needs to use IAudioClient3::InitializeSharedAudioStream! (see https://docs.microsoft.com/en-us/windows/win32/api/audioclient/nn-audioclient-iaudioclient3)
• Does it need to query latency with the AudioClient3 "GetSharedModeEnginePeriod"? I think so! (see URL above)
• Does it need to do anything special to enable WaveRT? It doesn't seem like it! I think WASAPI auto-selects WaveRT and uses that to report its low latencies when supported.

[Arcitec]

Okay I have done more research and edited the first post. What I have read so far seems to confirm that JUCE will need to use the AudioClient3 subclass, query the supported shared mode periods (aka latencies), then request IAudioClient3::InitializeSharedAudioStream to get a low-latency shared stream.

Furthermore, the low-latency shared mode is only supported if the audio drivers support WaveRT, but it seems most devices released for at least half a decade should support it now since it was introduced in Windows Vista and is the default audio driver protocol in Windows.

Lastly, when running in "realtime audio" shared mode, we can expect latencies of 2-3ms BUT the Microsoft docs (see end of previous post) say that applications DO NOT and CANNOT request specific buffer sizes; instead, they request a specific "period" (latency) and then it is up to the application to fill that variable-length buffer.

It would be a huge leap for JUCE to support this since it provides shared-mode audio with latencies about as low as ASIO, with no special drivers

All modern machines have WASAPI drivers. Few have ASIO drivers!

[Arcitec]

Found a good Microsoft FAQ:

https://docs.microsoft.com/en-us/windows-hardware/drivers/audio/low-latency-audio#faq

3. If a driver supports small buffer sizes (<10ms buffers), will all applications in Windows 10 automatically use small buffers to render and capture audio?

No. By default, all applications in Windows 10 will use 10ms buffers to render and capture audio. If an application needs to use small buffers, then it needs to use the new AudioGraph settings or the WASAPI IAudioClient3 interface, in order to do so. However, if one application in Windows 10 requests the usage of small buffers, then the Audio Engine will start transferring audio using that particular buffer size. In that case, all applications that use the same endpoint and mode will automatically switch to that small buffer size. When the low latency application exits, the Audio Engine will switch to 10ms buffers again.

[zacharyneveu]

Any progress on this? Is it as easy as just dropping in the AudioClient3 class and checking if the user is on Windows 10?

[shangjiaxuan]

How do you plan to deal with thread scheduling? Threads work on a time frame of a few ms, and the buffer size will be smaller than 10ms.

How to ensure that there's no glitches? I was looking at the docs and perhaps needs a real-time work queue api for the callbacks to operate on such a resolution?

(Checking support should be simple with MSCOM, using audioclient* and cast to audioclient3* and get the interface, if fails fallback to audioclient*, if succeeds, mark it as have support)

[ed95]

This has been added to the develop branch in 6195a5a in addition to supporing sample rate conversion for WASAPI input and output devices. Thanks for the detailed summary @Bananaman - those links were helpful in getting this added.