dsound: mixthread causes starvation when many sound buffers have to be mixed

[w-flo]

This is most likely an upstream bug, but I don't feel comfortable reporting this at wine bugzilla since I'm not sure how to reproduce this using wine, as the steam DRM prevents me from launching the game.

The symptom is that Zusi 3 Aerosoft Edition (1040730) freezes for seconds up to minutes, or maybe even indefinitely, when the player is close to some freight trains – but only when using the builtin dsound.dll. The freezes disappear when using the winetricks dsound.dll, but sound is distorted/weird/crackling when using the native winetricks dll, so that's not a good solution.

In Zusi, sometimes there are more than 100 "sound effect sources" close to the player, which causes more than 100 sound buffers to be created and added to the dsound device. Apparently in wine and proton the dsound "mixthread" is responsible for mixing all those buffers into one. It does so by waiting for work to come in, then calling PerformMix(), in a loop. This works fine as long as PerformMix() returns before more mixing work is available. In that case the mixthread blocks for some time while waiting for new work. However, as the number of buffers to mix approaches 100, my CPU (Phenom II X4 955) can't keep up and the mixthread ends up looping without blocking inbetween PerformMix() calls.

Before calling PerformMix, the mixthread locks "buffer_list_lock" (in shared mode) to make sure the buffer list doesn't change, and then releases the lock after mixing. This works fine most of the time, but when the mixthread can't keep up with incoming work, so it doesn't block between PerformMix() calls, the lock is released and then almost immediately re-acquired.

This is what causes the freezes in Zusi: When a new sound source comes into range, Zusi wants to create a new buffer and add it to the dsound device using the AddBuffer function. That function needs to lock "buffer_list_lock" exclusively. The lock implementation puts the AddBuffer thread to sleep (blocking on a semaphore) while waiting for the mixthread to finish its current work and release the lock. When the lock is released by the mixthread, I suspect (from looking at the Rtl lock release implementation) that the AddBuffer thread actually wakes up (semaphore gets signaled) and both mixthread & AddBuffer threads then race to acquire the lock, but the mixthread almost always wins that race and re-acquires the lock before the AddBuffer thread manages to lock it exclusively. So the AddBuffer thread is put to sleep again, blocking on the semaphore again to wait for exclusive access. This can lead to minute-long or maybe even infinite starvation of the AddBuffer thread, which is probably the main Zusi thread (or some other Zusi thread that needs to make progress for the game to continue), causing the freeze.

I have verified that adding a "Sleep(1);" at the top of the while loop inside DSOUND_mixthread, to allow some time for the AddBuffer thread to acquire the lock, fixes that freeze without negatively impacting sound performance in most situations. Obviously, in situations in which the mixthread can't keep up with incoming work, sound starts to stutter, but that's better than a freeze (and probably expected when mixing 100 or more sound buffers on a 10 year old CPU).

The sleep solution is not perfect. It prevents the mixthread from fully utilizing the CPU even when the lock is not needed exclusively by any other thread. Also, maybe 1ms sleep is already too much for some sound buffer formats and the mixthread would have to call PerfomMix() more than once per millisecond in those cases? For me, PerformMix() is typically called every 8-12ms. So it would probably be better to enforce fairness through the Rtl lock implementation if the Rtl lock specification allows that, to make sure threads waiting for exclusive lock access are eventually granted access, even with a different thread attempting to re-acquire the lock (in shared mode) very quickly after releasing it.

[w-flo]

I tried to create a simple reproducer, but the timing in that reproducer exe appears to be the opposite of Zusi: mixthread loses the race against my main thread (waiting in AddBuffer) most of the time, so it fails to reproduce the starvation. Not sure why, maybe related to CPU caches or work going on in other threads during Zusi gameplay?

[w-flo]

With esync enabled, my reproducer now suffers from the same issue. So it's a combination of esync + dsound-mixthread fails to keep up because there are so many secondary sound buffers.

mixthread calls WaitForSingleObject between releasing and re-acquiring the lock, so it's actually no surprise that enabling esync (which speeds up WaitForSingleObject) could lead to mixthread hogging that lock. Edit: I'd say it's an upstream dsound issue (it depends on WaitForSingleObject being slow to work correctly with lots of buffers)

My reproducer code: https://gist.github.com/w-flo/d822b9f43fb888af649a93f973166814
(Edit: and repro stdout:

[..]
Added buffer No. 75, took 0.074412 sec!
Added buffer No. 76, took 0.199616 sec!
Added buffer No. 77, took 0.718821 sec!
Added buffer No. 78, took 0.161324 sec!
Added buffer No. 79, took 2.369803 sec!
Added buffer No. 80, took 4.995385 sec!
Added buffer No. 81, took 1.480235 sec!
Added buffer No. 82, took 2.944519 sec!

)

[w-flo]

Upstream report: https://bugs.winehq.org/show_bug.cgi?id=48408

[w-flo]

This is now fixed upstream for upcoming wine 5.2:
https://source.winehq.org/git/wine.git/?a=commit;h=13087008437d80542123fc3087cbef85f636a50a

[qwertychouskie]

This is now in the new Proton RC, can this be closed? I can confirm sound now works in TrackMania Nations Forever.

[w-flo]

Confirming that 5.0-7 RC fixes the "Zusi 3 - Aerosoft Edition" freezing issue. I'm not sure if I am supposed to close the issue myself, but feel free to close this now.