[Audio] Add Pipewire playback/capture sink

[Kontrabant]

Pipewire audio playback and capture

This adds a Pipewire playback/capture sink for the audio subsystem. Pipewire is designed to be suitable for both typical desktop and low-latency audio applications and is slated to replace both Jack and Pulseaudio, while providing compatibility layers, in Fedora 34. Pipewire has measurably lower overhead than Pulseaudio and streams can be routed with standard Jack graph processing tools such as Catia and QJackCtl.

Barring an application requesting extremely short or long mixing periods (0.67 to 170ms at 48KHz), in which case an intermediate stream is required, when a Pipewire stream callback fires the buffers are simply passed directly through to the application mixing callback, so overhead is minimized as SDL is little more than a shim in this case. Filling and submitting the buffer is a zero-copy operation, and, even if the input/output device is disconnected, the application callback will continue to fire at regular intervals as the Pipewire streams are created with the PW_KEY_NODE_ALWAYS_PROCESS flag. Note that this requires a fairly recent version of Pipewire as the audio mixing has only been considered ready for public testing in the last couple of months and the headers for versions earlier than 0.3.18 caused some warnings when building SDL. Pipewire 0.3 is considered API stable at this point.

Points of contention

• Pipewire spins up and sets the priority of the threads internally using rtkit to make the threads realtime priority, with the exact values being user configurable in pipewire.conf. If it's preferred that SDL creates the threads with it's internal API, it's just a matter of changing the pw_thread_loop to a pw_main_loop and calling pw_main_loop_run() in locally created threads.
• Pipewire has no native push interface, so trying to adapt it to the processing loops in SDL_audio.c, while doable, gets very ugly and requires extra buffer allocations and tracking for what amounts to slightly increased latency.
• Pipewire is increasingly omnipresent on systems even when not used for audio mixing as it's also used for routing video and window surfaces, and there seems to be no way to cleanly detect if it's the system audio mixer at runtime. Even if a distro ships Pipewire as the default audio mixer, someone could switch back to Pulseaudio, in which case the Pipewire SDL sink would still initialize properly, but nothing would play as Pulseaudio would be doing the mixing and own the output devices. The solution, for now, is having the Pipewire sink below Pulse and ALSA in the audio sink list hierarchy and requiring the user to activate it manually via an envvar, or build SDL sans Pulse and ALSA support.

[flibitijibibo]

This looks great!

Regarding the points of contention...

• If the backend has its own thread that's actually preferred; the SDL thread is really just for backends that only have a push API.
• The push API in SDL is, as far as I know, solely for convenience and knowingly exchanges that for latency, so even for non-Pipewire scenarios it's expected that using the callback is the best way to get samples to the OS. This part I wouldn't really worry about.
• Putting it underneath Pulse/Jack makes sense to me... is there an intended timeline for the migration from Pulse to Pipewire? I guess as long as Pulse isn't outright deprecated we can never really assume that Pipewire is in use, but if there's an expected time where we can say "yes, having Pipewire installed == Pipewire is in use" that'll help move things along on our end (and as far as deprecation goes, my guess is Ryan would love to be able to go from 23 backends to 22 some day!).

Aside from that I did have a couple questions:

• Does the role string have to be anything specific? In FAudio we have a category enum that we would like to reimplement, but I dunno if we need to be specific here. My hope is that I can just strip the beginning of the enum names and pass "GameEffects" or "Communications", for example.
• Something I've been putting off for a really long time is a GetAudioDeviceSpec API... so while I'm thinking about it: What's the best way to get device format information from Pipewire? This topic can be addressed in that issue rather than here, but I wanted to bring it up either way.

[Kontrabant]

• Putting it underneath Pulse/Jack makes sense to me... is there an intended timeline for the migration from Pulse to Pipewire? I guess as long as Pulse isn't outright deprecated we can never really assume that Pipewire is in use, but if there's an expected time where we can say "yes, having Pipewire installed == Pipewire is in use" that'll help move things along on our end (and as far as deprecation goes, my guess is Ryan would love to be able to go from 23 backends to 22 some day!).

Putting it under Jack means that it will never start unless explicitly specified, as iterating through the audio driver array will get stuck on OSS, which seems to always be included in the Linux build and initializes successfully, but doesn't actually do anything on modern systems. Maybe it's time to move OSS further down the priority list as it's been deprecated since 2001?

Unless an objection is raised by the steering committee in the next day or so, Fedora 34 is set to replace Pulse and Jack with Pipewire when it releases in a couple of months. I think Pipewire is going to be a Gnome dependency starting in 40, so odds are that other distros will start moving in that direction soon as well. I don't know if there will ever be a case where you can 100% count on Pipewire being the audio mixer, for as long as Pulseaudio exists, someone out there will be using it. I'd love to figure out a way to query if Pipewire is the system audio mixer without resorting to hacky methods such as looking for sinks/sources that aren't guaranteed to be present at application startup (for example, the only output device may be Bluetooth and not yet connected).

Aside from that I did have a couple questions:

• Does the role string have to be anything specific? In FAudio we have a category enum that we would like to reimplement, but I dunno if we need to be specific here. My hope is that I can just strip the beginning of the enum names and pass "GameEffects" or "Communications", for example.

No, it's just used for node property strings and might show up in node editors or control panels. It doesn't affect the processing of the audio in any way.

• Something I've been putting off for a really long time is a GetAudioDeviceSpec API... so while I'm thinking about it: What's the best way to get device format information from Pipewire? This topic can be addressed in that issue rather than here, but I wanted to bring it up either way.

Answered in the linked issue. Pretty simple, just query a string in the node properties.

[flibitijibibo]

That all makes sense to me - we'll have to defer to @icculus for the driver ordering since there's probably a good reason it's ordered the way it is. I think detecting Pulse v Pipewire would be the only major worry for defaulting to Pipewire, but that's about all I can think of.

Excited for F34 either way!

[icculus]

Maybe it's time to move OSS further down the priority list as it's been deprecated since 2001?

Yes, but also we should just refuse to initialize OSS if there isn't a /dev/sound* device node at startup. Even though one could be added dynamically, I think it's a reasonable heuristic.

(Alternately: does anything use Open Sound System anymore? Surely everything has moved to ALSA by now, if not a higher layer like PulseAudio...?!)

I'm going to look at this patch shortly, but I haven't yet.

[sezero]

(Alternately: does anything use Open Sound System anymore? Surely everything has moved to ALSA by now, if not a higher layer like PulseAudio...?!)

FreeBSD might (not 100% sure though.)

[icculus]

This looks reasonable at first glance. The immediate and obvious nitpick is that we use One True Brace format:

if (x) {  // brace on same line _even if it's a single statement inside the if block_
   y();
}

I'll clean this up if you don't, but I'd appreciate it if you'd push another commit to change that. We'll sort out the proper order of audio backends once this is shipping in Fedora. After we're comfortable that this backend is stable, we'd likely move it to the top, to favor it over Pulse, if that's the direction distros are heading.

[Kontrabant]

This looks reasonable at first glance. The immediate and obvious nitpick is that we use One True Brace format:

if (x) {  // brace on same line _even if it's a single statement inside the if block_
   y();
}

I'll clean this up if you don't, but I'd appreciate it if you'd push another commit to change that. We'll sort out the proper order of audio backends once this is shipping in Fedora. After we're comfortable that this backend is stable, we'd likely move it to the top, to favor it over Pulse, if that's the direction distros are heading.

Sorry about that, force of habit. I'll take care of it.

[Kontrabant]

Style issues are fixed. I'll have another commit soon that should get it ready for #4109.

[Kontrabant]

Reworked the enumeration process to retrieve additional information (channel count and default sample rate). It just needs the SDL_AddAudioDevice() lines updated with the SDL_AudioSpec paramter to work with #4109 now.

[Oschowa]

Recording like this won't work. The latency specified in PIPEWIRE_NODE_LATENCY only guarantees a minimal latency to the app. The actual latency might be lower depending on the graph. For example, just opening pavucontrol or recording from the same source from a different app that request a lower latency will lower the latency of the graph enough, that chunk->size < this->spec.size is always true and we get broken recording. We probably need to put an additional ringbuffer in between.

[Kontrabant]

Ah, I see. Rewriting it to always buffer the audio is simple enough, and it's just a bit of overhead from copying some bytes around. I wanted to do some work on that function anyways since I realized I'm ignoring the n_datas member of spa_buf. Not an issue in the output callback as we only use one buffer, but there is no guarantee of that when dealing with incoming audio.

[Oschowa]

I might be wrong, but i''m pretty sure the n_datas member will always be 1 when you request an interleaved audio format, so it should be fine. It would be different with planar audio formats, but since SDL doesn't support those, we don't have to care.

[Kontrabant]

That's what it sounds like in the documentation, although it's a little vague.

Each data block is, for example, a video plane or an audio channel. There are n_datas of those blocks.

This is where it would be nice to have a detailed spec stating something like

If the audio sample format is interleaved, n_datas will always be 1

Probably not worth worrying about though.

[Oschowa]

Pipewire is increasingly omnipresent on systems even when not used for audio mixing as it's also used for routing video and window surfaces, and there seems to be no way to cleanly detect if it's the system audio mixer at runtime.

One idea to figure this out might be to just do an initial round of device detection in PIPEWIRE_Init, and fail initialization if we don't get at least one node that's 'Audio/Sink' or 'Audio/Source'. That should be a pretty clear sign that pipewire handles audio on the system..

Edit: actually I'm not sure that true, disregard this until I can maybe confirm it.

[Kontrabant]

One idea to figure this out might be to just do an initial round of device detection in PIPEWIRE_Init, and fail initialization if we don't get at least one node that's 'Audio/Sink' or 'Audio/Source'. That should be a pretty clear sign that pipewire handles audio on the system..

Edit: actually I'm not sure that true, disregard this until I can maybe confirm it.

This is the obvious solution, however there are many cases where it could fail, such as where the only sink devices are an HDMI device that's powered off at application startup or a disconnected Bluetooth speaker.

I was thinking that looking for ALSA or Bluez factory objects in general might be more reliable, but even that seems sketchy since it's still making assumptions about the system underneath.

[Oschowa]

One idea to figure this out might be to just do an initial round of device detection in PIPEWIRE_Init, and fail initialization if we don't get at least one node that's 'Audio/Sink' or 'Audio/Source'. That should be a pretty clear sign that pipewire handles audio on the system..
Edit: actually I'm not sure that true, disregard this until I can maybe confirm it.

This is the obvious solution, however there are many cases where it could fail, such as where the only sink devices are an HDMI device that's powered off at application startup or a disconnected Bluetooth speaker.

I was thinking that looking for ALSA or Bluez factory objects in general might be more reliable, but even that seems sketchy since it's still making assumptions about the system underneath.

Probably best to ask on the pipewire gitlab or IRC directly and move it below Pulse/ALSA/JACK for the time being if there is no clean solution. Otherwise this looks good to me now and also works fine in testing. Nice.

[Cacodemon345]

(Alternately: does anything use Open Sound System anymore? Surely everything has moved to ALSA by now, if not a higher layer like PulseAudio...?!)

Solely FreeBSD nowadays use Open Sound System.

[Kontrabant]

(Alternately: does anything use Open Sound System anymore? Surely everything has moved to ALSA by now, if not a higher layer like PulseAudio...?!)

Solely FreeBSD nowadays use Open Sound System.

The Arch wiki has instructions on how to set it up on Linux. I doubt that any significant number of people are using it, but one of the universal certainties of Linux is that if you can use it, someone is using it.

[flibitijibibo]

Only thing that's missing at this point is the vim lines at the bottom of the new files (and the style described inside):

/* vi: set ts=4 sw=4 expandtab: */

That's all style though - functionally it looks good!

[flibitijibibo]

Extra space

[Kontrabant]

With the latest changes and cleanups I'm pretty happy with the code at this point, so marking as ready for final review.

• If you want to land Add SDL_GetAudioDeviceSpec #4109 first, this can wait and have the trivial changes made before pulling it in.
• Maybe wait until we know what version of Pipewire is shipping in Fedora 34, as that will be the version with official, production ready audio, and bump the minimum version requirements in the build scripts to that? Probably best not to have people build and use this driver with older versions where the audio was still considered a work-in-progress.

[Cacodemon345]

I think it is wiser to land this PR first into SDL2. #4109 isn't finished yet. It seems to be far from finished too.

[Kontrabant]

Only thing that's missing at this point is the vim lines at the bottom of the new files (and the style described inside):

/* vi: set ts=4 sw=4 expandtab: */

That's all style though - functionally it looks good!

Added the lines and fixed indentation. I had my format script set to 2 instead of 4 indentation spaces, so it's a big update, but it's all style fixes.