The state of DSP functionality on the Linux Desktop is not good. What can we do?

[jrelvas-ipc]

Note: For this discussion, "Linux Desktop" refers to traditional Linux distributions which share the general APIs, projects and components belonging to freedesktop.org (Ubuntu, Fedora Workstation, RHEL Workstation, etc.). ChromeOS is not included.

Hey everyone. Long time lurker, but first time interacting with development here directly.

So, I've recently been looking at the state of media offloading in the Linux desktop ecosystem and trying to diagnose (and potentially fix) shortcomings with offloading/hardware acceleration stacks in Linux. This started with the most common part, video codec acceleration, where I made some discoveries (most notably, programs not using existing JPEG hardware decoding capabilities for webcams, increasing power consumption unnecessarily). Now, I'm looking into audio DSPs, which lead me to SOF.

Newish Intel and AMD platforms include a DSP, whose functionality is implemented in Linux by SOF. Unfortunately, the drivers, firmware and topologies SOF provides for these "PC platforms" is extremely limited and, to my understanding, provides little to no signal processing.

According to Intel's PCH datasheet for Raptor Lake, its DSP is capable of a wide range of functionality, such as post-processing, voice recognition, miscellaneous sensor/communication offloading and audio codec offloading.

Unfortunately, most of these features appear to be missing in action for intel avs topologies and firmware. The DSP has no functionality outside of basic DMIC recording, audio buffer passthrough (no actual processing done to audio), and very recently, bluetooth audio offload.

Linux's audio drivers have been traditionally subpar for PCs. Although the sound stack is great, drivers for PC audio are limited and lack a lot of functionality. This is also the case with SOF's implementation of intel/amd platforms, where enablement is limited "making audio playback and recording work". As hardware vendors and third-parties start putting additional effort on Linux, especially in its graphics and power management stack, I believe it's time to consider giving PC audio some much needed love.

First of all, what's the state of DSP functionality in Intel and AMD platforms? Are there any plans for implementing any of the following functionalities?

• General post-processing (EQ, mixing, etc.)
• Capture pre-processing (Noise removal, etc.)
• Wake on Voice (WOV)
• Codec offloading (mp3, aac, opus, etc.)
• General algorithms and signal processing

If not, what's blocking these functionalities? Is this a 3rd party IP/licensing issue? Lack of interest? Something else? SOF's architecture is capable of implementing the functionalities I mentioned (done for some non-PC platforms), so I'm guessing there's a specific reason here.

SOF documentation has been pretty confusing for me, so I might have a few details wrong. If I'm misunderstanding something, please steer me in the right direction.

[lgirdwood]

I'll give some comments inline, that being said there are some difficulties around FW code signing, 3rd party processing IP and proprietary compilers that make it more difficult to get involved than other FW projects or to freely share binaries on Linux.

First of all, what's the state of DSP functionality in Intel and AMD platforms? Are there any plans for implementing any of the following functionalities?

• General post-processing (EQ, mixing, etc.)

Upstream and available today. Topologies using most of the upstream modules are in production today on Chrome.

• Capture pre-processing (Noise removal, etc.)
• Wake on Voice (WOV)
  A lot of these are 3rd party algorithms with licensing/royalties.

• Codec offloading (mp3, aac, opus, etc.)

More than happy to accept code for codecs upstream where legally permitted.

• General algorithms and signal processing

If not, what's blocking these functionalities? Is this a 3rd party IP/licensing issue? Lack of interest? Something else? SOF's architecture is capable of implementing the functionalities I mentioned (done for some non-PC platforms), so I'm guessing there's a specific reason here.

Main reason is most of the codec/algo IP that can run on the xtensa DSP is not opensource and is proprietary. This is fine for Chrome/Windows, but not possible on Linux.

SOF documentation has been pretty confusing for me, so I might have a few details wrong. If I'm misunderstanding something, please steer me in the right direction.

Please do raise bugs in sof-docs for anything that is unclear or ambiguous, most of the community are experienced SOF developers so we may not see any missing detail in the docs.

There are 2 ways to get involved and add more pre/post processing today if you do have time to contribute

1. Via Chromebook/UP board - as SOF code signing disabled in Chrome developer mode or public key is used. i.e. native SOF FW code can be developed and tested on HW (needs Cadence compiler for processing work as GCC xtensa is very slow as it has no xtensa SIMD support). Clang with xtrensa SIMD is being upstreamed atm.

2. Via ALSA plugin - this is SOF pipeline/topology/modules running in userspace that can be used for development on a regular Linux distro with no special tools or code signing. i.e. generic C code written and working here should work on DSP.

@singalsu is adding more offloaded pre/post processing for Linux in v2.9 (end of March), he should be able to share his topologies soon.

[jrelvas-ipc]

Thanks for your write-up, it makes things a lot clearer!

Main reason is most of the codec/algo IP that can run on the xtensa DSP is not opensource and is proprietary. This is fine for Chrome/Windows, but not possible on Linux.

So let me clarify, if I were to write an opus decoder algorithm from scratch, for instance, could this be accepted for upstreaming? I guess for situations where the issue is with proprietary decoding of otherwise open codecs, an open implementation would be the right fix.

Also, quick question about xtensa. Is this ubiquitous to the "XTOS architecture" mentioned in docs? If so, does the move to the "Zephyr architecture" open up new possibilities for signal processing?
I'm pretty sure XTOS is just the SOF architecture and isn't the actual DSP's architecture, but I'd like to clear that up.

[marc-hb]

Via Chromebook/UP board - as SOF code signing...

All options should be listed here:
https://thesofproject.github.io/latest/getting_started/intel_debug/introduction.html#firmware-binary
https://thesofproject.github.io/latest/getting_started/intel_debug/introduction.html#chromebooks-and-sof

This section looks up to date.

[jrelvas-ipc]

Hmm, bummer. I don't have that spare hardware around.

Would it be any help to prototype via an emulated DSP? Is it feasible to get a rough implementation of a codec done with a generic topology?

[lgirdwood]

Thanks for your write-up, it makes things a lot clearer!

Main reason is most of the codec/algo IP that can run on the xtensa DSP is not opensource and is proprietary. This is fine for Chrome/Windows, but not possible on Linux.

So let me clarify, if I were to write an opus decoder algorithm from scratch, for instance, could this be accepted for upstreaming? I guess for situations where the issue is with proprietary decoding of otherwise open codecs, an open implementation would be the right fix.

Yes, providing it is upstream quality, fast on target DSPs, passes CI, passes fuzzing and Opus is fully patent/license free.

Also, quick question about xtensa. Is this ubiquitous to the "XTOS architecture" mentioned in docs? If so, does the move to the "Zephyr architecture" open up new possibilities for signal processing? I'm pretty sure XTOS is just the SOF architecture and isn't the actual DSP's architecture, but I'd like to clear that up.

SOF can use 2 RTOSes today 1) xtos, original RTOS on xtensa only arch, 2) Zephyr - can run on multiple architectures including host PC as application.

Btw, there are other things that also need work on Linux that are less complex than algorithms/codecs. e.g.

1. Desktop (Gnome/KDE) support for EQ sliders in sound settings. This would runtime load new coefficients to DSP and alter output.
2. Desktop visualization of beamformer position - think Alexa blue leds showing speaker direction.
3. DRC tuning tool to enable customized speaker output, this would use existing DRC Octave scripts, but would allow the users to customize their own configuration.

Today there are quote a lot of processing modules that would benefit from nice userspace UI information/controls/tuning.

[plbossart]

So let me clarify, if I were to write an opus decoder algorithm from scratch, for instance, could this be accepted for upstreaming? I guess for situations where the issue is with proprietary decoding of otherwise open codecs, an open implementation would be the right fix.

Yes, providing it is upstream quality, fast on target DSPs, passes CI, passes fuzzing and Opus is fully patent/license free.

To be clearer, it's not a one-time code import. Such libraries would need a clear long-term maintenance plan to deal with vulnerabilities or broken streams.

[jrelvas-ipc]

After doing some research, these codecs seem to be legally free of restrictions for SOF, while still having enough usage to justify an implementation:

• Opus: Patent licensing is royalty-free with software. Most popular VoIP and streaming audio format.
• Vorbis: No patents.
• MPEG Layer III (MP3): All patents expired. Significant amount of legacy content still exists.
• Dolby Digital (AC-3): All patents expired.
• FLAC: No patents.

The only missing codec would be AAC, which unfortunately offers no open-source friendly licensing. Obviously, proper legal research (not mine) would have to be made to confirm the suitability of these codecs.

If there's developer interest, these seem to be good candidates for implementation into SOF.