Implementing a custom audio processor

[OneBadNinja]

Greetings,

I would like to create a music/video player app which applies a custom audio effect based on a personal C++ library that I have JNI bindings for. I am very new to Android development, and from my reading I have seen that custom audio processing is to be done within the Audio Processor interface: https://exoplayer.dev/doc/reference/com/google/android/exoplayer2/audio/AudioProcessor.html

My plan is to inject the C++ DSP code within the queueInput() method and run block-based processing there. I have been looking through example code, github and the exoplayer.dev guides but I'm struggling to get a clear picture of how to proceed with this. The first few questions i'd like to ask are the following:

• My DSP code uses block-based processing in the frequency domain requiring a fixed audio frame size. Is there anywhere I can set this to ie (256 frames, ie 512 samples for 2 channels)? Can I set a different number of channels for the output if I needed to (ie certain samsung tablets with 4 outputs)?

• The DSP library works with pcm float format but I've read that this is not supported for Audio Processors (ie Is it possible to get PCM float audio stream in AudioProcessor from the MP4 video? #8384 if I've understood this correctly). Should I implement my own converter or is there a native converter/configuration I can set somewhere?

• I need to work with a fixed sample rate when initializing my DSP library classes. From what I've read the SonicAudioProcessor can be used for this, so am I correct to assume that I would be placing it before my Audio Processor in the AudioProcessorChain to get ie 48khz audio passed from it?

Finally, is there any example code with custom audio processors that could help me understand the overall setup/process? I've come across a few bits of code that seem to be outdated/contradictory ie Audio Processors are now in the Audiosink instead of Renderers Factories. I would appreciate any pointers towards the right direction

Many thanks in advance for your time!

J

[krocard]

I would like to create a music/video player app which applies a custom audio effect based on a personal C++ library that I have JNI bindings for.

Is your effect having a low latency requirement? ExoPlayer doesn't support low latency playback currently (see #8665 (comment) and #8722). We might improve the situation in the future but have no timeframe to give you. If your algorithm doesn't take real time input (eg the user head direction) ExoPlayer latency should not be an issue.

I am very new to Android development, and from my reading I have seen that custom audio processing is to be done within the Audio Processor interface: https://exoplayer.dev/doc/reference/com/google/android/exoplayer2/audio/AudioProcessor.html

Indeed AudioProcessor is the API to use to inject custom DSP code in ExoPlayer audio pipeline. A custom AudioProcessor can be injected by subclassing DefaultRenderersFactory and overriding buildAudioSink to pass a processing chain with you custom AudioProcessor.

My DSP code uses block-based processing in the frequency domain requiring a fixed audio frame size. Is there anywhere I can set this to ie (256 frames, ie 512 samples for 2 channels)?

ExoPlayer pcm pipeline is stream oriented, not block based. As such your AudioProcessor should handle itself splitting and joining the ByteBuffer received in AudioProcessor.queueInput to the desired size.

Can I set a different number of channels for the output if I needed to (ie certain samsung tablets with 4 outputs)?

Yes, simply return the output format in AudioProcessors.configure().

The DSP library works with pcm float format but I've read that this is not supported for AudioProcessors (ie #8384 if I've understood this correctly). Should I implement my own converter or is there a native converter/configuration I can set somewhere?

The AudioProcessor pipeline currently only supports 16 bit pcm input. You have multiple solutions to work around the issue:

1. the simplest is probably to convert all input from 16bit pcm to float in you custom AudioProcessor. Take a look at FloatResamplingAudioProcessor that implements 32bit pcm to float conversion.
2. Moderate: add support all pcm formats in FloatResamplingAudioProcessor (and consider sending us a PR) and add the class before your custom AudioProcessor.
3. The most complex: add support for float processing chain in DefaultAudioSink and send us a pull request.

I need to work with a fixed sample rate when initialising my DSP library classes. From what I've read the SonicAudioProcessor can be used for this, so am I correct to assume that I would be placing it before my Audio Processor in the AudioProcessorChain to get ie 48khz audio passed from it?

Adding a SonicAudioProcessor configured for outputing at 48khz at the beginning of your chain (before converting pcm to float at Sonic doesn't support float) would indeed be the best solution.

Finally, is there any example code with custom audio processors that could help me understand the overall setup/process? I've come across a few bits of code that seem to be outdated/contradictory ie Audio Processors are now in the Audiosink instead of Renderers Factories. I would appreciate any pointers towards the right direction

The RendererFactories is the component that creates the AudioSink and passes it the AudioProcessors to use. The AudioProcessors are used in the AudioSink.

Don't hesitate if you have further questions.

[OneBadNinja]

Hi Kevin,

Thanks for your clarifications! Unfortunately there would be a low-latency requirement with realtime input after the initial groundwork, so I'll see if I can setup something with Oboe. It would be great if there would be a way to pass audio from exoplayer through to Oboe at some point :)

Many thanks
J

[krocard]

I've open an enhancement issue #8970 to track support of low latency ADSP. No work is schedule on this issue right now, so please don't expect any short term improvements.
If you desire to implement this yourself, we will gladly review any pull request.