MTAudioProcessingTap to NVDSP

[Maigre]

Hi!
I am trying to apply an EQ on a iOS AVPlayer.
I know that it is not the best way to go, but i don't have room to choose the player used here.

From what i found here:
[https://chritto.wordpress.com/2013/01/07/processing-avplayers-audio-with-mtaudioprocessingtap/]
it is possible to extract raw audio data from AVPLayer using MTAudioProcessingTap.
Using this, i end up with a "process" method which is the way to treat raw data from AVPlayer.
In the example below this "process" method apply vDSP_vsmul on the audio flux
(it's a simplified example from the link attached).

If i am right, i should be able to use your Filters here instead of vDSP_vsmul.
My goal is to make a 10(ish)-band EQ (with your PeakingEQ which seems nice)...
Could you point me on how to achieve this inside that "process" function ?
I am a bit lost with those vDSP / Filters stuffs... but i feel i am not far from it:
The processing in your library is vDSP based too, so the works done by your library could seat here..

// Function called by MTAudioProcessingTap allowing treatment of audio flux..
void process(
    MTAudioProcessingTapRef tap, 
    CMItemCount numberFrames,
    MTAudioProcessingTapFlags flags, 
    AudioBufferList *bufferListInOut,
    CMItemCount *numberFramesOut, 
    MTAudioProcessingTapFlags *flagsOut)
{
    // Error detection
    OSStatus err = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, NULL, numberFramesOut);
    if (err) NSLog(@"Error from GetSourceAudio: %ld", err);

   // Arbitrary treatment given as exemple
    float scalar = 5;
    vDSP_vsmul(bufferListInOut->mBuffers[0].mData, 1, &scalar, bufferListInOut->mBuffers[0].mData, 1, bufferListInOut->mBuffers[0].mDataByteSize / sizeof(float));
    vDSP_vsmul(bufferListInOut->mBuffers[1].mData, 1, &scalar, bufferListInOut->mBuffers[1].mData, 1, bufferListInOut->mBuffers[1].mDataByteSize / sizeof(float));
}

Thank you !

Best Regards,

Thomas

[bartolsthoorn]

Hi Thomas, that looks very promising indeed! I will first show you how I'd approach it for a single peaking EQ:

Firstly, include NVDSP and initialise a NVPeakingEQFilter. It is important that you do this outside your process function, because it is more efficient and it also saves tiny pieces of audio between buffers, to ensure a smooth filter. You initialise a filter only once, and just change the parameters (such as gain) on the fly.

You can change the .centerFrequency and .Q to match your needs, and you can also change these while the audio is playing. You probably want to make a slider that changes the .G, the gain (given in dB).

#import "NVDSP/NVDSP.h"
#import "NVDSP/Filters/NVPeakingEQFilter.h"
NVPeakingEQFilter *PEF = [[NVPeakingEQFilter alloc] initWithSamplingRate:audioManager.samplingRate];
PEF.centerFrequency = 1000.0f;
PEF.Q = 3.0f;
PEF.G = 20.0f;

It seems your bufferListInOut->mBuffers data contains two channels, whereas NVDSP's default filter function filterData expects interleaved data (a single array containing two channels interleaved). However, right before NVDSP processes data, it deinterleaves data, resulting in two channels, similar to your set-up. This is why we can simply skip a step and use the internal filterContiguousData function directly. So to use NVDSP in your set-up, you would do something like this:

void process(
    MTAudioProcessingTapRef tap, 
    CMItemCount numberFrames,
    MTAudioProcessingTapFlags flags, 
    AudioBufferList *bufferListInOut,
    CMItemCount *numberFramesOut, 
    MTAudioProcessingTapFlags *flagsOut)
{
    // Error detection
    OSStatus err = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, NULL, numberFramesOut);
    if (err) NSLog(@"Error from GetSourceAudio: %ld", err);

   // Arbitrary treatment given as exemple
   [PEF filterContiguousData:bufferListInOut->mBuffers[0].mData, numFrames:(bufferListInOut->mBuffers[0].mDataByteSize / sizeof(float)) channel:0];
   [PEF filterContiguousData:bufferListInOut->mBuffers[1].mData, numFrames:(bufferListInOut->mBuffers[1].mDataByteSize / sizeof(float)) channel:1];
}

This ensures that both the channels are individually processed with the same filter. NVDSP changes the data variable in-place, so nothing has to be returned.

Outside the process function, you can asynchronously change the parameters (such as the gain .G) of a filter and the process function will automatically use your latest settings. The process function is called very often, for each audio buffer, so you should keep the code inside it minimal.

Once you got this working, it is quite easy to expand it to 10 filters. I recommend you to take a look at my example snippet: https://github.com/bartolsthoorn/NVDSP/blob/master/Examples/Equalizer.mm
It contains code snippets that are useful when making a 10-band filter (such as setting up 10 filters).

I hope this solves your problem, please let me know.

[Maigre]

Hi Bart,
thanks for your answer

after posting the question i was indeed looking into "filterData / filterContiguousData" and the Equalizer.mm example, and your answer confirms it, so it seems i am on the right track :)

I will try this approach this afternoon, and see if it works..
I'll keep you in touch.

Thanks again !

Best,
Thomas

[bartolsthoorn]

@Maigre any luck with implementing the solution? For now I'm closing the issue.