Extracting MFCCs on Mobile Devices in React Native

[certainlyWrong]

Hi guys, I'm trying to extract MFCCs from 5 second audios on mobile devices. The problem is that this requires some CPU stress, so I believe it's not that simple to do this with Javascript in React Native. I would like to know if anyone has any experience or information to help me.

[hughrawlinson]

Hi! I haven't tried this on react native. Have you tried this? I would love to see a flame graph if you have it.

[certainlyWrong]

Oi! Eu não tentei isso em reagir nativo. Você já tentou isso? Eu adoraria ver um gráfico de chamas se você tiver.

I'm working on it, but the main problem is the naive implementation of the DCT. So far I'm using a different implementation to extract MFCCs, but I'll test with Meyda's implementation too, maybe I have to use a different DCT.

In the implementation I'm using, the DCT needs to process an array with 17k elements.

[hughrawlinson]

Oh wow that's a lot. Have you considered splitting your array into chunks, running the extraction on each chunk and taking the average result?

[certainlyWrong]

Uau isso é muito. Você já pensou em dividir sua matriz em pedaços, executando a extração em cada pedaço e obtendo o resultado médio?

Yes, now I'm working on an implementation that cuts the audio by several frames, but the previous implementation didn't use this approach.

[hughrawlinson]

Which previous implementation? Was it working with meyda before and has since stopped working?

[certainlyWrong]

Sorry, I was referring to a function for MFCC made by a friend of mine. Anyway you were right about the strategy of slicing the audio into frames. I found a Jupyter Notebook that made things pretty clear to me. I'll leave the link here, although I know it won't add much to what you already know.

https://www.kaggle.com/code/ilyamich/mfcc-implementation-and-tutorial/notebook

[hughrawlinson]

Cool, thanks for the update, and I'm glad you got it working :)