What changes are needed to support streaming/real-time operation?

[thirtythreeforty]

Hello! I am looking for a library to perform analysis on a live stream from a microphone. Based on the example scripts, Acoular seems to do generally what I need, but it seems like it is only designed for prerecorded data. And in fact, in #12 you mentioned that it isn't designed for real-time use.

What changes would need to be made to do live processing? I am a programmer and can make modifications, but I would like to get a general idea about the scale of the work involved.

[1oly]

Are you only using a single microphone? In that case maybe start with portaudio and some python-bindings, e.g., PyAudio.
For real-time analysis with microphone arrays, I would assume that the hardware is doing the heavy-lifting DSP stuff and no software package can replace that... If that's your target maybe look at the NIdaq interface?

[thirtythreeforty]

no software package can replace that

My impression was that modern PCs should be more than capable of handling 4 or 6 mics at 44.1KHz, including the signal processing of those streams. (Now, Python may not be up to the task, but that's a different story...)

I was wondering what changes need to be made to Acoular to support streaming from something like PyAudio, performing the computation in real-time. For example, currently Acoular has the cache hard-wired in at several places. Obviously there is little use for that when the audio data is never written to disk in the first place.

[1oly]

My impression was that modern PCs should be more than capable of handling 4 or 6 mics at 44.1KHz, including the signal processing of those streams.

That is quite possibly true. The systems I have seen usually have a dedicated DSP chip or FPGA to handle the I/O and signal processing but that may stem from a time where processors were less powerful and more expensive. On the other hand, most microphone arrays have at least 32 channels and that might still be a bit too much for a pc that is not necessarily dedicated to that kind of I/O.

It's an interesting idea nonetheless! And I think it would be a nice to see how much is capable with just a modern pc. Cannot suggest any other paths to follow unfortunately.

[gherold]

While it strongly depends on what exactly you’re planning to do, realtime applications in Acoular are possible in principle. For our measurements, we actually do use classes with TimeSamples-like API (“result” generator etc.) whose task is to deliver measurement data from our hardware in realtime. At the moment, we do not use this to do beamforming, just to write the data directly on a hard drive. For time domain beamforming, you could tunnel the data through a BeamformerTime object. For this, you would have to solve two problems:

1. Speeding up the slow Python implementation of the TD beamformer
2. Find a fast display solution for the output

We are in fact working on both at the moment, but I don’t want to give a prediction as to when working solutions will be implemented in Acoular.

[dimakozin]

I tried to use SoundDeviceSamplesGenerator, but when I called method synthetic process freezes.

ts = acoular.SoundDeviceSamplesGenerator(device=5)
ps = acoular.PowerSpectra( time_data=ts, block_size=128, window='Hanning' )
st = acoular.SteeringVector(grid=rg, mics=mg)

bb = acoular.BeamformerBase(freq_data=ps, steer=st)
pm = bb.synthetic(8000, 3)
Lm = acoular.L_p(pm)

print(Lm)

Nothing is printed.
What I should to do?

[dimakozin]

Ok, I've debugged some code and discovered:
file spectra.py

for data in t.result(bs):
    ns = data.shape[0]
    temp[bs:bs+ns] = data
        while pos+bs <= bs+ns:
            ft = fft.rfft(temp[int(pos):int(pos+bs)]*wind, None, 0).astype(self.precision)
            calcCSM(csmUpper, ft)  # only upper triangular part of matrix is calculated (for speed reasons)
            pos += posinc
        temp[0:bs] = temp[bs:]
        pos -= bs

We have infinity loop 'cause t.result(bs) returns generator (and also have data from mics every audio stream callback)
Should it be async?

[gherold]

The frequency domain methods are currently designed to deliver only one result, i.e. the average over all time. If you have an endless audio stream, this would indeed mean that the PowerSpectra object is collecting data forever. For doing frequency domain beamforming using only part of a stream you could try putting a MaskedTimeInOut object between the SoundDeviceSamplesGenerator and the PowerSpectra objects (where you appropriately set "start" and "stop" sample values).
Alternatively, you could look into our "spectacoular" repository, where we implemented something like your intended application in the MeasurementApp. (for the relevant code, look at the "CSMInOut" class in here). Hope this helps.

[esarradj]

If you want to do realtime beamforming, you should only use classes that are derived from TimeInOut. The PowerSpectra class is not meant for this. You check out the examples for the time domain beamformers for a start.
Moreover the spectacoular package has some app examples that you can use to understand how realtime operation works. Within this package there is also the acoular_future sub-package with a preliminary BeamformerFreqTime class that allows to use frequency domain beamformers in this context.

[dimakozin]

from acuma16 import UMA16SamplesGenerator ModuleNotFoundError: No module named 'acuma16'

acoular/spectacoular#5