odascore exits spontaneously (RPi B+ w/Matrix Voice)

[elijahparker]

First, thanks for such a great tool! This is something I've been searching for and am excited to start using it.

I'm using it with an RPi B+ and the Matrix Voice. I think I've got it all setup ok, but now after running for 30 seconds to a minute or so, it exits. During the time it is running, it connects to ODAS Studio and I can see the audio activity, so I can confirm the mics and directional locating are working.

But then it exits with code 0. Here's the output:

~/odas $ ./bin/odascore -c ./config/matrix_voice.cfg -v
+--------------------------------------------+
|    ODAS (Open embeddeD Audition System)    |
+--------------------------------------------+
| Author:  Francois Grondin                  |
| Email:   francois.grondin2@usherbrooke.ca  |
| Website: introlab.3it.usherbrooke.ca       |
| Version: 1.0                               |
+--------------------------------------------+
| + Initializing configurations...... [Done] |
| + Initializing objects............. [Done] |
| + Launch threads................... [Done] |
| + Threads running.................. [Done] |
| + Free memory...................... [Done] |
+--------------------------------------------+

It pauses at Threads running.................. for a little while while ODAS Studio shows it is connection and operational.

As a quick test I added for(;;) after that step in client.c, and while this kept it from exiting, it still only connected to ODAS Studio the same amount of time, so it seems the threads are exiting on their own. I don't know enough about the architecture of the application yet to dig further, but I thought I would start here to see if there's something I'm missing.

Thanks!

[FrancoisGrondin]

Thank you for your feedback! Are you running both Odas Studio and Odas on your RPi B? If so, what you are most probably experiencing is buffer overflow: odas keeps a buffer of messages, and if it is not able to process them in real-time, it will eventually accumulate until it overflows. Odas Studio can use quite a lot of processing power to render the 3D graphs. My suggestion is that you run Odas Studio on a personal computer, and let odas run by itself on the RPi, and connect both via sockets. I think I'll add a message that tells when there is an overflow, instead of simply exiting... this will help troubleshooting.

Please tell me if this fixes your issue!

Cheers,

Francois

[elijahparker]

Thanks for the reply! I should have mentioned that, but no, I'm running Odas Studio on my PC and only running odascore on the RPi (and actually I just realized it's the B+). The RPi is not running a GUI and I'm connected via SSH.

Here's the output of free before I start odascore:

~/odas $ free
             total       used       free     shared    buffers     cached
Mem:        379568     213376     166192       9852      24060     140752
-/+ buffers/cache:      48564     331004
Swap:       102396          0     102396

And while it's running and connected to Odas Studio:

~/odas $ free
             total       used       free     shared    buffers     cached
Mem:        379568     232916     146652       9852      24020     140432
-/+ buffers/cache:      68464     311104
Swap:       102396          0     102396

I look forward to the buffer overflow message. I could just add it myself as well -- what file would that go in? Thanks!

[FrancoisGrondin]

Ok good thanks for the precision. Do you also have info about cpu usage? In this case you might have enough memory, but if the processor cannot cope with real-time the buffers will overflow (I've set the number of messages in the queue to 100... to the limitation here is this magic number, and not the memory available on the system). Either way, if the RPi is not able to do things in real-time, accumulation will lead to overflow soon or later, even with more memory available. I still find this strange as I was able to have SSL, SST and SSS to run in real-time on a Pi3 with 16 mics. If you could send me the config file I'd be curious to have a look at your config see if there is something that eats up all processing power. Also, if you run the system with a recorded RAW file and call odascore this way:

odascore -c myConfig.cfg -v -s

The system will run everything sequentially (instead of using multiple thread) and will produce a report at the end with the % of processing usage for each module. That's something I used a lot for troubleshooting. If you exceed 100%, then you know this is the problem, and then you can spot which modules take more processing power.

Now, as for buffer overflow, the number of messages that can be queue is defined in demo/objects.c, at line 765:

objs->nMessages = 100;

You can change it to a larger value (say 500). If my logic is correct, odas should be able to run a bit longer before terminating.

Now, this problem will only occur when you are getting your raw signals from the sound card, which feeds odas with synchronous frames. In file src/source/src_hops.c, at line 363:

` if (err = snd_pcm_readi(obj->ch, obj->buffer, obj->hopSize) > 0) {

        rtnValue = 0;    

    }
    else {

        rtnValue = -1;

    }`

When returning -1, the source object basically tells the system something went wrong. You could try to add a printf that prints an error message in the else section of the condition. I should eventually set up some sort of list of return values that carry the info regarding the type of error met, so that it can propagates through the pipeline.

Please keep me updated,

Francois

[elijahparker]

Thanks for the detailed info! You're right, that is what's happening. It seems the CPU can't keep up. With the single-threaded mode it exits immediately instead of running for a little first. It appears to be the SSL and Classify modules that are using the most CPU. I tried to disable the Classify module by removing the block in the config, but it complained about it missing. Then I commented out all the lines in demo/threads.c , but it just hung on startup. Here's the output from the single-threaded mode (and you can see the printf I added where you mentioned):

$ ./bin/odascore -c ./config/matrix_voice.cfg -v -s
+--------------------------------------------+
|    ODAS (Open embeddeD Audition System)    |
+--------------------------------------------+
| Author:  Francois Grondin                  |
| Email:   francois.grondin2@usherbrooke.ca  |
| Website: introlab.3it.usherbrooke.ca       |
| Version: 1.0                               |
+--------------------------------------------+
| + Initializing configurations...... [Done] |
| + Initializing objects............. [Done] |
| + Processing....................... Error reading source, exiting...[Done] |
| + Free memory...................... [Done] |
+--------------------------------------------+
|              Profiler summary              |
+--------------------------------------------+
| + Raw                                      |
|    - Source.......... 000.002 (007.33%)    |
|    - Connector....... 000.000 (000.75%)    |
| + Mapping                                  |
|    - Module.......... 000.000 (000.63%)    |
|    - Connector....... 000.000 (000.55%)    |
| + Resample                                 |
|    - Module.......... 000.015 (063.80%)    |
|    - Connector....... 000.000 (000.27%)    |
| + STFT                                     |
|    - Module.......... 000.002 (006.83%)    |
|    - Connector....... 000.000 (000.88%)    |
| + Noise                                    |
|    - Module.......... 000.001 (005.80%)    |
|    - Connector....... 000.000 (000.35%)    |
| + SSL                                      |
|    - Module.......... 000.040 (172.85%)    |
|    - Connector....... 000.000 (000.28%)    |
|    - Sink............ 000.001 (004.07%)    |
| + Target                                   |
|    - Injector........ 000.000 (000.19%)    |
|    - Connector....... 000.000 (000.15%)    |
| + SST                                      |
|    - Module.......... 000.000 (001.37%)    |
|    - Connector....... 000.000 (000.25%)    |
|    - Sink............ 000.001 (004.80%)    |
| + SSS                                      |
|    - Module.......... 000.001 (006.43%)    |
|    - Connector....... 000.000 (000.00%)    |
| + ISTFT                                    |
|    - Module.......... 000.002 (008.43%)    |
|    - Connector....... 000.000 (000.24%)    |
| + Resample                                 |
|    - Module.......... 000.001 (004.75%)    |
|    - Connector....... 000.000 (000.27%)    |
| + Volume                                   |
|    - Module.......... 000.000 (000.62%)    |
|    - Connector....... 000.000 (000.19%)    |
|    - Sink............ 000.001 (003.75%)    |
| + Classify                                 |
|    - Module.......... 000.039 (168.41%)    |
|    - Connector....... 000.000 (000.19%)    |
|    - Sink............ 000.000 (000.14%)    |
+--------------------------------------------+
| + TOTAL.............. 000.108 (464.79%)    |
|    - Module.......... 000.102 (439.93%)    |
|    - Connector....... 000.001 (004.76%)    |
|    - Source.......... 000.002 (007.33%)    |
|    - Sink............ 000.003 (012.77%)    |
+--------------------------------------------+

I think you're right that it seems to be that the CPU is not keeping up, so I bought an RPi3 that's arriving today. So don't worry about figuring out anything more until I test with the RPi3. I will report my results with the RPi3 as soon as I get the chance.

Thanks again for the excellent program and your time and assistance!

[FrancoisGrondin]

Good! At least we know what's going on! I'm currently working on the granular processing version (I'll push a branch soon), which means unused module will be automatically disabled. In this example, the classify module will not run. But yes using a RPi3 is definitely a good way to start ;) Resampling only is taking 63%... which is a lot :O Please keep me updated!

[leejaeuk]

How is it going? I have same issue ...

[leejaeuk]

I'm make socket server use Python code.
It worked a few second but like above issue, immediately exits all thread.