Downmix example -- issue in Bypass mode (AUD-4628)

[Allan-Perez]

Environment

• Audio development kit: none (ESP32-D0WD with a class-D amplified hooked up through I2S)
• [Required] Module or chip used: ESP32-D0WD-v3
• [Required] IDF version (run git describe --tags in $IDF_PATH folder to find it): v4.4.4-278-g3c8bc2213c
• [Required] ADF version (run git describe --tags in $ADF_PATH folder to find it): v2.5-28-gf060491
• Build system: idf.py
• [Required] Running log: All logs from power-on to problem recurrence
• Compiler version (run xtensa-esp32-elf-gcc --version in your project folder to find it): (crosstool-NG esp-2021r2-patch5) 8.4.0
• Operating system: macOS
• Using an IDE?: No
• Power supply: Battery

Problem Description

The example examples/advanced_examples/downmix_pipeline does not work properly when the mode of the downmixer is set to bypass. In particular, the audio, although reproducing, is choppy and not at all what it should be. I changed the example so as to run with spiffs (I have no sdcard at the moment), but I've also run the spiffs_mp3 example successfully, so I'm fairly certain it's no problem of spiffs reader. I would think the problem was somewhere in my changes to the example code, but after noticing some strange behaviour and setting the downmix to bypass for a couple seconds before introducing a newcome channel, I see that when the downmixer is set to bypass the audio is choppy and broken, but when the downmixer is set to on, while both, base and new, channels are playing, the audio is smooth, nice and behaving as expected. I thought that maybe the resampling is the problem, but then I resampled the audio files (in my own computer with ffmpeg) so as to no resampling in runtime is needed, but the same problem persists. I found it necessary to remark that the program does not crash or give any relevant warnings or errors. The only warning it gives is AUDIO_THREAD: Make sure selected the 'CONFIG_SPIRAM_BOOT_INIT' and 'CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY' by 'make menuconfig' which I doubt is an issue since the spiffs_mp3 example also gives the same warning, but still behaves as expected.

Expected Behavior

When running the downmix example, the audio sounds smooth before activating downmixer, and when two channels are playing, both audio files sound smooth.

Actual Behavior

When running both channels (downmixer on) both audio files sound smooth, but when only base is playing (downmixer set to bypass) the audio is choppy.

Steps to Reproduce

1. Board.h with the example code in get-started/play_mp3_control, setting my own pins for I2S (BCK 27, WS 12, DO 26, DI -1, MCK -1). I don't set anything else in my board since I only require i2s to drive my amp. By this I mean that everything else in the board component example (from get-started/play_mp3_control) I set to -1.
2. The audio files are as in the example examples/player/pipeline_spiffs_mp3/tools/. There, I have two audio files, one adf_music.mp3 (which I resampled to mono, 48000Hz) and another called mariobros.mp3 (also mono, 48000Hz) (resampling done with ffmpeg)
3. Load the tools directory into spiffs with spiffs_create_partition_image(storage ../tools FLASH_IN_PROJECT) in main/CMakeLists.txt
4. idf.py -p PORT_PATH flash monitor

Code to Reproduce This Issue

https://gist.github.com/Allan-Perez/8494abef87e701e933f2248664b87c2e

Debug Logs

Debug log (taken with CTRL-R CTRL-L) can be found in the comment in the gist given above. Apologies for the debug logs that mention and SDCARD -- I didn't change the logs given in the example, so although SD card is mentioned, my code makes no use of it, as it can be seen in the aforementioned gist.

[Allan-Perez]

After doing some deeper digging, I found out that setting the ringbuffer timeout to 50 ticks on the base channel is sufficient for the chopping to stop (i.e. once I added downmix_set_input_rb_timeout(downmixer, 50, INDEX_BASE_STREAM); right after audio_pipeline_run(base_stream_pipeline); the behavior of the audio is again as expected, with smooth audio in bypass mode).

I'm not sure if this is a fix or a workaround, since for previous versions this was not needed (and in the official example, this line is not added), and I can't really work out the need for this. Tweaking around with esp-adf-libs/esp_codec/downmix.c I found out that downmix->ticks_to_wait[0] is 0 if I don't set_input_rb_timeout, so one would think that with 0 ticks timeout, the buffer reading should be smooth instead of cracky (which is not what happens). Moreover, out of curiosity I commented out the set_input_rb_timeout after the downmixer is switched on with the new income channel, expecting to get cracky audio once the downmixer is turned on too, but the audio was actually smooth. I expected cracky audio because when downmixer is in bypass mode without using the input rb timeout function call, the base audio is cracky.

I'm not sure if all this is intended behaviour or if it's really a bug. That's why I leave this issue open even though I found a sufficient solution (or workaround) for my problem. Hopefully someone much more acquanted with ADF might give a definitive call.

[houhaiyan]

@Allan-Perez If downmix is in bypass status, the smaller the ringbuffer timeout of newcome music, the better.A little longer the ringbuffer timeout of base music, the better. If new music is coming, please make the ringbuffer timeout period of newcome music longer.In fact, you already do. The reason for this is that the arrival of each path of audio depends on whether the data can be read in the ringbuffer in time. If it can't read the data, the mixer waits for it to arrive. When there is only one path of data, expect a slightly longer base stream ring buffer timeout so that the data is read.

[Allan-Perez]

@houhaiyan thanks for replying. It is rather unclear to me how exactly downmixer is meant to be used. I am trying to use it with respect to some events (from sensors and internal states), with which sometimes we require only one channel (a base music) and some other times we need to put in an extra sound (e.g. think of mario bros jumping, while the background music is playing). However, the behaviour of the downmixer is very erratic, for very unclear reasons. It fails without any error or warning message whatsoever. In particular, when I add, for example, the mario jumping sound, I do so by resetting ringbuffer and elements of "newcome_stream_pipeline", setting uri (in my example the uri of spiffs reader element), and running the pipeline, setting the Downmixer ON (since base music is still playing), and changing input buffer to 50 (downmix_set_input_rb_timeout(downmixer, 50, INDEX_NEWCOME_STREAM); and with INDEX_BASE_STREAM too), but after the new channel finishes playing, sometimes the base music keeps playing (as expected), sometimes it stops playing (clearly not expected), and sometimes it keeps playing extremely cracked and barely recognisable (also not expected). I understand I should change the input ringbuffer timeout everytime the downmixer state changes (one channel playing-> bypass and timeout 0, two channels playing-> switch on and timeout 50 ticks), but the behaviour of the program is just glitchy. I suspect is because I use it in unintended ways, but I've been trying too many different things for me not to think that there is some bug.

[Allan-Perez]

After doing further investigation, I have found that the problem rises with mutex problems (e.g. starting or stopping a channel while it was being stopped) rather than the downmixer bypass mode. Apologies for any inconvenience and many thanks for your help.