No AudioOut sounds with ESP-IDF v4.4

[meganetaaan]

Build environment: Linux
Target device: M5Stack Basic/ M5Stack CORE2

Description
With M5Stack and the latest ModdableSDK public build, no sound can be played through the AudioOut class, leaving a slight noise at the beginning.

Steps to Reproduce

1. Build and install piu/sounds example app using mcconfig -m -p esp32/m5stack.
2. The device cannot play any sound including startup sound(bflatmajor.maud) .

Other information
After a little digging:

• After updating Moddable to the version that requires esp-idf v4.4 the lack of sound occurred.
• No exception thrown. AudioOut.Callback is successfully invoked.
• AudioOut.Tone also cannot be played.

Then I guessed there is a regression or breaking change in i2s features of esp-idf but couldn't go further.

[phoddie]

Thanks for the report. We'll take a look. This wouldn't be the first time that an ESP-IDF update caused audio problems, alas.

[phoddie]

Based on testing from @mkellner, it appears that I²S audio works, for example on the M5Atom Echo. The failures occur when using the internal ESP32 DAC, such as on the M5Stack and M5Stack Core2.

[phoddie]

This one is stubborn. There are no errors reported by the i2s_* driver calls. Audio is written to the driver, but it doesn't play beyond some pops. There has been work on I2S in ESP-IDF 4.4, from the release notes:

I2S: Fixed mono mode support, slave full-duplex mode and event queue bug, finish the refactor for I2S driver layer.
I2S: Improved clock division calculation speed, expanded the supported frequency range of APLL, 8K sample rate is supported now.

It appears that at least one other developer has encountered this problem with ESP-IDF 4.4, issue #6470 in the ESP-IDF repository.

[L-KAYA]

Hi @phoddie @meganetaaan,

Sorry for the late reply, we have looked into this issue, it cause by the data bit shift which is introduced by I2S_COMM_FORMAT_STAND_I2S. The built-in DAC is using MSB format to transport data, so setting communication_format to I2S_COMM_FORMAT_STAND_MSB can solve this problem.

There will be docs or force setting the communication mode to MSB.

[phoddie]

@L-KAYA – thank you for the reply. We'll give that a try.

[phoddie]

@L-KAYA – Unfortunately, setting the communication_format to I2S_COMM_FORMAT_STAND_MSB does not make a difference in our testing:

	i2s_config_t i2s_config = {
		.mode = I2S_MODE_MASTER | I2S_MODE_TX | I2S_MODE_DAC_BUILT_IN,
		.sample_rate = out->sampleRate,
		.bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
		.channel_format = I2S_CHANNEL_FMT_RIGHT_LEFT,
		.communication_format = I2S_COMM_FORMAT_STAND_MSB,
		.intr_alloc_flags = 0,
		.dma_buf_count = 2,
		.dma_buf_len = sizeof(out->buffer) / 2,
		.use_apll = false
	};

The audio played is still just a few pops.

The 3rd point in the Problem Description for espressif/esp-idf#8634 suggests that the problem isn't just the format, as the timing of the buffer writes is much faster than expected.

The I2s_write calls return nearly immediately. For example, about 1000 ms of 16-bit 11 kHz mono data is accepted by is2_write in about 50 ms. This suggests that the audio data is not being correctly queued by the driver. When it works correctly, the i2s_write calls are roughly evenly spaced across the one second interval

[meganetaaan]

Digging further.

I found another related issue (espressif/esp-idf#8326) and its fix merged into the release/v4.4 branch. Since moddable uses esp-idf from the released tag (not the release branch), this fix is not yet available.

I tried the latest commit in the release/v4.4 branch at my end. The sound did play, but twice as high and faster than it should.

According to git blame, it appears to be broken again in the commit (espressif/esp-idf@15e8601) that was applied afterwards.

I followed the first issue's fix commit (espressif/esp-idf@19f4eca) and changed clock calculate function to below. And now the correct playback speed is achieved.

#if SOC_I2S_SUPPORTS_ADC || SOC_I2S_SUPPORTS_DAC
/**
 * @brief   I2S calculate clock for built-in ADC/DAC mode
 *
 * @param   i2s_num         I2S device number
 * @param   clk_cfg         Struct to restore clock confiuration
 * @return
 *      - ESP_OK                Get clock success
 *      - ESP_ERR_INVALID_ARG   Invalid args
 */
static esp_err_t i2s_calculate_adc_dac_clock(int i2s_num, i2s_hal_clock_cfg_t *clk_cfg)
{
    ESP_RETURN_ON_FALSE(clk_cfg, ESP_ERR_INVALID_ARG, TAG, "input clk_cfg is NULL");
    ESP_RETURN_ON_FALSE(p_i2s[i2s_num]->hal_cfg.mode & (I2S_MODE_DAC_BUILT_IN | I2S_MODE_ADC_BUILT_IN), ESP_ERR_INVALID_ARG, TAG, "current mode is not built-in ADC/DAC");

    /* Set I2S bit clock */
    clk_cfg->bclk = p_i2s[i2s_num]->hal_cfg.sample_rate * I2S_LL_AD_BCK_FACTOR;
    /* Set I2S bit clock default division */
    clk_cfg->bclk_div = p_i2s[i2s_num]->hal_cfg.chan_bits;
    /* If fixed_mclk and use_apll are set, use fixed_mclk as mclk frequency, otherwise calculate by mclk = sample_rate * multiple */
    clk_cfg->mclk = (p_i2s[i2s_num]->use_apll && p_i2s[i2s_num]->fixed_mclk) ?
                    p_i2s[i2s_num]->fixed_mclk : clk_cfg->bclk * clk_cfg->bclk_div;
    /* Calculate bclk_div = mclk / bclk */
    // clk_cfg->bclk_div = clk_cfg->mclk / clk_cfg->bclk;
    clk_cfg->bclk_div = I2S_LL_AD_BCK_FACTOR * 16;
    /* Get I2S system clock by config source clock */
    clk_cfg->sclk = i2s_config_source_clock(i2s_num, p_i2s[i2s_num]->use_apll, clk_cfg->mclk);
    /* Get I2S master clock rough division, later will calculate the fine division parameters in HAL */
    clk_cfg->mclk_div = clk_cfg->sclk / clk_cfg->mclk;

    /* Check if the configuration is correct */
    ESP_RETURN_ON_FALSE(clk_cfg->mclk <= clk_cfg->sclk, ESP_ERR_INVALID_ARG, TAG, "sample rate is too large");

    return ESP_OK;
}
#endif // SOC_I2S_SUPPORTS_ADC || SOC_I2S_SUPPORTS_DAC

Still I'm not sure it is a degrade on the ESP-IDF side or if it can be corrected with arguments given to i2s_config, etc.

[phoddie]

@meganetaaan – Wow. Very impressive investigation! Thank you. I hope this will be enough to allow @L-KAYA to recommend a solution.

[L-KAYA]

@meganetaaan Thanks a lot for investigating on this issue, and really sorry for the late reply @phoddie , I'm busy on other tasks recently.

I have tested on it, for the issue that twice faster as it sound, it might be caused by the incorrect data format in the sending buffer or the incorrect channel_format.

The format I2S_CHANNEL_FMT_RIGHT_LEFT means stereo mode, for example, data in the sending buffer are 0xAB 0x12 0xCD 0x34, 0xAB and 0xCD will be converted by DAC channel 1 while 0x12 and 0x34 will be converted by DAC channel 2. (referring to the TRM chapter 12.5.3)

In this case, the two channels can transmit different data at the sample rate.

For the case that needs to transmit same data on two channels with the setting sample rate, the channel_format should be set to I2S_CHANNEL_FMT_ONLY_RIGHT or I2S_CHANNEL_FMT_ONLY_LEFT, it will work in mono mode, each byte in the sending buffer will be sent twice, the data signal in the figure above will be 0xAB 0xAB 0x12 0x12 0xCD 0xCD 0x34 0x34

Therefore, setting the channel_format to I2S_CHANNEL_FMT_ONLY_RIGHT or I2S_CHANNEL_FMT_ONLY_LEFT can fix the twice speed. As for the fix @meganetaaan mentioned, the coefficient 16 can only work under I2S_BITS_PER_SAMPLE_16BIT case, it can't meet all the cases.

Summarily, the driver before v4.4 used a lot of fixed configurations while ignoring the configurations given by user, and unfortunately, the ignored configurations in the example is not correct, it is actually a breaking change I think.

[phoddie]

@L-KAYA – Thank you for the additional details. I understand what you are saying and agree that it could explain the 2x speed difference.

However, there is a separate problem to address first -- because I have no audio, not audio at the wrong speed. As @meganetaaan notes, we are using the v4.4 tag. From the Moddable SDK set-up instructions

git clone -b v4.4 --recursive https://github.com/espressif/esp-idf.git

If I am following what @meganetaaan said, I need to switch to some other branch / tag / commit to get a version of I²S support that can work, then I can make the change to deal with mono/stereo correctly.

But... what is the branch / tag / commit to use?

[L-KAYA]

@phoddie You can try to use the following command to clone the latest version

git clone -b release/v4.4 --recursive https://github.com/espressif/esp-idf.git

or just switch to release/v4.4 branch

cd xxx/esp-idf
git checkout release/v4.4
git submodule update
./install.sh
. export.sh

The branch release/v4.4 will keep updating while the tags like v4.4 is fixed to a specific commit of release/v4.4, so the bugfix won't be updated to the tag v4.4.

[phoddie]

We'e committed changes that work on the M5Stack Fire with the release/v4.4 branch.

@meganetaaan, it would be much appreciated if you could confirm that this works for you as well. If it does, we will need to update the Moddable SDK instructions for ESP-IDF set-up, though I'm a little uncomfortable not having a tag to reference (we have had problems before where the ESP-IDF changed in an incompatible way, causing problems for developers before we noticed).

@L-KAYA, thank you very much for your patient help with this!!

[meganetaaan]

I tried the latest commit of esp-idf:release/v4.4 (c29343eb94d2f2ca17b3a5b38c82452e556147f2) and moddable:public (9a760a9).
AudioOut now plays sounds, But a bit higher than actual (say B-flat major -> C major?). This is something we will need to fix a bit further.

out.mp4

Left: the last working commit, Right: current commit

though I'm a little uncomfortable not having a tag to reference

+1. These kind of breaking changes is likely to happen in the future. If tags are not available, I would suggest to rely on a specific commit rather than a branch.

[phoddie]

@meganetaaan – good observation about the pitch difference! It does seem to be high by about two half-steps. I was so happy that sound played at all, I completely overlooked that.

I have two suggestions:

• Update the Moddable SDK to use this commit. While not perfect, it is preferable to have audio play, even if at the wrong pitch, then no audio at all.
• Open a new issue in the ESP-IDF repository to track the pitch shift issue, as the "no audio" problem has been resolved. FWIW – the Moddable SDK provides the sample rate to the I2S driver, but doesn't configure the hardware clock for playback. The audio pitch shift may be due to an imperfect clock calculation in the ESP-IDF.

If you agree, I will move ahead with those. Please let me know.

[meganetaaan]

Agreed with your suggestions 🙆‍♀️. Since we've got the audio playback back, we can close this issue (and open new one at esp-idf repos).
I'm glad I can drive my project forward. Thank you for your patient support!

[phoddie]

Perfect. Thank you.

I'm glad I can drive my project forward.

Me too. I'm looking forward to see what comes next.

[meganetaaan]

I noticed the latest v4.4.1 release includes the fix described above. I think we can switch to this version. It is more reasonable than pointing to a specific commit.