Stereo capture for openai realtime demo

[diegoasua]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate.
• Described the feature in detail and justified the reason for the request.
• Provided specific use cases and examples.

Feature description

Korvo boards have dual mic and currently we are not taking advantage of the second mic as it only uses 1 mic + AEC reference in TDM. As far as I can tell the i2s path can take up to 4 channels in TDM, only 2 used and 2 masked. We could use one of the two available channels for the second mic to do beamforming and improve capture from further away. Reading about esp-sr AFE should support this to a point as it already can do dual-channel BSS:

Supports dual-channel processing, which can well separate the target sound source from the rest of the interference sound, so as to extract the useful audio signal and ensure the quality of the subsequent speech.

How difficult would it be to enable a second channel over TDM? What changes would it need on the capture side (codec init, TDM setup, AFE changes etc)?

Related, OpenAI realtime seems to be capturing stereo Opus 48kHz, would it be possible to send both channels as stereo? Or else, beamform into mono and let esp-peer upsample to stereo 48kHz as it does now.

Use cases

Better capture from far away or with surrounding noise.

Alternatives

Single mic (already implemented)

Additional context

No response

[TempoTian]

Yes, it is easy to modify esp_capture_audio_aec_src_cfg_t to support audio capture from dual mics. However, currently it does not support outputting 2-channel audio. The esp-sr module supports 1, 2, or 3 mics + 1 reference input, with a single-channel output for better performance.

For TDM mode, the actual I2S channel data order is: 1,3,4,2

For Korvo2, the mic connections are: channel 1 = Mic1, channel 2 = Mic2, channel 3 = Reference, channel 4 = unconnected.

The new esp_capture adds a mic_layout field, where each character indicates the channel function:

• R: Reference
• M: Mic data
• N: Not-care channel

Below is a simple table for Korvo2 mic layout configuration:

Actual channel selected	Mic layout	Channel	Channel mask
Ref + Mic1	MR	4	1|2
Ref + Mic2	RM	4	2|4
Ref + Mic1 + Mic2	MRM	4	1|2|4
Ref + Mic1 + Not Care + Mic2	MRMN	4	1|2|4|8

typedef struct {
    const char *mic_layout;     /*!< Mic data layout, e.g. "MR", "RMNM"" */
    void       *record_handle;  /*!< Record handle of `esp_codec_dev` */
    uint8_t     channel;        /*!< Audio channel */
    uint8_t     channel_mask;   /*!< Bit mask to select which channels to process
                                      (e.g. 0x1 for left channel, 0x2 for right channel) */
} esp_capture_audio_aec_src_cfg_t;

[diegoasua]

To be clear, you are saying it would not be possible to run custom beamforming inside the AFE? Only BSS that comes out-of-the-box with esp-sr no custom DSP functions? I understand it always needs to output mono.

[TempoTian]

Currently esp-sr not support customized algrithom. But you can combine esp-sr and other algrithom, processing data in chain then write a capture audio source interface to wrapper for it. It is a easy way I think, you can copy aec source out and change it as demand. https://github.com/espressif/esp-gmf/blob/main/packages/esp_capture/impl/capture_audio_src/capture_audio_aec_src.c

static void audio_aec_src_buffer_in_thread(void *arg)
{
    audio_aec_src_t *src = (audio_aec_src_t *)arg;
    int read_size = src->cache_size * 2;
    uint8_t *feed_data = malloc(read_size);
    if (feed_data) {
        while (!src->stopping) {
            int ret = esp_codec_dev_read(src->handle, feed_data, read_size);
            if (ret != 0) {
                ESP_LOGE(TAG, "Fail to read data %d", ret);
                break;
            }
            // This is the place where get data from I2S, can process and then feed to AFE
            ret = src->afe_handle->feed(src->afe_data, (int16_t *)feed_data);
            if (ret < 0) {
                ESP_LOGE(TAG, "Fail to feed data %d", ret);
                break;
            }
        }
        capture_free(feed_data);
    }
    src->in_quit = true;
    ESP_LOGI(TAG, "Buffer in exited");
    capture_thread_destroy(NULL);
}

static esp_capture_err_t audio_aec_src_read_frame(esp_capture_audio_src_if_t *h, esp_capture_stream_frame_t *frame)
{
    audio_aec_src_t *src = (audio_aec_src_t *)h;
    if (src->start == false) {
        return ESP_CAPTURE_ERR_NOT_SUPPORTED;
    }
    frame->pts = (uint32_t)(src->samples * 1000 / src->info.sample_rate);

    int need_size = frame->size;
    uint8_t *frame_data = frame->data;
    while (need_size > 0) {
        if (src->cached_read_pos < src->cache_fill) {
            int left = src->cache_fill - src->cached_read_pos;
            if (left > need_size) {
                left = need_size;
            }
            memcpy(frame_data, src->cached_frame + src->cached_read_pos, left);
            src->cached_read_pos += left;
            need_size -= left;
            frame_data += left;
            continue;
        }
        if (src->in_quit) {
            return ESP_CAPTURE_ERR_INTERNAL;
        }
        src->cache_fill = 0;
        src->cached_read_pos = 0;
        afe_fetch_result_t *res = src->afe_handle->fetch(src->afe_data);
        if (res->ret_value != ESP_OK) {
            ESP_LOGE(TAG, "Fail to read from AEC");
            return ESP_CAPTURE_ERR_INTERNAL;
        }
        if (res->data_size <= src->cache_size * 2) {
            memcpy(src->cached_frame, res->data, res->data_size);
            src->cache_fill = res->data_size;
        } else {
            ESP_LOGE(TAG, "Why so huge %d", res->data_size);
        }
    }
   // Add code here to process data after AEC
    src->samples += frame->size / 2;
    return ESP_CAPTURE_ERR_OK;
}

[diegoasua]

made it work with 2 mics and BSS. Then noticed all the pipeline runs at 16kHz for capture and playback. Is there a way to run at 48kHz? If I am correct there are some blobs inside esp-sr that are not open source and limit the whole pipeline to 16kHz is my intuition correct? if so does that mean AFE only runs at 16kHz, can't run 44.1 or 48?

[TempoTian]

Yes， AFE currently only support 16k to have better performance. For voice communication 16k is enough, some even run in 8k. If support 48k, need add a resample to resample it down to 16k to make it work. If you not use AFE, you can configure the capture as your demand easily, there is no such limitation if CPU is enough.