open ai demo panic when esp_srmodel_init

[shushenghong]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

run normally

Actual behavior (suspected bug)

panic

Error logs or terminal output

I (5263) ES7210: Enable ES7210_INPUT_MIC2
I (5293) ES7210: Enable ES7210_INPUT_MIC3
I (5293) ES7210: Enable ES7210_INPUT_MIC4
I (5303) ES7210: Enable TDM mode
I (5313) ES7210: Unmuted
I (5313) Adev_Codec: Open codec device OK
I (5313) MODEL_LOADER: The storage free size is 22016 KB
I (5313) MODEL_LOADER: The partition size is 5168 KB

assert failed: esp_cache_freeze_caches_disable_interrupts esp_cache_utils.c:96 (s_task_stack_is_sane_when_cache_frozen())


Backtrace: 0x4037a02d:0x3c328a40 0x40385631:0x3c328a60 0x4038e275:0x3c328a80 0x403839c4:0x3c328ba0 0x4037cb5f:0x3c328bc0 0x4037cb7f:0x3c328be0 0x4211689e:0x3c328c10 0x42116b7e:0x3c328c60 0x42115f7d:0x3c328ca0 0x420cc28a:0x3c328cc0 0x420cc389:0x3c328cf0 0x4201b7d1:0x3c328d10 0x4201b8c5:0x3c328d30 0x4201aef2:0x3c328d60 0x420c78ab:0x3c328d90 0x420ca9e5:0x3c328dc0 0x420cad80:0x3c328e10 0x40386195:0x3c328e40
--- 0x4037a02d: panic_abort at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/esp_system/panic.c:469
--- 0x40385631: esp_system_abort at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/esp_system/port/esp_system_chip.c:87
--- 0x4038e275: __assert_func at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/newlib/src/assert.c:80
--- 0x403839c4: esp_cache_freeze_caches_disable_interrupts at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/esp_mm/esp_cache_utils.c:96
--- 0x4037cb5f: s_stop_cache at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/esp_mm/esp_mmu_map.c:381
--- 0x4037cb7f: s_do_mapping at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/esp_mm/esp_mmu_map.c:441
--- 0x4211689e: esp_mmu_map at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/esp_mm/esp_mmu_map.c:594
--- 0x42116b7e: spi_flash_mmap at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/spi_flash/flash_mmap.c:88
--- 0x42115f7d: esp_partition_mmap at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/esp_partition/partition_target.c:172
--- 0x420cc28a: srmodel_mmap_init at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__esp-sr/src/model_path.c:330
--- 0x420cc389: esp_srmodel_init at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__esp-sr/src/model_path.c:526
--- 0x4201b7d1: open_afe at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__esp_capture/impl/capture_audio_src/capture_audio_aec_src.c:54
--- 0x4201b8c5: audio_aec_src_start at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__esp_capture/impl/capture_audio_src/capture_audio_aec_src.c:163
--- 0x4201aef2: audio_src_el_open at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__esp_capture/impl/capture_gmf_path/src/elements/gmf_audio_src.c:166
--- 0x420c78ab: esp_gmf_element_process_open at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__gmf_core/src/esp_gmf_element.c:266
--- 0x420ca9e5: process_func at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__gmf_core/src/esp_gmf_task.c:184
--- 0x420cad80: esp_gmf_thread_fun at /Users/ss/Documents/esp-webrtc/solutions/openai_demo/managed_components/espressif__gmf_core/src/esp_gmf_task.c:298
--- 0x40386195: vPortTaskWrapper at /Users/ss/Documents/tools/esp/v5.5/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

Steps to reproduce the behavior

partiion.csv

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 3M,
model,  data, spiffs,         , 5168K

Project release version

main branch

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

MacOS

Operating system version

13.6.7 (22G720)

Shell

ZSH

Additional context

No response

[TempoTian]

Currently run AEC no need to load model. So you do not need to add model partition in partition table.
If you want to use other features, can move the model load and unload call env stack into RAM, the example code default put it to PSRAM.
To workaround for this you can modify:

static void capture_scheduler(const char *name, esp_capture_thread_schedule_cfg_t *schedule_cfg)
{
    media_lib_thread_cfg_t cfg = {
        .stack_size = schedule_cfg->stack_size,
        .priority = schedule_cfg->priority,
        .core_id = schedule_cfg->core_id,
    };
    schedule_cfg->stack_in_ext = true;
    thread_scheduler(name, &cfg);
    // Add following code to put stack in ram
    if (strcmp(name, "aenc_0") == 0 || strcmp(name, "aud_src") == 0 ) {
        schedule_cfg->stack_in_ext = false; 
    }
   // Add ended
    schedule_cfg->stack_size = cfg.stack_size;
    schedule_cfg->priority = cfg.priority;
    schedule_cfg->core_id = cfg.core_id;
}

esp_capture(managed_components/espressif__esp_capture/impl/capture_gmf_path/src/gmf_capture_path_mngr.c) need adjust also for current code put all stack into psram.

static esp_capture_err_t prepare_pipeline(gmf_capture_path_mngr_t *mngr, uint8_t path, gmf_capture_prepare_path_cb prepare_cb)
{
    gmf_capture_path_res_t *res = gmf_capture_path_mngr_get_path(mngr, path);
    uint8_t path_mask = (1 << path);
    if (res->negotiated == false) {
        if (mngr->pipeline_builder->negotiate) {
            int ret = ESP_CAPTURE_ERR_OK;
            CAPTURE_PERF_MON(path, (mngr->stream_type == ESP_CAPTURE_STREAM_TYPE_AUDIO) ? "Negotiate Audio Sink" : "Negotiate Video Sink", {
                ret = mngr->pipeline_builder->negotiate(mngr->pipeline_builder, path_mask);
            });
            if (ret != ESP_CAPTURE_ERR_OK) {
                ESP_LOGE(TAG, "Fail to negotiate audio pipeline");
                return ret;
            }
        }
    }
    for (int i = 0; i < mngr->pipeline_num; i++) {
        if ((mngr->pipeline[i].path_mask & path_mask) == 0) {
            continue;
        }
        // Task already created just add reference count
        if (mngr->task[i]) {
            continue;
        }
        esp_gmf_pipeline_handle_t pipeline = mngr->pipeline[i].pipeline;
        if (capture_pipeline_is_sink(pipeline)) {
            int ret;
            CAPTURE_PERF_MON(path, (mngr->stream_type == ESP_CAPTURE_STREAM_TYPE_AUDIO) ? "Prepare Audio Sink" : "Prepare Video Sink", {
                ret = prepare_cb(res);
            });
            if (ret != ESP_CAPTURE_ERR_OK) {
                return ret;
            }
        }
        esp_gmf_task_cfg_t cfg = DEFAULT_ESP_GMF_TASK_CONFIG();
        esp_capture_thread_scheduler_cb_t scheduler = capture_thread_get_scheduler();
        cfg.thread.stack_in_ext = true;
        if (scheduler && mngr->pipeline[i].name) {
            esp_capture_thread_schedule_cfg_t scheduler_cfg = {
                .priority = CAPTURE_GMF_TASK_DEFAULT_PRIORITY,
                .stack_size = CAPTURE_GMF_TASK_DEFAULT_STACK_SIZE,
            };
            scheduler_cfg.stack_in_ext = true; // Add this line
            scheduler(mngr->pipeline[i].name, &scheduler_cfg);
            cfg.thread.core = scheduler_cfg.core_id;
            cfg.thread.prio = scheduler_cfg.priority;
            cfg.thread.stack = scheduler_cfg.stack_size;
            cfg.thread.stack_in_ext = scheduler_cfg.stack_in_ext; // Add this line
        }
        cfg.name = mngr->pipeline[i].name;
        CAPTURE_PERF_MON(path, (mngr->stream_type == ESP_CAPTURE_STREAM_TYPE_AUDIO) ? "Create Audio Task" : "Create Video Task", {
            esp_gmf_task_init(&cfg, &mngr->task[i]);
        });

[shushenghong]

thank you , i will try later, i see strcmp(name, "aud_src") == 0, is this should be AUD_SRC?

[TempoTian]

Capture is replaced to esp-gmf version and task name is changed to pipeline name, so it is aud_src, when whole path contain only encoder audio source pipeline and process pipeline will merge to one (named aenc0) :
https://github.com/espressif/esp-gmf/blob/main/packages/esp_capture/impl/capture_gmf_path/src/gmf_capture_auto_audio_pipeline.c#L449

[shushenghong]

试过，能用vc模式 + ns降噪，不过有时候会报

E (8131) AUD_AEC_SRC: Fail to read from AEC
E (8131) AUD_SRC: Failed to read audio frame ret -8
E (8141) AUD_SRC: Fail to read data
E (8141) AUD_SRC: Acquire on in port, ret:-1
E (8141) ESP_GMF_TASK: Job failed[tsk:aenc_0-0x3fcd924c:0x3c295414-0x3c203b74-aud_src_proc], ret:-1, st:ESP_GMF_EVENT_STATE_RUNNING
I (8141) AUD_SRC: Audio src thread exited
I (8151) AUD_SRC: Closed, 0x3c203b74
I (8161) AFE: exit

不知道啥原因。

另外，我想问几个问题，
1、我有双麦（esp32s3 + 双外接麦 + es7210，有点类似korvo-2的开发板，只不过我的麦是外接驻极麦），esp_capture里也只能固定设置的单声道+"MR"吗？我看esp-skainet工程的hardware_driver里 korvo-2板子是RMNM

2、可以用波束形成就是se吗？

3、我采到的音量很小，建议用 esp_codec_dev_set_in_gain 设大采集音量还是用agc增强呢？还是说要靠换更灵敏的麦克风？设大采集音量有什么弊端吗？esp_codec_dev_set_in_gain 和 esp_codec_dev_set_in_channel_gain区别是啥？

[TempoTian]

1. 多声道的支持可以看Stereo capture for openai realtime demo #93 (comment)
2. 你说的是BSS吗，BSS是在SE里面打开的
3. esp_codec_dev_set_in_gain 是作用于所有channel的，esp_codec_dev_set_in_channel_gain 可以设置单个channel，适用于单独改变mic 采集音量不改变ref音量的场景。可以优先用esp_codec_dev_set_in_channel_gain提高mic的音量看看

• Read fail的问题之前倒是没遇到过，是开了特定效果才有的吗？我可以对齐配置本地看下

[shushenghong]

多声道我用korvo-2开发板，试了下面代码，不行，回声都没消掉，是不是 "RMNM" 不太对？

而且会报看门狗超时错误

[TempoTian]

我测试了下确实还有bug, 读数据的时候少读了，可以替换为下面的代码到esp_capture(managed_components/espressif__esp_capture/impl/capture_audio_src/capture_audio_aec_src.c)

channel的排布实际是1,3,2,4 对TDM模式来说, 我更新了下：

Actual channel selected	Mic layout	Channel	Channel mask
Ref + Mic1	MR	4	1|2
Ref + Mic2	RM	4	2|4
Ref + Mic1 + Mic2	MRM	4	1|2|4
Ref + Mic1 + Not Care + Mic2	MRMN	4	1|2|4|8

static void audio_aec_src_buffer_in_thread(void *arg)
{
    audio_aec_src_t *src = (audio_aec_src_t *)arg;
    int ch = src->channel;
    if (src->channel_mask) {
        ch = __builtin_popcount(src->channel_mask);
    }
    printf("Has channel %d\n", ch);
    int read_size = src->cache_size * ch;
    uint8_t *feed_data = malloc(read_size);

[shushenghong]

我用1.0.0版本的esp-webrtc，启用sr好像没效果，程序正常跑，但是感觉没有降噪

除了将"pc_task"任务不用psram还要做什么其他的调整吗？

[TempoTian]

bool ns_init; // Whether to init ns 这个要打开下

[shushenghong]

貌似没有这个属性？

esp-sr是1.9.5版本，可能还是更早点的esp-webrtc版本:)

[TempoTian]

目前根据owner提供的信息 只有在使能voice_communication，model加载成功并且正确配置path才有效。

.voice_communication_init = true;

[shushenghong]

设置了的

从打印看应该也是启用了的