Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0). (IDFGH-4630)

[zjykymt]

Hi,

Environment

• Module or chip used: ESP32-WROOM-32E
• IDF version: ESP-IDF v4.2-dirty
• SPI Speed: 80MHz
• SPI Mode: DIO
• SPI Flash Size : 8MB
• Watchdog config:
  
• UART ISR:
  

Problem Description

UART1 is connected to a 4G modem at 115200bps, and the communication uses AT commands.

When I use HTTP to download a file (approximately 1MBytes), every 512 bytes after the download is completed, it is stored in the internal flash that in the ESP32-WROOM-32E module. In this case, this Guru Meditation Error happens occasionally.

Please help check the crash log:

Debug Logs

ASSERT_PARAM(-218959118 0), in arch_main.c at line 343
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x40083358  PS      : 0x00060e34  A0      : 0x80129d44  A1      : 0x3ffbe260
0x40083358: r_assert_param at ??:?

A2      : 0x00000001  A3      : 0x00000000  A4      : 0x60008048  A5      : 0x00000000
A6      : 0x00000004  A7      : 0x3ffbdbe4  A8      : 0x80083358  A9      : 0x3ffbe240
A10     : 0x00000000  A11     : 0x00000037  A12     : 0x00000014  A13     : 0xffffffff
A14     : 0x00000000  A15     : 0xfffffffc  SAR     : 0x00000004  EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000  LBEG    : 0x40083290  LEND    : 0x40083297  LCOUNT  : 0x00000000
0x40083290: r_assert_param at ??:?

0x40083297: r_assert_param at ??:?

Core  0 was running in ISR context:
EPC1    : 0x4009692f  EPC2    : 0x4012a16f  EPC3    : 0x00000000  EPC4    : 0x40083358
0x4009692f: uart_hal_write_txfifo at C:/esp/esp-idf-v4.2/components/soc/src/hal/uart_hal_iram.c:35

0x4012a16f: r_rw_schedule at ??:?

0x40083358: r_assert_param at ??:?


Backtrace:0x40083355:0x3ffbe260 0x40129d41:0x3ffbe280 0x40019fb5:0x3ffbe2a0 0x40046683:0x3ffbe2d0 0x40047515:0x3ffbe2f0 0x400864b1:0x3ffbe310 0x40086105:0x3ffbe330 0x400887f2:0x3ffbe350 0x400898e3:0x3ffbe370 0x40082a49:0x3ffbe390 
0x4012a16c:0x3ffb5670 0x4012a711:0x3ffb5690 0x40090a99:0x3ffb56c0
0x40083355: r_assert_param at ??:?

0x40129d41: r_platform_reset at ??:?

0x400864b1: r_lld_evt_end at ??:?

0x40086105: r_lld_evt_end_isr at ??:?

0x400887f2: r_rwble_isr at ??:?

0x400898e3: r_rwbtdm_isr_wrapper at intc.c:?

0x40082a49: _xt_lowint1 at C:/esp/esp-idf-v4.2/components/freertos/xtensa/xtensa_vectors.S:1105

0x4012a16c: r_rw_schedule at ??:?

0x4012a711: btdm_controller_task at ??:?

0x40090a99: vPortTaskWrapper at C:/esp/esp-idf-v4.2/components/freertos/xtensa/port.c:143

Core  1 register dump:
PC      : 0x4000bfe2  PS      : 0x00060034  A0      : 0x80090bb6  A1      : 0x3ffbe910
A2      : 0x00060021  A3      : 0x00060023  A4      : 0x00060021  A5      : 0x00060f23  
A6      : 0x00000001  A7      : 0x00000000  A8      : 0x3ffc6d74  A9      : 0x00000001  
A10     : 0x0000abab  A11     : 0x00000000  A12     : 0x00060f23  A13     : 0x00060f23
A14     : 0x00000001  A15     : 0x00060c23  SAR     : 0x00000000  EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000

Backtrace:0x4000bfdf:0x3ffbe910 0x40090bb3:0x3ffbe920 0x400918ab:0x3ffbe940 0x40090b6b:0x3ffbe960 0x40090e01:0x3ffbe980 0x40082a51:0x3ffbe990 0x40166f83:0x3ffbc4d0 0x400d4aa2:0x3ffbc4f0 0x400914b9:0x3ffbc510 0x40090a99:0x3ffbc530 
0x40090bb3: xPortInIsrContext at C:/esp/esp-idf-v4.2/components/freertos/xtensa/port.c:351

0x400918ab: xTaskIncrementTick at C:/esp/esp-idf-v4.2/components/freertos/tasks.c:2471

0x40090b6b: xPortSysTickHandler at C:/esp/esp-idf-v4.2/components/freertos/xtensa/port.c:300

0x40090e01: _frxt_timer_int at C:/esp/esp-idf-v4.2/components/freertos/xtensa/portasm.S:328

0x40082a51: _xt_lowint1 at C:/esp/esp-idf-v4.2/components/freertos/xtensa/xtensa_vectors.S:1105

0x40166f83: esp_pm_impl_waiti at C:/esp/esp-idf-v4.2/components/esp32/pm_esp32.c:484

0x400d4aa2: esp_vApplicationIdleHook at C:/esp/esp-idf-v4.2/components/esp_common/src/freertos_hooks.c:63

0x400914b9: prvIdleTask at C:/esp/esp-idf-v4.2/components/freertos/tasks.c:3386 (discriminator 1)

0x40090a99: vPortTaskWrapper at C:/esp/esp-idf-v4.2/components/freertos/xtensa/port.c:143



ELF file SHA256: 1fee18bca5575ed4

Rebooting...

[Alvin1Zhang]

Thanks for reporting and sorry for the inconvenience, we will look into.

[patrikhuss]

Hi, we also see this issue also from time to time,

ESP-IDF: 6f3f2e875

ASSERT_PARAM(-218959118 0), in arch_main.c at line 343
Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0).

Core 0 register dump:
PC : 0x400845bc PS : 0x00060634 A0 : 0x80118364 A1 : 0x3ffbe280
A2 : 0x00000001 A3 : 0x00000000 A4 : 0x60008048 A5 : 0x00000000
A6 : 0x00000004 A7 : 0x3ffbdbe4 A8 : 0x800845bc A9 : 0x3ffbe260
A10 : 0x00000000 A11 : 0x00000037 A12 : 0x00000014 A13 : 0xffffffff
A14 : 0x00000000 A15 : 0xfffffffc SAR : 0x00000004 EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000 LBEG : 0x400844f4 LEND : 0x400844fb LCOUNT : 0x00000000
Core 0 was running in ISR context:
EPC1 : 0x401ba237 EPC2 : 0x00000000 EPC3 : 0x00000000 EPC4 : 0x400845bc

Backtrace:0x400845b9:0x3ffbe280 0x40118361:0x3ffbe2a0 0x40019fb5:0x3ffbe2c0 0x40088cba:0x3ffbe2f0 0x4008763d:0x3ffbe330 0x40087359:0x3ffbe350 0x40089a46:0x3ffbe370 0x4008ab37:0x3ffbe390 0x40083b3d:0x3ffbe3b0 0x40118784:0x3ffb56e0 0x40118d2d:0x3ffb5700

0x400845b9: r_assert_param at ??:?

0x40118361: r_platform_reset at ??:?

0x40088cba: r_lld_pdu_rx_handler at ??:?

0x4008763d: r_lld_evt_end at ??:?

0x40087359: r_lld_evt_end_isr at ??:?

0x40089a46: r_rwble_isr at ??:?

0x4008ab37: r_rwbtdm_isr_wrapper at intc.c:?

0x40083b3d: _xt_lowint1 at /home/patrik/Dev/Njord/njord/3rdParty/esp-idf/components/freertos/xtensa/xtensa_vectors.S:1105

0x40118784: r_rw_schedule at ??:?

0x40118d2d: btdm_controller_task at ??:?

Core 1 register dump:
PC : 0x401b687a PS : 0x00060834 A0 : 0x800d61e4 A1 : 0x3ffbc490
A2 : 0x00000000 A3 : 0x00060023 A4 : 0x00060020 A5 : 0x3ffcd8d0
A6 : 0x3ffc7908 A7 : 0x3ffc7914 A8 : 0x800d60d2 A9 : 0x3ffbc460
A10 : 0x00000000 A11 : 0x3ffbedc8 A12 : 0x3ffbedc8 A13 : 0x00000000
A14 : 0x00060020 A15 : 0x00000000 SAR : 0x00000000 EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000

Backtrace:0x401b6877:0x3ffbc490 0x400d61e1:0x3ffbc4b0 0x40092128:0x3ffbc4d0

0x401b6877: esp_pm_impl_waiti at /home/patrik/Dev/Njord/njord/3rdParty/esp-idf/components/esp32/pm_esp32.c:484

0x400d61e1: esp_vApplicationIdleHook at /home/patrik/Dev/Njord/njord/3rdParty/esp-idf/components/esp_common/src/freertos_hooks.c:63

0x40092128: prvIdleTask at /home/patrik/Dev/Njord/njord/3rdParty/esp-idf/components/freertos/tasks.c:3386

ELF file SHA256: b64cec68af151861

Rebooting...

[eriknorth]

Hi,

After extensive testing and bisecting, we might have pin pointed the fault.
It seems that crashing started at BT Controller commit:
espressif/esp32-bt-lib@85be5e8
This led us to check flags related to sleep modes. After disabling CONFIG_BTDM_MODEM_SLEEP, we were not able to replicate the crash anymore.

It is unfortunate that BT Lib is closed source which makes debugging issues like this very time consuming. And even now after finding the issue we are not sure why this flag exactly fixes the issue.

Please, whoever is experiencing this issue, check if this flag also mitigates/fixes the crash.

/Erik

[ZweiEuro]

Could this be the same issue as in:
#4413 (comment)
?

adding
CONFIG_BTDM_MODEM_SLEEP=n
to sdkconfig.defaults did not help

[eriknorth]

The crash seems similar but the cause might be different as the task watchdog triggers first which might indicate that you are starving the task. I have not seen your code but I would guess that you might be executing long commands within Bluetooth callbacks or Timer callbacks which should be short as possible. Sadly BT controller is closed source but I think it is extensively using FreeRTOS timers, so if you block the timer, it will block some functions within the BT controller and it might crash because of that.

[ZweiEuro]

My code is pretty much just a slightly modified BT example, with no long executions in BT.
I am just using Wifi and a homekit implementation https://github.com/maximkulkin/esp-homekit
Along side it.
Do you know of any other error source i could search for ?
Or maybe there is some memry i am not supposed to touch when BT is aktive

[eriknorth]

Are always getting first the task watchdog and then it crashes with interrupt WDT or it crashes instantly with interrupt WDT?

[ZweiEuro]

The WDT triggers a few times and then on the 3rd or 4th it crashes for good.

[xiewenxiang]

This is because there is no available memory in the controller. Can you provide a demo that can be reproduce this issue? @zjykymt

[alexmantaut]

I think this issue #6517 is also related to this issue.

There might be a change in the esp-bt-lib addressing this issue. Can you check this comment?

[zjykymt]

Hi, my test results are as follows (ESP-IDF 4.2):
When CONFIG_BTDM_MODEM_SLEEP is in the enabled state, there are 2 crash within 4 times;
When CONFIG_BTDM_MODEM_SLEEP is in the disabled state, there is no problem for 10 consecutive times;

[alexmantaut]

hi @zjykymt

We had the same problem, keep in mind that disabling CONFIG_BTDM_MODEM_SLEEP increases the power consumption significantly (on our board current was about 100 mA)

Can you check if you get the issue with the version on master with CONFIG_BTDM_MODEM_SLEEP enabled?

[zjykymt]

hi @alexmantaut

I will try to test on the master branch in these two days.

[zjykymt]

hi @alexmantaut

I used the esp-idf(v4.4) on the master branch for testing.

When CONFIG_BTDM_CTRL_MODEM_SLEEP is enabled, there is no problem.

[zjykymt]

Release 4.2.1 and 4.3 has solved this problem, I closed this ISSUE.

ESP-IDF Release v4.2.1

Classic Bluetooth
Fixed bug of modem sleep which may lead to the crash issue "assert(-218959118,0)"