[TW#12670] Panic handler no longer functioning as expected

[MrSurly]

Using older versions of the IDF (e.g. f73c6f8) with CONFIG_ESP32_PANIC_PRINT_REBOOT, I would get useful backtraces, like this:

Guru Meditation Error of type LoadProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x400dd086  PS      : 0x00060830  A0      : 0x800dd0b2  A1      : 0x3ffe85e0  
A2      : 0x3ffd99fc  A3      : 0x3f440265  A4      : 0x3ffe8660  A5      : 0x3ffd99a4  
A6      : 0x00000000  A7      : 0x00000000  A8      : 0x800dd086  A9      : 0x3ffe85c0  
A10     : 0x00000000  A11     : 0x00000004  A12     : 0x00000001  A13     : 0x00000080  
A14     : 0x00000001  A15     : 0x00000000  SAR     : 0x00000004  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x0000000c  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffb  

Backtrace: 0x400dd086:0x3ffe85e0 0x400dd0b2:0x3ffe8610 0x400dc40e:0x3ffe8630 0x400f1632:0x3ffe86b0 0x40105325:0x3ffe86e0

Now, (1e0710f) I get:

Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
... <several more> ...
<reboot>

Even when CONFIG_ESP32_PANIC_GDBSTUB is set, the same behavior is observed.

[projectgus]

Hi @MrSurly,

The "cache disabled but cached memory region accessed" fatal exception is new. Older versions of IDF wouldn't crash on this, they'd just read garbage from the unmapped cache and go off into the weeds (possibly resulting in a LoadStoreException like the one in your first example, but possibly also crashing in other weird and hard-to-decipher ways.)

Can you please post the full output from the current crash you're seeing?

Angus

[MrSurly]

@projectgus

The full dump is below. But let me give you some more context, since I think it add a lot of value.

The code in question is testing the BLE implementation in MicroPython. The code reads a config file from flash (if it exists), then tries to connect to WiFi. If there's no config, or it can't connect, it advertises on BLE, and awaits WiFi credentials from another device.

When it gets the credentials, it writes the config file to flash. This is where it crashes. If I don't write to flash, everything works fine.

I'm very sure it crashes in the middle of a flash operation, because after the reboot, my main.py file is gone. I suspect it is trying to write the config file to the same flash block (or more likely same FAT block), and died sometime after the erase cycle.

However, without a better backtrace, it's hard to know where exactly.

E (13254) BT: MTU request PDU with MTU size 100

E (13254) BT: BTM_SetBleDataLength failed, peer does not support request
E (13254) BT: Call back not found for application conn_id=4
E (13254) BT: gatt_disc_cmpl_cback() - Register for service changed indication failure
network_bluetooth_gatts_event_handler(event = 04 / MTU, if = 03, param = (conn_id = 0000, mtu = 0064)
network_bluetooth_gatts_event_handler(event = 02 / WRITE, if = 03, param = (conn_id = 0000, trans_id = 00000001, addr = 5C:F3:70:7D:8A:D6, handle = 002A, offset = 0000, need_resp = false, is_prep = false, len = 002D, data = <redacted>  )
Callback
got b'{"password": "<redacted>", "ssid": "<redacted>"}'
before config json
after config json
config is now {"password": "<readacted>", "ssid": "<redacted>"}
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error of type IllegalInstruction occurred on core  0. Exception was unhandled.
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)led.
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached mem�ets Jun  8 2016 00:22:57

rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0008,len:8
load:0x3fff0010,len:3416
load:0x40078000,len:9492
load:0x40080000,len:252
entry 0x40080034
W (265) boot: PRO CPU has been reset by WDT.
W (265) boot: WDT reset info: PRO CPU PC=0x400877b2
W (265) boot: WDT reset info: APP CPU PC=0x6c74bbf9
I (2108) cpu_start: Pro cpu up.
I (2108) cpu_start: Single core mode
I (2110) heap_alloc_caps: Initializing. RAM available for dynamic allocation:
I (2122) heap_alloc_caps: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (2143) heap_alloc_caps: At 3FFB3000 len 00005000 (20 KiB): DRAM
I (2164) heap_alloc_caps: At 3FFBBB28 len 00002000 (8 KiB): DRAM
I (2184) heap_alloc_caps: At 3FFDF7F0 len 00000810 (2 KiB): DRAM
I (2205) heap_alloc_caps: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (2226) heap_alloc_caps: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (2248) heap_alloc_caps: At 4009ABF8 len 00005408 (21 KiB): IRAM
I (2269) cpu_start: Pro cpu start user code
I (2328) cpu_start: Starting scheduler on PRO CPU.
OSError: [Errno 2] ENOENT
OSError: [Errno 2] ENOENT
MicroPython v1.8.6-1157-g5da175f-dirty on 2017-05-16; ESP32 module with ESP32
Type "help()" for more information.
>>> ets Jun  8 2016 00:22:57

[igrr]

Could you please check what's at 0x400877b2?

[MrSurly]

Could you please check what's at 0x400877b2?

Yes, but I'd need to re-compile my older code. I can do that if it still necessary, but:

I tried again today with some updates from the mainline micropython-esp32. Now it doesn't crash.

I suspect some of of the changes made by @dpgeorge when merging in micropython/micropython-esp32#89, particularly to the LD script, might have fixed this issue.

Did you want me to go back and try to re-create using my older code?

[MrSurly]

Appears this was probably a linking error.