[TW#23300] Guru Meditation Error: Core 1 panic'ed (IllegalInstruction) . Why this Exception is happening ? (IDFGH-11023)

[jegajith]

Is there any way we can TRAP all exception and decide to Not to reset the ESP
We have some use case , which cannot reset in between.

I (35132) APP_DEBUG: Time left for entire regen (Recharge.Time_Left)= 6823
I (35142) APP_DEBUG: Valve.Error_Timer = 3258
I (35152) APP_DEBUG: Valve.Longest = 3840 Valve.Next_Longest = 3840 Valve.Long_Position = 0
I (35162) APP_DEBUG: END*
I (36122) APP_DEBUG: System Power Up, Days: 0 and Time: 12:00:34
I (36122) APP_DEBUG: START(In Regen)*
I (36122) APP_DEBUG: Valve.Position is (SERVICE ) Valve.Req_Position (SERVICE )
I (36132) APP_DEBUG: Time left in current regen positon (Recharge.Position_Timer) = 0
I (36132) APP_DEBUG: Time left for entire regen (Recharge.Time_Left)= 6823
I (36142) APP_DEBUG: Valve.Error_Timer = 3241
I (36152) APP_DEBUG: Valve.Longest = 3840 Valve.Next_Longest = 3840 Valve.Long_Position = 0
I (36162) APP_DEBUG: END*
Guru Meditation Error: Core 1 panic'ed (IllegalInstruction)
. Exception was unhandled.
Core 1 register dump:
PC : 0x401434e1 PS : 0x00060e30 A0 : 0x800d71af A1 : 0x3ffdcf20
A2 : 0x00000000 A3 : 0x00000000 A4 : 0x00060e20 A5 : 0x00000001
A6 : 0x00060e20 A7 : 0x00060023 A8 : 0x800d956c A9 : 0x3ffdcf10
A10 : 0x00000001 A11 : 0x000000e8 A12 : 0x3ffddcd4 A13 : 0x3ffdcf20
A14 : 0x00000000 A15 : 0x3ffddcd4 SAR : 0x00000003 EXCCAUSE: 0x00000000
EXCVADDR: 0x00000000 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff

Backtrace: 0x401434e1:0x3ffdcf20 0x400d71ac:0x3ffdcf50 0x400d7285:0x3ffdcf70 0x400df56f:0x3ffdcf90 0x400d9559:0x3ffdcfb0

Rebooting...
I (30) boot: ESP-IDF v3.1-dev-1171-g358c822d-dirty 2nd stage bootloader
I (30) boot: compile time 12:23:40
I (31) boot: Enabling RNG early entropy source...
I (36) boot: SPI Speed : 40MHz
I (40) boot: SPI Mode : DIO
I (44) boot: SPI Flash Size : 16MB
I (48) boot: Partition Table:

[Dazza0]

An Illegal Instruction Exception occurs when the CPU runs across an invalid assembly instruction.
This usually occurs for the following reasons

1. The CPU attempts to read an instruction from IRAM at an address in which never contained an instruction to begin with, leading to the CPU interpreting the contents at the unused address as an invalid instruction..
2. The CPU attempts to read an instruction which is not currently accessible (such as reading an instruction from flash cache whilst the flash cache is disabled)

If your application does not utilize SPI Flash, I suspect the illegal instruction exception was due to the first reason, and probably caused by returning from a FreeRTOS task.
Could you provide more details with regards to what the application is doing, and also double check that every FreeRTOS task within the application terminates properly via a call to vTaskDelete()

[igrr]

Also please use idf monitor, it will decode the backtrace, telling where the crash is happening.

[jegajith]

@Dazza0
I will see your point and get back to you.

[jegajith]

@Dazza0 Is there any possibility these exceptions are due to free RTOS Internals ? . When we do some very basic testing this Exception never happens , But when we have multiple task getting this issue , Not so often once in a while , So we are scared to release the code to customer. we cannot accept abrupt reset of system.

[jegajith]

@igrr we will see the back trace and analyse it Thank you very much for your response.

[Dazza0]

@jegajith I think it's unlikely to be due to FreeRTOS. But the only way to really tell is to look at the back trace to see what code/function is running at address 0x400d71ac before it jumps to 0x401434e1 .

[jegajith]

@Dazza0 The bad part is that its not coming every time , So I am waiting for next guru meditation exception to do a back trace .. Thanks for your reply 👍

[Dazza0]

@jegajith fyi, if you still have the .elf file of the build from the log output above, you can use GDB to find the function at an address. You'll need to navigate to the build directory of your project, then open the elf file using GDB

cd C:/Users/UserName/Desktop/test-project/build
xtensa-esp32-elf-gdb test-project.elf

and then use the command
info symbol 0x401434e1
or whichever address you want to query.

[jegajith]

Hi All ,
I am curious to know is there any exception handler where we can capture this exception and do accordingly.
@Dazza0 I am able to track it where this is happening. But every time this exception happens in different functions.

[igrr]

Could you please post a couple of examples of stack traces?
Also please fill in the fields of the issue template you have not provided: IDF version (commit ID), hardware being used, environment (IDF/Arduino/etc).

[jegajith]

@igrr I will collect back trace log and share you . Waiting for the next exception :-)
IDF -Version 3 , Hardware ESP32 Custom Board , IDE ( Note Pad ++ and Windows tool chain)

[jegajith]

@igrr

Please find the Log and Back Trace

I (10546402) APP_DEBUG: JSON - MQTT TOPIC PUBLISHED SUCCESSFULLY
I (10547352) APP_DEBUG: System Power Up, Days: 0 and Time: 19:45:49
I (10558352) APP_DEBUG: New Time is available:
I (10558352) APP_DEBUG: *******************************************
I (10558352) APP_DEBUG: The current is: Tue Jun 19 19:46:00 2018
I (10558352) APP_DEBUG: sync_with_UTC_time(601) Time not updated, Day of system and SNTP is same,time diff between them is only : 0 seconds. Device time not updated
I (10559342) APP_DEBUG: esphal_getAdc1Voltage(291) *voltage = 709
I (10562042) APP_DEBUG: Initializing SNTP
I (10562342) APP_DEBUG: System Power Up, Days: 0 and Time: 19:46:04
Guru Meditation Error: Core 1 panic'ed (LoadProhibited)
. Exception was unhandled.
Register dump:
PC : 0x4011049e PS : 0x00060930 A0 : 0x801197b2 A1 : 0x3ffdbca0
A2 : 0x00000000 A3 : 0x00000000 A4 : 0x8008abbf A5 : 0x3ffdbc20
A6 : 0x00000003 A7 : 0x00060023 A8 : 0x0fbf2222 A9 : 0x00000000
A10 : 0x3ffdbca0 A11 : 0x3f40018c A12 : 0x00000040 A13 : 0x3ffdbce0
A14 : 0x5d5c5a59 A15 : 0x6462605f SAR : 0x00000003 EXCCAUSE: 0x0000001c
EXCVADDR: 0x0fbf2222 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0x00000000

Backtrace: 0x4011049e:0x3ffdbca0 0x401197af:0x3ffdbd10 0x40112bed:0x3ffdbd30

Rebooting...
I (10) boot: ESP-IDF v3.0 2nd stage bootloader
I (10) boot: compile time 15:03:37
I (10) boot: Enabling RNG early entropy source...
I (13) boot: SPI Speed : 40MHz
I (17) boot: SPI Mode : DIO
I (21) boot: SPI Flash Size : 16MB
I (25) boot: Partition Table:
I (29) boot: ## Label Usage Type ST Offset Length
I (36) boot: 0 phy_init RF data 01 01 00009000 00001000
I (44) boot: 1 aws_config unknown 41 00 0000a000 00001000
I (51) boot: 2 aws_cert unknown 40 00 0000b000 00002000
I (59) boot: 3 tlc_bin unknown 42 00 0000d000 00002000
I (66) boot: 4 nvs WiFi data 01 02 0000f000 00009000
I (73) boot: 5 model_bin unknown 43 00 0001d000 00001800
I (81) boot: 6 valve_bin unknown 44 00 0001e800 00000300
I (88) boot: 7 prog_bin unknown 45 00 0001eb00 00000100
I (96) boot: 8 factory factory app 00 00 00020000 00150000
I (103) boot: End of partition table

[Dazza0]

There's some sort of memory corruption going on. The error this time around is a "LoadProhibited" error which occurs when the CPU attempts to write/read from in invalid address which in this case is 0x0fbf2222. This looks like a pointer that was overwritten to an invalid address due to memory corruption (i.e. stack overflow or out of bounds array access). I suggest you take check what variables/pointers are being accessed at led.c:285.

Also might be a good idea to enable stack overflow canary to check if the memory corruption is indeed due to a stack overflow.

[jegajith]

@Dazza0 Thanks a lot I will check and update you

[jegajith]

I am closing this issue . If any one one has any idea to Write a general handler to take care this exception it will be great.
We have done a lot of clean up in code. Exception frequency has come down.
Thank you very much for your support . Happy Coding !

[urbalke]

One possibility could be missing return in function body. In my case I had function returning struct but I forgot to return it and it crashed with this exact log. :))

[igrr]

Just a note, we have added a line about this in the programming guide in 31310d4. (See also #8244.)