ESP32 crash after jag run

[muhlpachr]

I have got ESP32 crash after ~5th jag update after jag flash (just simple 4 line hello word).
I am not able to reproduce it.

Guru Meditation Error: Core  1 panic'ed (StoreProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x400862a0  PS      : 0x00060a30  A0      : 0x801053dc  A1      : 0x3ffcadf0
A2      : 0x0000008d  A3      : 0x00000000  A4      : 0x00000000  A5      : 0x00000000
A6      : 0x3ffd489c  A7      : 0x3ffc8308  A8      : 0x3f430e86  A9      : 0x3f41225c
A10     : 0x3f41225c  A11     : 0x00000000  A12     : 0x00000155  A13     : 0x3f41225c
A14     : 0x3f41225c  A15     : 0x3f41225c  SAR     : 0x0000000c  EXCCAUSE: 0x0000001d
EXCVADDR: 0x0000008d  LBEG    : 0x400818d1  LEND    : 0x400818d9  LCOUNT  : 0x00000027

Backtrace:0x4008629d:0x3ffcadf0 0x401053d9:0x3ffcae70 0x4011315f:0x3ffcaea0 0x4011334e:0x3ffcaef0 0x40113381:0x3ffcaf20 0x4010701e:0x3ffcaf40 0x40107045:0x3ffcaf60


ELF file SHA256: 49bc775c76ae2181

Rebooting...

[muhlpachr]

jag version
Version:         v1.0.3
SDK version:     v1.6.6
Build date:      2022-02-10T16:35:08Z

[muhlpachr]

I am not sure, weather is is related, I got another after jag run:

Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.

Core  0 register dump:
PC      : 0x6b636f6c  PS      : 0x00060330  A0      : 0x801053dc  A1      : 0x3ffc9df0
A2      : 0x3f412658  A3      : 0x3f41225c  A4      : 0x6b636f6c  A5      : 0xffffffff
A6      : 0x3ffd48cc  A7      : 0x3ffc80c8  A8      : 0x3f431dea  A9      : 0x3ffc9dd0
A10     : 0x3ffc80c8  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x00000000
A14     : 0x3ffc7fcc  A15     : 0x3ffc7f78  SAR     : 0x0000000c  EXCCAUSE: 0x00000000
EXCVADDR: 0x00000000  LBEG    : 0x400818d1  LEND    : 0x400818d9  LCOUNT  : 0x00000027

Backtrace:0x6b636f69:0x3ffc9df0 |<-CORRUPTED


ELF file SHA256: 49bc775c76ae2181

Rebooting...

[muhlpachr]

Another:

----
Decode system message with:
----
jag decode --system WyNVBVVYU1UGdjEuNi42U1UAWyRVI1UQAAAAAAAAAAAAAAAAAAAAAFsjVQRVRVNVCUVYQ0VQVElPTlNVEkNvbm5lY3Rpb24gcmVmdXNlZFsjVQJVU1sjVRpbI1UDVUZVAEkxFVsjVQNVRlUBSUIyWyNVA1VGVQJJSqlbI1UDVUZVA0lJXlsjVQNVRlUESV5jWyNVA1VGVQVJCftbI1UDVUZVBklgD1sjVQNVRlUHSRW7WyNVA1VGVQhJFdBbI1UDVUZVCUkVrVsjVQNVRlUKSWBdWyNVA1VGVQtJXwBbI1UDVUZVDEkVu1sjVQNVRlUNSRXQWyNVA1VGVQ5JFVVbI1UDVUZVD0lfLVsjVQNVRlUQSV+OWyNVA1VGVRFJJBBbI1UDVUZVEklkmVsjVQNVRlUTSRW7WyNVA1VGVRRJFdBbI1UDVUZVFUkVVVsjVQNVRlUWSWSsWyNVA1VGVRdJZMFbI1UDVUZVGEkjtVsjVQNVRlUZSSQp

EXCEPTION error.
Connection refused
  0: tcp_write_                <sdk>/net/modules/tcp.toit:183:3
  1: TcpSocket.write           <sdk>/net/modules/tcp.toit:147:14
  2: Writer.write              <sdk>/writer.toit:37:23
  3: ChunkedWriter.write       <pkg:pkg-http>/chunked.toit:73:13
  4: ResponseWriter_.write     <pkg:pkg-http>/server.toit:114:18
  5: serve_incoming_requests.<lambda> /home/runner/work/jaguar/jaguar/src/jaguar.toit:184:14
  6: Server.run_connection_.<block> <pkg:pkg-http>/server.toit:85:17
  7: catch.<block>             <sdk>/core/exceptions.toit:114:10
  8: catch                     <sdk>/core/exceptions.toit:112:1
  9: catch                     <sdk>/core/exceptions.toit:87:10
 10: Server.run_connection_    <pkg:pkg-http>/server.toit:84:7
 11: Server.listen.<lambda>.<block>.<block> <pkg:pkg-http>/server.toit:66:24
 12: catch.<block>             <sdk>/core/exceptions.toit:114:10
 13: catch                     <sdk>/core/exceptions.toit:112:1
 14: catch                     <sdk>/core/exceptions.toit:63:10
 15: Server.listen.<lambda>.<block> <pkg:pkg-http>/server.toit:65:16
 16: Server.listen.<lambda>    <pkg:pkg-http>/server.toit:52:24

[muhlpachr]

It seams jag monitor hangs sometimes on MSW:

PS C:\Users\michal\test> jag monitor -a
Starting serial monitor of port 'COM11' ...
o world!
158 Hello world!
159 Hello world!
160 Hello world!
1

Serial port seams to be fine, when I connect with terminal emulator or from Python, it continues without problem and do not hang.

Just for reference used toit code is this simple:

main:
  i := 1
  while true:
    print "$i Hello world!"
    i++
    sleep --ms=500

[muhlpachr]

It seams there are more issues, now with another jag run hello.toit there are two apps running on esp32 (there was not any change in source code):

67 Hello world!
68 Hello world!
69 Hello world!
70 Hello world!
71 Hello world!
72 Hello world!
73 Hello world!
74 Hello world!
[jaguar] INFO: program 2 starting from flash @ [0,24576]
1 Hello world!
75 Hello world!
2 Hello world!
76 Hello world!
3 Hello world!
77 Hello world!
4 Hello world!
78 Hello world!
5 Hello world!
79 Hello world!

[muhlpachr]

May be issues are related to ESP32-PICO-D4, I have it on https://docs.m5stack.com/en/core/atom_lite

[kasperl]

I think some of these can be explained by toitlang/jaguar#124. Thanks for reporting these, Michal!

[kasperl]

@muhlpachr Maybe give Jaguar v1.0.5 a spin? I don't think we've resolved all these issues just yet, but hopefully we've made a bit of progress.

[kasperl]

We believe the corruption issues may have been fixed by toitlang/jaguar@81eac4d. Closing, but feel free to reopen if you continue to run into issues, @muhlpachr.

If the flash is indeed corrupt in the nvs partition, it might make sense to erase the flash of your device before trying again. Use:

$HOME/.cache/jaguar/esptool --chip esp32 erase_flash

[muhlpachr]

Some issues are solved by erase_flash and v1.0.5, but double application running survives repetily in this sequence:

1. jag monitor
2. jag run app
3. exit monitor
4. jag monitor - will reboot esp32 and start app installed by jag run app in step 2.
5. jag run app - will start another app together with app from first jag run in step 2. or crash
   Example of crash:

Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.

Core  0 register dump:
PC      : 0x6b636f6c  PS      : 0x00060f30  A0      : 0x80106620  A1      : 0x3ffc9df0
A2      : 0x3f41262c  A3      : 0x00000000  A4      : 0x6b636f6c  A5      : 0x3f412230
A6      : 0x3ffd48bc  A7      : 0x3ffc80c8  A8      : 0x3f431dca  A9      : 0x3ffc9dd0
A10     : 0x3ffc80c8  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x00000000
A14     : 0x3ffc7fcc  A15     : 0x3f412230  SAR     : 0x0000000c  EXCCAUSE: 0x00000000
EXCVADDR: 0x00000000  LBEG    : 0x400818d1  LEND    : 0x400818d9  LCOUNT  : 0x00000027

Backtrace:0x6b636f69:0x3ffc9df0 |<-CORRUPTED

ELF file SHA256: b03f22a339ead692

Rebooting...

[floitsch]

We are investigating, but the current theory is that we start erasing the program while the old application is still running.
At some point the program reads the erased data which makes it crash.

[kasperl]

We were definitely overwriting programs that were started at boot time while they were still running. Fixed in toitlang/jaguar@f940471. Fix will be released in Jaguar v1.0.7.

[muhlpachr]

Fixed. Thank you!

[kasperl]

Thank you for reporting back, @muhlpachr!