Stack protection fault for ESP32C6 micropython_alpha

[freejygithub]

Recently, I found a MicroPython firmware for the ESP32C6 online, and luckily, I have a development board to try it out. It worked, but I encountered a crash error while testing the WiFi to send data packets. Here is the error I encountered:

[Error message and code provided]

I don’t know which repository this firmware was compiled from; I only know that this bug is easily reproducible and will cause a crash and reboot after running for a few dozen seconds.

Please help me take a look at what might be going on.

`>>> %Run -c $EDITOR_CONTENT

MPY: soft reboot
WiFi..
WiFe~
IP: 192.168.5.6
Sent 1000 times.Failsend 336
Sent 2000 times.Failsend 546
Sent 3000 times.Failsend 824
Sent 4000 times.Failsend 1125
Sent 5000 times.Failsend 1386
Sent 6000 times.Failsend 1647
Sent 7000 times.Failsend 1999
Sent 8000 times.Failsend 2254
Sent 9000 times.Failsend 2572
Guru Meditation Error: Core 0 panic'ed (Stack protection fault).

Detected in task "mp_task" at 0x400313de
Stack pointer: 0x40825dd0
Stack bounds: 0x40822228 - 0x40826220

Core 0 register dump:
MEPC : 0x420179da RA : 0x40803848 SP : 0x40825dd0 GP : 0x408155d4
TP : 0x407ec518 T0 : 0x40022494 T1 : 0x4080fbac T2 : 0x44332201
S0/FP : 0x40834040 S1 : 0x0000000c A0 : 0x4213b290 A1 : 0x4213b464
A2 : 0x00000019 A3 : 0x4083405c A4 : 0x4083405c A5 : 0x4213b000
A6 : 0x00000004 A7 : 0x22116655 S2 : 0x00000000 S3 : 0x00000000
S4 : 0x40828dcc S5 : 0x40828af2 S6 : 0x40825f60 S7 : 0x00000000
S8 : 0x00000050 S9 : 0x42121000 S10 : 0x42121000 S11 : 0x42121000
T3 : 0x66554433 T4 : 0x22116655 T5 : 0x44332211 T6 : 0x66554433
MSTATUS : 0x00001881 MTVEC : 0x40800001 MCAUSE : 0x0000001b MTVAL : 0x00001101
MHARTID : 0x00000000

Stack memory:
40825dd0: 0x40825f60 0x40825f90 0x40825f60 0x421480a0 0x4213db88 0x40825f8c 0x408189f4 0x00000000
40825df0: 0x40825f60 0x00000004 0x40827d00 0x40825f90 0x0000000f 0x40828adc 0x40826008 0x40834040
40825e10: 0x408039f8 0x40834000 0x0000000c 0x00000000 0x00000000 0x40828dcc 0x40828af2 0x40825f60
40825e30: 0x00000000 0x00000050 0x42121000 0x42121000 0x42121000 0x40825dd0 0x00000000 0x42121000
40825e50: 0x42121000 0x42121000 0x00000080 0x408285e0 0x00000030 0x00000032 0x408285f4 0x40825f60
40825e70: 0x00000002 0x00000001 0x00000002 0x4203e4b6 0x40825f60 0x408289dc 0x40825f88 0x00000000
40825e90: 0x00000008 0x4082861c 0x00000000 0xffffffff 0x00000000 0x4082860c 0x40825f60 0x4203c73e
40825eb0: 0x00000008 0x4082861c 0x0000011a 0x42048e7c 0x40825f60 0x40828634 0x0000011a 0x4203abd0
40825ed0: 0x42121000 0x42121000 0x00000080 0x00000000 0x00000007 0x00000006 0x00000005 0xffffffff
40825ef0: 0x00000000 0x40828634 0x40825f60 0x4203ae6a 0x00000000 0x0000002a 0x00000000 0x0000001f
40825f10: 0x40825f60 0x00000007 0x0000001c 0x00000007 0x00000041 0x000000e9 0x40826000 0x42121000
40825f30: 0x42121000 0x42121000 0x00000050 0x00000000 0x40825f60 0x00000007 0x00000000 0x00000000
40825f50: 0x00000000 0x40818a50 0x40825fe0 0x4201085c 0x40827c60 0x40828ecc 0x40825f74 0x00000007
40825f70: 0x40818a50 0x4213aa74 0x40827f00 0x408279a0 0x40834030 0x00003e83 0x00000000 0x00000000
40825f90: 0x40828ed1 0x40825f70 0x00000000 0x4200cdb4 0x4213b7a8 0x00000004 0x40825fe0 0x420107b0
40825fb0: 0x42121000 0x42121000 0x00000080 0x42125000 0x00000041 0x00000005 0x42122000 0x00000004
40825fd0: 0x00000003 0x00000002 0x40827c60 0x4202792c 0x40825fc0 0x00000041 0x40826164 0x00000001
40825ff0: 0x4082615c 0x420277e0 0x42027a3e 0x408372c4 0x40828648 0x40828590 0x00000000 0x00000000
40826010: 0x420278a8 0x00000001 0x00000002 0x00000003 0x00000004 0x42122000 0x00000005 0x00000041
40826030: 0x42125000 0x00000080 0x42121000 0x42121000 0x42121000 0x40825fe0 0x00000000 0x00000000
40826050: 0x420278a8 0x00000000 0x42121100 0x00000001 0x408189f4 0x42007000 0x408272c4 0x00003800
40826070: 0x408372c4 0x42121000 0x42121000 0x42121000 0x42121000 0x40826020 0x4080dd58 0x000007ff
40826090: 0x40818b90 0x40819000 0x40818b90 0xffffffff 0x42007848 0x4200783c 0x40826120 0x408155d4
408260b0: 0x407ec518 0x40027bd4 0x4080dd58 0xffffffff 0x6000f000 0x4081e000 0x00004038 0x00000003
408260d0: 0x00000002 0x00000001 0x00000002 0x00000000 0x00000002 0x00000008 0x40826158 0x00000003
408260f0: 0x0000404c 0x00000005 0x00000041 0x42125000 0x00000080 0x42121000 0x42121000 0x42121000
40826110: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0x00000003 0x40819000 0x42122000 0x00000004
40826130: 0x00000003 0x40826158 0x00000003 0x42007a0a 0x00000003 0x00000002 0x00000001 0x42027b6a
40826150: 0x00003800 0x00000000 0x40010080 0x0080db01 0x00000058 0x4082615c 0x420277e0 0x42027a3e
40826170: 0x00000020 0x00000002 0x40827980 0x40818900 0x00000002 0x4081e000 0x42121000 0x408372c4
40826190: 0x00003800 0x408272c4 0x42007000 0x408189f4 0x00000001 0x4081e000 0x4200743c 0x420075e6
408261b0: 0x00000000 0x408261b0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

ELF file SHA256: ea53b1c36

Rebooting...
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0xc (SW_CPU),boot:0x1c (SPI_FAST_FLASH_BOOT)
Saved PC:0x4001975a
SPIWP:0xee
mode:DIO, clock div:2
load:0x40875720,len:0xed4
load:0x4086c410,len:0xb40
load:0x4086e610,len:0x2b58
entry 0x4086c410
MicroPython a3cf263-dirty on 2024-03-19; ESP32C6 module with ESP32C6
Type "help()" for more information.

`

Uploading micropython_esp32c6_alpha_20240319.zip…

[freejygithub]

Uploading ESP32C6testWIFI6.zip…

[freejygithub]

ESP32C6_test_WIFI6.py

import network
import usocket as socket
import ustruct as struct
from machine import UART
import time
import gc

buf = bytearray(36)
bytess = b'\x01\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66'

WIFI_SSID = 'CNIDEWIFI6'
WIFI_PASSWORD = 'CNIDEWIFI6'

SELF_IP = ''
POSTURE_SERVER_PORT = 8001
POSTURE_LOCAL_PORT = 8001
POSTURE_SERVER_IP = '255.255.255.255'
##########################


errin = "" # 记录发送失败的原因
errsend = 0 # 记录发送失败的次数
count = 0  # 记录发送成功次数
last_count = 0  # 上一次打印时的次数
buf = bytearray(20) # 传感器发来的一条数据

sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)
sta_if.connect(WIFI_SSID, WIFI_PASSWORD)
print('WiFi..')
while not sta_if.isconnected():
  pass
print('WiFe~')

zidong = sta_if.ifconfig()
if SELF_IP:sta_if.ifconfig((SELF_IP,zidong[1],zidong[2],zidong[3]))
print('IP:', sta_if.ifconfig()[0])

posture_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
posture_sock.bind(('0.0.0.0', POSTURE_LOCAL_PORT))

while True:
    if sta_if.isconnected():
        try:
            posture_sock.sendto(b'\x01\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66\x11\x22\x33\x44\x55\x66', (POSTURE_SERVER_IP, POSTURE_SERVER_PORT))
        except OSError:
            errsend += 1
        count += 1
        if count - last_count >= 1000:
            last_count = count
            print(f"Sent {count} times.Failsend {errsend}")
    else:
        time.sleep_ms(1000)

    

[robert-hh]

Please enclose code in line with three backticks ```

[freejygithub]

thanks Get it now , haha~

[freejygithub]

New discovery: Adding a delay can make it last a bit longer, but it still crashes eventually.
But it still crashes here.

Sent 1122000 times.Failsend 60077
Guru Meditation Error: Core  0 panic'ed (Stack protection fault). 

Detected in task "mp_task" at 0x400313de
Stack pointer: 0x40825dd0
Stack bounds: 0x40822228 - 0x40826220


Core  0 register dump:...

[freejygithub]

Tests have found that adding an appropriate delay within the loop can stabilize its operation, but why does it cause a stack overflow and crash when it runs too fast? How can it be made to run quickly without crashing?

[andrewleech]

Thanks for the report and the reproduction script, that helps a lot!
The C6 support is under active development here: #11869
There's quite a lot changed in the IDF for supporting this part so there are a bunch of surprising issues unfortunately.

Oddly though, my build from the latest on that PR fails much worse than yours.... running the steps of your script manually, my C6 seems to just hard-reset as soon as I try to connect wifi at all - regardless of whether a valid ssid/pass is used at all:

>>> errin = "" #
>>> errsend = 0 #
>>> count = 0  #
>>> last_count = 0  #
>>> buf = bytearray(20) #
>>>
>>> sta_if = network.WLAN(network.STA_IF)
>>> sta_if.active(True)
True
>>> sta_if.connect(WIFI_SSID, WIFI_PASSWORD)
>>> ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0xc (SW_CPU),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x4001975a
SPIWP:0xee
mode:DIO, clock div:2
load:0x40875720,len:0xed4
load:0x4086c410,len:0xb68
load:0x4086e610,len:0x2b58
entry 0x4086c410
MicroPython v1.23.0-preview.353.g0f5d549602.dirty on 2024-05-22; ESP32C6 module with ESP32C6
Type "help()" for more information.
>>>

I don't even get a guru meditation to track down...

[freejygithub]

Thank you for the guidance, I have finally found the community. Currently, I am continuously increasing the delay to test stability. As shown in the figure above, it took more than four million sends at a delay of 0.3ms before it crashed. However, there is an obvious pattern that every crash stops at the pointer 0x40825dd0. This led me to suspect a hardware issue, so I plan to buy several development boards to test. This is all I can do at the moment, as I haven’t had time to learn more about the deeper levels of development… If there is anything you need me to assist with, such as testing the latest firmware, etc., please let me know, and I will actively cooperate with you:) By the way, the development board I am using is this one, for your reference: github.com/wuxx/nanoESP32-S3

[freejygithub]

The same script code, without setting any delay, can achieve a maximum receive frequency of up to 3kHz, but it will soon crash at 0x40825dd0. With an appropriate delay, it can sometimes send stablely for up to forty million times (eleven hours) without crashing, as shown in the following diagram.

[freejygithub]

Interestingly, after I switched to another WIFI6 router and set the latency to 0.1ms, it has been stable…

[freejygithub]

我用同样的代码同样的开发板持续进行了三天WIFI6测试, 发现导致崩溃和路由器有关. 虽然不知道是为什么,但目前测试结果说明,实体店购买的中兴AX3000旗舰版将近600元, 会频繁导致崩溃, 而网购的中兴AX3000巡天版159元, 高速不设延迟时也会发生崩溃, 但发送延迟增加到0.2ms时, 会稳如老狗, 由图上可以看出, 总计运行了31万秒, 接收5亿多条, 速率稳定在1700fps左右, 仍然不会发生崩溃.
这个问题到此结束.

[Edit by Robert-hh: Added Linguee translation]
I used the same code and the same development board for three days WIFI6 test, found that the cause of the crash and router related. Although I do not know why, but the current test results show that the brick-and-mortar store to buy the ZTE AX3000 flagship version of nearly 600 yuan, will frequently lead to a crash, while the online purchase of the ZTE AX3000 patrol version of 159 yuan, high-speed without delay will also be crashed, but the delay is increased to 0.2ms when the sender, will be as stable as an old dog, as can be seen from the chart, a total of 310,000 seconds of runtime, to receive 500 million As you can see from the graph, after running for a total of 310,000 seconds and receiving 500 million messages, the rate stabilizes at around 1700fps, and still no crash occurs.
The problem ends here.

[freejygithub]

另外还有一个坑，由于新买的路由器是白色，我放的位置又挨着窗户，为了不让它的外壳弄脏我把内包装的塑料袋子套上了，，，这就导致了它难以散热，导致速率变得比实际低了许多。当我摘了袋子之后速率肉眼可见的涨了一截。
这个坑真是屡错屡犯啊。。。