QT PY ESP32-S2 core crash when Wi-Fi router is power-cycled

[rdagger]

CircuitPython version

Adafruit CircuitPython 8.0.0-beta.4-21-g8f414eb4e on QT PY ESP32-S2.  

Code/REPL

from secrets import secrets
from time import sleep
import wifi

def connect_to_wifi():
    """Connect to Wi-Fi."""
    print(f'Connecting to {secrets["ssid"]}...')
    # Test Wi-Fi signal strength
    rssi = -255
    for network in wifi.radio.start_scanning_networks():
        if network.ssid == secrets["ssid"] and network.rssi > rssi:
            rssi = network.rssi
    wifi.radio.stop_scanning_networks()
    print(f'Wi-Fi rssi: {rssi} db')

    while True:
        try:
            wifi.radio.connect(secrets["ssid"], secrets["password"])
            print(f'Connected to {secrets["ssid"]}')
            break
        except Exception as wifi_err:
            print(f"Wi-fi connection error: {wifi_err}")
        sleep(1)

connect_to_wifi()

gateway_ip = wifi.radio.ipv4_gateway

while True:
    # Check Wi-Fi is connected
    ping = wifi.radio.ping(gateway_ip, timeout=2)
    if ping is None:
        print("Gateway could not be pinged!")
        connect_to_wifi()
        continue
    print(f"Ping: {ping}") 
    sleep(1)

Behavior

Had 4 QT-PY ESP32-S2's running in my office. They all core crashed when I restarted my Wi-Fi router. Simplified my code and reproduced the problem multiple times.
The code runs until the router is rebooted or powered down. Then it throws the following error:

�]0;�Wi-Fi: off | Done | 8.0.0-beta.4-21-g8f414eb4e�\Auto-reload is off.
Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Crash into the HardFault_Handler.
Please file an issue with the contents of your CIRCUITPY drive at
https://github.com/adafruit/circuitpython/issues

Description

I don't think this is a duplicate of "ping too frequently results in Safe Mode #5980" because I am waiting 1 second between pings and the code doesn't crash from frequent pings. Instead, it crashes while the Wi-Fi is trying to reconnect. Furthermore, I've had a more complicated version of the code running for several days on 4 QT-PY's and only encountered a core crash when the router went down.

Additional information

Removing the RSSI check that uses wifi.radio.start_scanning_networks() causes the crashes less frequently but they can still occur during reconnect. Occasionally, the code will not crash and reconnect properly. I suspect the longer the router is down the more likely the core crash. My router takes over a minute to restart.

[rdagger]

I have done more testing using my 4 QT-PY ESP32-S2's. They will often run for a dozen hours and then all core crash within minutes of each other. I've been checking the memory and there doesn't appear to be any leaks.

I've been trying to come up with a way to reliably reconnect the WiFi when it's disrupted. Pinging the gateway, and then reconnecting on no response, does work most of the time. However, certain types of disconnections seem to always cause a core crash.

All 4 QT PY's are in the same room with direct line of sight to my access point. I don't notice any WiFi issues with any of the other equipment in the house including an older MicroPython ESP32 in my vegetable garden outside which has been running great for over a year.

btw: The docs state: "Reconnections are handled automatically once one connection succeeds." This is not the case with the QT PY ESP32-S2.

I added multiple ping attempts before reconnection to avoid unnecessary reconnects. It seems that many failed pings to the gateway will resolve on a 2nd or 3rd try.

        # Check Wi-Fi is connected
        wifi_verified = False
        for i in range (10):
            if wifi.radio.ping(gateway_ip, timeout=2):
                wifi_verified = True
                break
            else:
                print(f'Gateway could not be pinged! Attempt {i+1} of 10.')
                sleep(3)
        if not wifi_verified:
            connect_to_wifi()
            continue

I also removed the RSSI check prior to WiFi connection because the start_scanning_networks() command was definitely causing more frequent core crashes.

[anecdata]

On espressif port (ESP32-S2), you can call connect() whenever needed with negligible penalty even if the device is already connected, so there's no harm and no need to ping.

The truest test of whether the device is connected is checking for an IPv4 address (wifi.radio.ipv4_address); if there is an IPv4 address, then the device is connected to the AP. This is how it's tracked internally on espressif port.

btw: The docs state: "Reconnections are handled automatically once one connection succeeds." This is not the case with the QT PY ESP32-S2.

There are several automatic reconnection attempts, but not infinite, so connection retries are ultimately needed if there is an extended disconnection.

[rdagger]

@anecdata The crashes occur on wifi.radio.connect(). They don't occur on the pings. That's why I'm trying to minimize the reconnects. Thanks, for the tip on the ipv4_address! I will try setting up one of the QT PY's with an ipv4_check instead of pinging to determine if the WiFi connection has dropped.

[anecdata]

How did you determine that the connect() is triggering the hardfault? I guess it's the print statements, but it is possible some issue takes time to manifest.

btw, espressif boards will automatically connect to the AP with the best signal, among multiple of the same SSID (no need to scan and find best RSSI):

Another thing that need to be considered is that the reconnection may not connect the same AP if there are more than one APs with the same SSID. The reconnection always select current best APs to connect.

https://docs.espressif.com/projects/esp-idf/en/latest/esp32s2/api-guides/wifi.html#wi-fi-reconnect

There was an issue with unstable wifi scanning, it's ostensibly fixed but maybe try an example without pings or scanning just to see if that makes any difference?

[rdagger]

I have relied on the print statements. I have a print statements on the line before and after wifi.radio.connect. When a crash occurs the preceding print statement prints but the subsequent one never prints.

There is only 1 access point in the house.
I was using the RSSI code because I was having signal strength issues with 2 of my QT PY's which have external antennas instead of built-in on the board. The built-in antennae seem to work better although it doesn't matter when everything is in the same room. I did find that WiFi scanning did result in more core crashes. That's why I removed the RSSI check.

I have 2 QT PY's running now using only the ipv4_address to verify the WiFi connection. I also have 2 QT PY's running my previous code slightly modified to reattempt failed pings before reconnecting.

[tannewt]

You can also do a DEBUG=1 build to get a back trace when the crash occurs. There is a backtrace decoder in ports/espressif/tools

[rdagger]

Never tried to build CP. Is there a build guide for the QT PY ESP32-S2 or are there any prebuilt UF2's with debug enabled?

[dhalbert]

@rdagger See https://learn.adafruit.com/building-circuitpython/ and particularly https://learn.adafruit.com/building-circuitpython/espressif-build. If you have or can set up a Linux box, that's generally easiest. The DEBUG=1 build might be too big, and you may need to disable some features. Setting CIRCUITPY_ULAB=0 is often enough, because that's large. We can help you with builds in https://adafru.it/discord in #circuitpython-dev.

[rdagger]

I was able to build with DEBUG=1. For some reason the I2S is not working but I don't really need it to test the WiFi. I will left you know how it goes and thanks for the build help.

[rdagger]

I’m stuck trying to debug the core crash. I tried multiple custom builds of CircuitPython using DEBUG=1 but I haven’t been able to collect any data yet because the core crash takes out the USB serial before any data is displayed. I’ve tried to implement a console uart to catch the debug data but I can’t get it working. The QT PY ESP32-S2 does not have a 'Send ESP_LOG output to TX/RX pins' sample in the sdkconfig file so I copied the section from the QT PY ESP32 Pico sdkconfig. I modified the console uart TX and RX pins to 5 and 16 to match the TX and RX labels on the QT PY ESP32-S2:

# Uncomment (remove ###) to send ESP_LOG output to TX/RX pins
#
# ESP System Settings
#
CONFIG_ESP_SYSTEM_PANIC_PRINT_HALT=y
# CONFIG_ESP_SYSTEM_PANIC_SILENT_REBOOT is not set
CONFIG_ESP_CONSOLE_UART_CUSTOM=y
# CONFIG_ESP_CONSOLE_NONE is not set
CONFIG_ESP_CONSOLE_UART=y
CONFIG_ESP_CONSOLE_UART_CUSTOM_NUM_0=y
# CONFIG_ESP_CONSOLE_UART_CUSTOM_NUM_1 is not set
CONFIG_ESP_CONSOLE_UART_NUM=0
CONFIG_ESP_CONSOLE_UART_TX_GPIO=5
CONFIG_ESP_CONSOLE_UART_RX_GPIO=16
CONFIG_ESP_CONSOLE_UART_BAUDRATE=115200
# CONFIG_ESP_SYSTEM_CHECK_INT_LEVEL_5 is not set
# end of ESP System Settings

Unfortunately, I’m not getting any serial output on either GPIO pin. I hooked the QT PY up to an oscilloscope to verify no communication. Is there something else I have to modify or are my settings wrong?

Btw: switching from a pinging approach to an ipv4_address check for WiFi connectivity has increased the time between core crashes but they still occur.

[tannewt]

That sdkconfig seems to work for me. Here is my build:

firmware.uf2.zip

Part of the output is:

I (1380) cpu_start: Pro cpu start user code
I (1380) cpu_start: cpu freq: 240000000
I (1380) cpu_start: Application information:
I (1385) cpu_start: Project name:     circuitpython
I (1391) cpu_start: App version:      8.0.0-beta.4-194-g2f5ec1cab-dir
I (1398) cpu_start: Compile time:     Nov 30 2022 11:16:18
I (1404) cpu_start: ELF file SHA256:  824bf4d7c9ce4654...
I (1410) cpu_start: ESP-IDF:          716d8531d7