Safemode entered because of WDT trigger instead of resetting the ESP32

[jfabernathy]

CircuitPython version

CP 9.2.3 on a Adafruit ESP32 Huzzah

Code/REPL

# The version is for the ESP32-S2 Feather with AHTx0 connected to onboard
# STEMMA-QT connector.
#
import os
import adafruit_connection_manager
import socketpool
import ssl
import adafruit_minimqtt.adafruit_minimqtt as MQTT
import wifi
import time
import board
import digitalio
import alarm
import busio
import adafruit_ahtx0

PUBLISH_DELAY = 60
MQTT_TOPIC = "environmentTopic"
USE_DEEP_SLEEP = True

from microcontroller import watchdog as w
from watchdog import WatchDogMode
w.timeout = 70  # Set a timeout of 70 seconds
w.mode = WatchDogMode.RESET
w.feed()


def alarm_deepSleep(how_long):
    time_alarm = alarm.time.TimeAlarm(monotonic_time=time.monotonic() + how_long)
    alarm.exit_and_deep_sleep_until_alarms(time_alarm)


def debug_log(msg):
    print(msg)


def post_data(msg):
    try:
        mqtt_client.connect()
        mqtt_client.publish(MQTT_TOPIC, msg)
        mqtt_client.disconnect()
    except Exception as e:
        print("Exception in post_data ", str(e))
        alarm_deepSleep(2)

# Create sensor objects, using the board's default I2C bus.
i2c = board.I2C()

sensor = adafruit_ahtx0.AHTx0(i2c)

print(f"My MAC address: {[hex(i) for i in wifi.radio.mac_address]}")
print(f"Connecting to {os.getenv('CIRCUITPY_WIFI_SSID')}")
wifi.radio.connect(os.getenv("CIRCUITPY_WIFI_SSID"), os.getenv("CIRCUITPY_WIFI_PASSWORD"))
print(f"Connected to {os.getenv('CIRCUITPY_WIFI_SSID')}")
print(f"My IP address: {wifi.radio.ipv4_address}")

mqtt_username = os.getenv("mqtt_name")
mqtt_password = os.getenv("mqtt_password")

# Create a socket pool
pool = adafruit_connection_manager.get_radio_socketpool(wifi.radio)
ssl_context = adafruit_connection_manager.get_radio_ssl_context(wifi.radio)

# Set up a MiniMQTT Client
mqtt_client = MQTT.MQTT(
    broker=os.getenv('mqtt_broker'),
    port=os.getenv('mqtt_port'),
    username=mqtt_username,
    password=mqtt_password,
    socket_pool=pool,
    ssl_context=ssl_context,
)

x = round(sensor.temperature * 1.8 + 32.0, 1)
y = round(sensor.relative_humidity, 1)
temp = str(x)
humid = str(y)

loop_count = alarm.sleep_memory[5] | alarm.sleep_memory[6] << 8
loop_count += 1
alarm.sleep_memory[6] = loop_count >> 8
alarm.sleep_memory[5] = loop_count & 255

station = '"' + os.getenv("location") + '"'

# format the readings data as  InfluxDB expects: Array of 2 object:
# 1st is readings, 2nd is tag

data = (
    """[{"Temperature": """
    + temp
    + """ , "Humidity": """
    + humid
    + """,  "Loop Count": """
    + str(loop_count)
    + """}, {"Location": """
    + station
    + """}]"""
)
post_data(data)
debug_log(data)

debug_log("Just posted data, now into deep sleep!\n\n")

w.feed()

# Create an alarm that will trigger 60 seconds from now to awake from deep sleep
alarm_deepSleep(60)

Behavior

The code runs and deep sleeps for 60 seconds. There is a WDT set to 70 seconds to reset the device if triggered. I noticed that when the board was running just connected to a power adapter that after some hours of running I would find the red LED flashing slowly. I connected the ESP32 Hazzah Feather to a Linux PC running Thonny and when I'd see the red LED flashing, I see on Thonny the following.

rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:396
ho 0 tail 12 room 4
load:0x40078000,len:13892
load:0x40080400,len:4
load:0x40080404,len:3160
entry 0x40080558
Serial console setup

Auto-reload is off.
Running in safe mode! Not running saved code.

You are in safe mode because:
Internal watchdog timer expired.
Press reset to exit safe mode.

I do not have this problem with the same code on a ESP32-s2 Feather. However, I don't know that I've ever trigger the WDT reset on the EPS32-S2. I have a loop counter in Static RAM that gets incremented. It only gets reset when the microcontroller is reset and not coming out of Deep Sleep.

Description

No response

Additional information

No response

[jepler]

I think that watchdog timeout is intended to cause a safe mode reset, though I'd welcome correction if I'm wrong.

Even if that's the case, it looks like the documentation could be improved here, clarifying what kind of reset the watchdog produces.

By putting code in safemode.py, you can control what happens after various types of safe mode resets:

Safe mode may be handled programmatically by providing a safemode.py. safemode.py is run if the board has reset due to entering safe mode, unless the safe mode initiated by the user by pressing button(s). USB is not available so nothing can be printed. safemode.py can determine why the safe mode occurred using supervisor.runtime.safe_mode_reason, and take appropriate action. For instance, if a hard crash occurred, safemode.py may do a microcontroller.reset() to automatically restart despite the crash. If the battery is low, but is being charged, safemode.py may put the board in deep sleep for a while. Or it may simply reset, and have code.py check the voltage and do the sleep."

[jfabernathy]

I really don't know if the WDT code I'm using is really working. It's hard to test if connected to Mu or Thonny. That coupled to the fact that if the WiFi connects correctly you should never have a WDT event. Back in CP 7.2 days the WiFi had issues with fringe located ESP32-S2s where I'd see my loop counter go to zero and I'd know a WDT reset had occurred. However since CP 9.2.1, I update all my wifi connection logic to the new method and now I've never had a WDT event on a ESP32-S2.

This new test is with a ESP32 Hazzah with CP. Mainly it's to see how CP works on the ESP32. But I had an old Hazzah on a proto board with a AHT20 solder down. It had been running Arduino doing the same tasks as my EPS32-S2 did with CP but poorly. I'm really not an Arduino guy. So I'd like to use this as a CP remote senor board. You can never have too many megabytes of temperature and humidity data :-)

[jfabernathy]

I think that watchdog timeout is intended to cause a safe mode reset, though I'd welcome correction if I'm wrong.

It seems to me that based on my testing on the ESP32-S2 back on CP 7.2 that the WDT did what it said, either RAISE or RESET. A reset always yanked the chain on the reset pin and you got a clean start up. Hence the loop counter in SRAM cleared.

I could see the Safemode being a REPL feature the way that Deep Sleep is simulated in REPL.

[anecdata]

With full deep sleep unconnected to USB, you could try writing to nvm from safemode.py and boot.py to capture the reasons for safemode and reset (example).

[jfabernathy]

Okay, I've been digging through the safemode documenation and on the Learn article I see an interesting statement.
https://learn.adafruit.com/circuitpython-safe-mode/safe-mode-reasons

SafeModeReason.WATCHDOG
An internal watchdog timer went off. Something is looping or taking more time than expected. This is not the watchdog timer used for microcontroller.watchdog. If microcontroller.watchdog is enabled to do a reset, it will do a regular reset; it will not go into safe mode.

So this document implies that what I saw as normal behavior in the past should still work. i.e. if you do:

from microcontroller import watchdog as w
from watchdog import WatchDogMode
w.timeout = 70  # Set a timeout of 70 seconds
w.mode = WatchDogMode.**RESET**
w.feed()

And it times out, it will yank reset and no safe mode. That seems like a bug, but all I can do for testing is put in the safemode.py and see if I can force the reset that way after a WDT happens.

[dhalbert]

And it times out, it will yank reset and no safe mode. That seems like a bug, but all I can do for testing is put in the safemode.py and see if I can force the reset that way after a WDT happens.

As described in the doc, watchdog timers are not related to internal watchdog errors. watchdog was implemented before safemode.py. You could use WatchDogMode.RAISE (https://docs.circuitpython.org/en/latest/shared-bindings/watchdog/index.html#watchdog.WatchDogMode.RAISE) to raise an exception if you want to do something without having to have a reset happen.

A feature we could add is, say, a WatchDogMode.SAFEMODE, which would be in addition to .RESET, instead of changing the behavior of .RESET.

[jfabernathy]

So it sounds like this case is really a bug that the WatchDogMode.RESET is entering safemode and not resetting the ESP32. My current test is to enable the watchdog timer again and have the savemode.py to simply do a reset. So if my program is still working in the morning and my loop counter has reset then I know the workaround is working. Then it's up to you experts on how you want to fix it.

[dhalbert]

So it sounds like this case is really a bug that the WatchDogMode.RESET is entering safemode and not resetting the ESP32.

I thought you were saying your test above that explicitly does a 70 second watchdog is not entering safemode. Maybe I'm confused. I want to emphasize that the word "watchdog" means completely different things here. SafeModeReason.WATCHDOG is for internal watchdogs that have nothing to do with wdt.

You can check for a wdt watchdog reset by looking at microcontroller.cpu.reset_reason. See https://docs.circuitpython.org/en/latest/shared-bindings/microcontroller/index.html#microcontroller.ResetReason.WATCHDOG : one of the ResetReasons is WATCHDOG, which means a wdt watchdog.

[jfabernathy]

Until today, I was not aware of safemode. I used the code:

from microcontroller import watchdog as w
from watchdog import WatchDogMode
w.timeout = 70  # Set a timeout of 70 seconds
w.mode = WatchDogMode.RESET
w.feed()

to force what I thought was a real reset like a power up reset. I think it works on the ESP32-S2, but not on the ESP32. On CP 7.2 it helped me out a lot. But on CP 9.2 I've never had a WDT trigger that I know of on a ESP32-S2. I could test with or without the REPL working with Mu. As long as I didn't try to mess with the timer after it was set, I had no issues.

I recently tried CP 9.2.3 on a ESP32 Hazzuh and used the exact same code as my EPS32-S2. I just change the settings.toml to make the sensor unique. I kept find it stuck and flashing Red in the morning. Hooking up Thonny to the ESP32 showed it was in Safemode. I tried a test removing the WDT code and that ran a long time without triggering anything. So now my test is to trap the WDT Safemode I'm seeing: in the REPL
rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
See the rest of the output about safe mode above in Behavior in the first post.

[jfabernathy]

And you are right the loop with deep sleep is 60 second and the WDT timeout is 70 sec.

[dhalbert]

Safe mode is after a real reset, but it's not a "clean reset", in the sense that it that something happened that usually was some kind of error. So the reason is checked and available.

If you want to force a clean reset after a safemode watchdog, you can put something like this in safemode.py:

import microcontroller
microcontroller.reset()

(You can actually use that anywhere, not just in safemode.py.)

Have you looked at https://learn.adafruit.com/circuitpython-safe-mode ? I think maybe yes.

[jfabernathy]

So this is a change since CP 7.2. I'm testing the version of safemode.py you showed above now. If it works overnight connected to Thonny, then I'll test just connected to USB Power.

I guess I should implement the safemode.py on even my ESP32-S2 that don't seem to even need a WDT. The CP 9.2 wifi connect that changed, to me, is a positive and that is why my ESP32.S2 run for 45 days before the loop count rolls over to 0 instead of ever resetting by WDT.