CircuitPython core code crashed hard. Whoops!

[devansau]

CircuitPython version

adafruit CircuitPython 8.0.3 on 2023-02-23; FeatherS2 with ESP32S2

Code/REPL

#
#   Motor
#
#   connects to WIFI and MQTT broker as specified in the screts file
#   main loop gets sensor  data from MQTT
#
# imports for the board
import gc
import board
import adafruit_dotstar
import time

# imports for WIFI and MQTT
import ssl
import socketpool
import wifi
import adafruit_minimqtt.adafruit_minimqtt as MQTT

# impport the oparameters for WIFI and MQTT
from secrets import secrets

# import for motor pin control
import digitalio

# status dictipnary denition
SensorData = {
    'TiltCommand': "STOP",
    'TiltDelta': 0,
    'PanCommand': "STOP",
    'PanDelta': 0,
}


# create a DotStar instance
dotstar = adafruit_dotstar.DotStar(
    board.APA102_SCK, board.APA102_MOSI, 1, brightness=0.5, auto_write=True)

# create the I2C bus instance
i2c = board.I2C()

# setup motor control pins
# Pan motor is Pins D12 and D9
# Tilt motor is Pins D15 and D19
PanClockWise = digitalio.DigitalInOut(board.D12)
PanClockWise.direction = digitalio.Direction.OUTPUT
PanAntiClockwise = digitalio.DigitalInOut(board.D9)
PanAntiClockwise.direction = digitalio.Direction.OUTPUT
TiltClockWise = digitalio.DigitalInOut(board.D19)
TiltClockWise.direction = digitalio.Direction.OUTPUT
TiltAntiClockwise = digitalio.DigitalInOut(board.D15)
TiltAntiClockwise.direction = digitalio.Direction.OUTPUT
PanClockWise.value = False
PanAntiClockwise.value = False
TiltClockWise.value = False
TiltAntiClockwise.value = False
print(PanClockWise)
print(PanAntiClockwise)
print(TiltClockWise)
print(TiltAntiClockwise)

# start the garbage collector
gc.collect()

# Network connection
print("Connecting to ", secrets["ssid"])
wifi.radio.connect(secrets["ssid"], secrets["password"])
print("Connected to ", secrets["ssid"])

pool = socketpool.SocketPool(wifi.radio)  # chnaged to POOL
print("my IP addr:", wifi.radio.ipv4_address)

# init variabes
mqtt_data_in = "SolarDirection/data"  # get Pan's position


def rulesEngine(TiltCommand, PanCommand):
    # PanCommand to motor driver instruction
    if PanCommand == "CLOCKWISE":
        PanAntiClockwise.value = False  # false state first 30 April 23
        PanClockWise.value = True
    elif PanCommand == "ANTI-CLOCKWISE":
        PanClockWise.value = False
        PanAntiClockwise.value = True
    else:
        PanClockWise.value = False
        PanAntiClockwise.value = False

    # TiltCommand to motor driver instruction
    if TiltCommand == "UP":
        TiltAntiClockwise.value = False  # false state first 30 April 23
        TiltClockWise.value = True
    elif TiltCommand == "DOWN":
        TiltClockWise.value = False
        TiltAntiClockwise.value = True
    else:
        TiltClockWise.value = False
        TiltAntiClockwise.value = False
    time.sleep(0.02)
    print(SensorData)


def log(client, userdata, level, buf):
    print('log: ', buf)


def connect(mqtt_client, userdata, flags, rc):
    print("Connected to MQTT Broker!")
    print("Flags: {0}\n RC: {1}".format(flags, rc))
    mqtt_client.subscribe(mqtt_data_in)


def disconnect(mqtt_client, userdata, rc):
    mqtt_client.subscribe(mqtt_data_in)
    print("Disconnected from MQTT Broker!")


def subscribe(mqtt_client, userdata, topic, granted_qos):
    print("Subscribed to {0} with QOS level {1}".format(topic, granted_qos))


def unsubscribe(mqtt_client, userdata, topic, pid):
    print("Unsubscribed from {0} with PID {1}".format(topic, pid))


def publish(mqtt_client, userdata, topic, pid):
    # print("Published to {0} with PID {1}".format(topic, pid))
    pass


def message(client, topic, message):
    # Method callled when a client's subscribed feed has a new value.
    # print("New message on topic {0}: {1}".format(topic, message))
    global SensorData
    SensorData = eval(message)
    TiltCommand = SensorData['TiltCommand']
    PanCommand = SensorData['PanCommand']
    print("Tilt : ", TiltCommand)
    print("Pan : ", PanCommand)
    rulesEngine(TiltCommand, PanCommand)


# Create a socket pool - dulticated above
# pool = socketpool.SocketPool(wifi.radio)


# Set up a MiniMQTT Client
mqtt_client = MQTT.MQTT(
    broker=secrets["broker"],
    port=secrets["port"],
    username=secrets["broker_username"],
    password=secrets["broker_password"],
    client_id="PanDrive1",
    socket_pool=pool,
    ssl_context=ssl.create_default_context(),
)

# Connect callback handlers to mqtt_client
mqtt_client.on_connect = connect
mqtt_client.on_disconnect = disconnect
mqtt_client.on_subscribe = subscribe
mqtt_client.on_unsubscribe = unsubscribe
mqtt_client.on_publish = publish
mqtt_client.on_message = message
mqtt_client.on_log = log

print("Attempting to connect to %s" % mqtt_client.broker)

mqtt_client.connect(clean_session=True, keep_alive=None)

while True:

    try:
        mqtt_client.loop()
    except Exception as error:
        print("Failed to get data from mqtt loop, retrying\n", error)
        mqtt_client.connect()
        continue

Behavior

Auto-reload is off.
Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Crash into the HardFault_Handler.
Please file an issue with the contents of your CIRCUITPY drive at
https://github.com/adafruit/circuitpython/issues

Press any key to enter the REPL. Use CTRL-D to reload.

Adafruit CircuitPython 8.0.3 on 2023-02-23; FeatherS2 with ESP32S2

Description

There is not enough info i the crash to say any more.
Pins D12, D9, D19 and D15 are driving a relay board.
The program runs for hours and then a hard fault on circuitython.

Additional information

I have had this running on a MAC serial port and it shows the hard fault.
I have had it running initiallyy on batteries with no console and it stops and I assume it is with the same hard fault.
Any fault finding suggestions would be appreciated.

[DJDevon3]

adafruit CircuitPython 8.0.3 on 2023-02-23; FeatherS2 with ESP32S2

Please update your UM FeatherS2 to Circuit Python 8.0.5 stable or newer release and try again. Most of the hard fault issues dealing with Wifi & MQTT have been addressed in 8.0.5 as beta tested in issue #7907 If your project still hard faults after upgrading then we will need another round of beta testing with your code.

[devansau]

Thanks DJDevon.
I upgraded to 8.0.5 but the same hard crash

CircuitPython core code crashed hard. Whoops!
Crash into the HardFault_Handler.

The UM FeatherS2 had been running for 2.5 hours, at the time of the crash the MQTT topic was static so no changes being made to the GPIOs. Just monitoring in the MQTT loop.

Archive.zip

[dhalbert]

Could you also try with "Absolute Newest", linked from the board download page on CircuitPython.org?

[DJDevon3]

I actually have a pan/tilt servo kit. Can help test this on latest beta release sometime this week on the UM Feather S2 with motor featherwing.

The UM FeatherS2 has a special featherS2.py that is created upon initial install with many pre-defined functions. I'm not sure if they're required or just helpers but in all of my uses I've always included it in my code.py

def enable_LDO2(state):
    """Set the power for the second on-board LDO to allow no current draw when not needed."""
    ldo2.value = state
    # A small delay to let the IO change state
    time.sleep(0.035)

import feathers2
# Make sure the 2nd LDO is turned on
feathers2.enable_LDO2(True)
# Turn on the internal blue LED
feathers2.led_set(True)

Unlike the Adafruit Feather ESP32-S2 the UM FeatherS2 has a 2nd LDO that should be enabled for some things. I haven't used it specifically yet, I think it's for helping low power consumption by automatically shutting off the 2nd LDO during deep sleep. If there is an issue with sleep commands possible hard faults might occur? Needs investigation.

I hammered on MMQTT and I2C on the UM Feather S2 last week with over 25,000 transactions over 72 hours. The code I used is now an example in my Github for the UM FeatherS2. This hard fault has to be coming from something else that I didn't specifically test for last week. Will look into it with you.

[devansau]

Thanks, I will add the LD02 code and have a look at your example.
I will test over the weekend and update here.

[devansau]

Could you also try with "Absolute Newest", linked from the board download page on CircuitPython.org?

Hi Dan, yes I will try the newest over the weekend as well. Thanks

[devansau]

Using the latest "Adafruit CircuitPython 8.1.0-beta.2 on 2023-04-26; FeatherS2 with ESP32S2" and the code has the "feathers2.enable_LDO2(True)" the UMFeatherS2 has run for 10 hours without crashing. Using 2 UMFeatherS2 I have sent 140,000 MQTT messages from one to the other via a raspberry pi running mosquito. One way only and neither have crashed.
Thanks for your advise and look froward to the new 8.1 release.
I guess the remaining question is; should I try the test agains without the "feathers2.enable_LDO2(True)"?

[DJDevon3]

@devansau That's great news! If you could test it without the LDO2(True) to intentionally try to get it to fail that would definitely help.

[devansau]

@DJDevon3 @dhalbert All good with the LDO2(True) removed.
I ran the code over 2 days, 10 hours at a time, >100k MQTT messages each day and all good on CircuitPython 8.1.0-beta.2
Thanks for your help.

@DJDevon3 I have used the adafruit motor featherwings and it works well. I recently upgraded the motors and smoked 2 motor featherwings. I think it was the back EMF so I have reverted to my own board which deals with this.