HardFault_Handler triggered by pressing buttons too quickly

[raquo]

CircuitPython version

Adafruit CircuitPython 7.1.1 on 2022-01-14; Adafruit Feather M4 Express with samd51j19
Board ID:feather_m4_express

Code/REPL

# --- code.py ---
import time

import board
import digitalio
from analogio import AnalogIn

from src.common import Clock
from src.common.IntervalTimer import IntervalTimer

ledPin = digitalio.DigitalInOut(board.LED)
ledPin.direction = digitalio.Direction.OUTPUT

time.sleep(1)

ledTimer = IntervalTimer(1000)

debugTimer = IntervalTimer(10)

initialMs = Clock.initialTime()

pin = AnalogIn(board.A1)

while True:
    now = Clock.now()

    if debugTimer.checkIntervalPassed(now):
        print((0, 2 ** 16 - 1, pin.value))

    if ledTimer.checkIntervalPassed(now):
        ledPin.value = not ledPin.value




# --- src/common/Clock.py ---
#  Copyright (c) Nikita Gazarov 2021.

import supervisor


THROTTLE_INTERVAL_MS = 20

__TICKS_PERIOD = 1 << 29

__TICKS_HALF_PERIOD = __TICKS_PERIOD // 2

__TICKS_MAX = __TICKS_PERIOD - 1

__INITIAL_MS: int = supervisor.ticks_ms()


def initialTime() -> int:
    return __INITIAL_MS


def now() -> int:
    return supervisor.ticks_ms()


def timeSinceInitial(nowTime = None) -> int:
    if nowTime is None:
        nowTime = supervisor.ticks_ms()
    return timeDiff(nowTime, __INITIAL_MS)


def timeDiff(ticks1, ticks2):
    diff = (ticks1 - ticks2) & __TICKS_MAX
    diff = ((diff + __TICKS_HALF_PERIOD) & __TICKS_MAX) - __TICKS_HALF_PERIOD
    return diff




# --- src/common/IntervalTimer.py ---
#  Copyright (c) Nikita Gazarov 2021.

import supervisor

from src.common import Clock


class IntervalTimer:

    intervalMs = 0

    lastResetMillis = supervisor.ticks_ms()

    def __init__(self, ms, initialMs = None):
        self.intervalMs = ms
        self.lastResetMillis = supervisor.ticks_ms() if initialMs is None else initialMs

    def resetMillis(self, newMs):
        self.lastResetMillis = newMs

    def checkIntervalPassed(self, currentMs):
        passed = Clock.timeDiff(currentMs, self.lastResetMillis + self.intervalMs) > 0
        if passed:
            self.resetMillis(currentMs)
        return passed

Behavior

I have three buttons connected to the board, on pins A1, A2, and D12. Each has an external 10K pullup resistor near the board. It's pulled up to the 3V pin, not the BAT pin.

The code works fine in principle, except when I start mashing the buttons like a madman. That is, pressing and releasing them very quickly, as fast as I can, maybe 10 times per second or so? If I do this long enough, all hell breaks loose.

• Sometimes I get HardFault_Handler and the board goes into safe mode
• Other times I get "impossible" python exceptions, for example:

Traceback (most recent call last):
  File "code.py", line 26, in <module>
NameError: name 'debugTimer' is not defined

or

Traceback (most recent call last):
  File "code.py", line 29, in <module>
TypeError: keywords must be strings

Those exceptions are for the exact code that I pasted. As you see from the code, they shouldn't happen (and they don't, until this problem is somehow triggered). This is really weird, there has to be some kind of corruption going on, but I don't know what kind.

PS I don't care about my copyright for this code, I left it in just to make sure the files are verbatim.

Description

Sometimes it takes me a minute or more of button mashing to trigger the crash or exception, other times it takes just a few seconds. Mashing two buttons together seems to trigger it more reliably than mashing a single button.

That said, I originally noticed this issue because I was testing my triple-press detection code, so I was pressing one button only 3-5 times in quick succession, nothing too crazy.

Additional information

• The buttons are of similar form factor (example), but are of two different models from different suppliers

• The buttons have different connections to the board – one is on a breadboard with short jumper wires, other two are connected in a longer chain, like this: board->headers->PCB->JST XH->3 foot wires->button. The pullups for these two buttons are on the PCB.

  • Note: I got the breadboard button to cause this failure on its own, with nothing else connected to the board – no PCB, nothing on other pins, just the USB for power and serial logging

• The board is powered by the computer's USB. I am looking at serial output with the Mu Editor 1.7.1 on MacOS.

• Each button has an external 10K pullup resistor near the board. It's pulled up to the 3V pin, not the BAT pin. There are no capacitors on those circuits.

• I did check that the CIRCUITPY volume actually has the code that I showed here, and not something else.

• I checked that the buttons and pull-ups produce the expected voltages on the pins using a multimeter (while pressing the buttons slowly, the multimeter isn't fast enough to show useful values while button mashing) – 3.25V when idle, 0.00V when pressed down.

• I closed the Mu editor, then from the code I shared, I replaced the print(...) statement with pass to make sure the issue wasn't caused by printing somehow overflowing the serial somehow, but it still failed with HardFault_Handler after a few seconds of mashing buttons (confirmed a few times).

• I tried rebooting the board many times (no effect)

• I tried reformatting the CIRCUITPY drive several times using storage.erase_filesystem() in safe mode (no effect)

• I tried upgrading the board's Circuitpython from 7.0.0 to 7.1.1 (no effect)

I am at a complete loss with this issue, I can't imagine how pressing buttons quickly can result in circuitpython breaking so weirdly and so completely. I'm not sure if it's a hardware or a software issue... either way it seems like this shouldn't happen.

Sadly I expect that this might be hard to reproduce, so I would really appreciate any pointers on what this could be, or how to debug this further.

[figbash]

This is happening to me as well, running CircuitPython 7.2.5 also on a Feather M4 Express. Even just running the simple button example on Adafruit's site will cause it to crash.

For me it either only happens with the D12 pin, or it's just more easily reproduced on that pin. Using a 100 ohm resistor on the + wire from the button to the pin seems to fix it. I am using the internal pull up resistor.

[ladyada]

@figbash2 can you post the minimal example that demos this?

[figbash]

As an example, this code to try the button before using debounce:

https://learn.adafruit.com/debouncer-library-python-circuitpython-buttons-sensors

import board
import digitalio

pin = digitalio.DigitalInOut(board.D12)
pin.direction = digitalio.Direction.INPUT
pin.pull = digitalio.Pull.UP

button_state = False
while True:
    pressed = pin.value
    if pressed != button_state:
        print(pressed)
    button_state = pressed

[raquo]

For the record, I now use an external pulldown resistor (10K) on the button, and also another 1K resistor positioned in series with the button's signal pin, i.e. between the button's signal pin and the board's D12 pin, physically close to the board. So far I was unable to trigger a hard fault using this setup (using my code above). I'm not sure why the series resistor seemingly fixes it (the pulldown alone doesn't).

[dhalbert]

The resistor will tend to remove spiky noise on an input pin, because it lengthens the RC time constant on the pin, where C is the capacitance of the wiring and the input pin capacitance.

If I understand correctly, both of you are seeing this issue on D12 only (which is PA22)? I did some searching on this, and don't see specific complaints about noise on PA22.

The SAMD51 can get upset when pin voltages exceed VCC. I think this is an electrical noise problem, and doesn't have to do with anything CircuitPython is doing. It might be interesting to try to reproduce this on the same pin and wiring with an Arduino program.

[raquo]

For me D12 is just the pin where I can most reliably reproduce the issue. I managed to rarely and sporadically reproduce it on other digital pins too... I think D6, D9 and D10, but not 100% sure exactly which ones, it's been a while. It's quite a bit harder to reproduce on pins that aren't D12 for some reason. It also seemed easier to reproduce with a pullup resistor vs a pulldown, but not sure if that's a true pattern, or just randomness. I don't recall if I ever managed to reproduce this on analog pins, but if I did, it was also rare and sporadic.

By the way, one other fun consequence of triggering this issue for me is that my circuitpython code gets semi-broken somehow after the hard crash. The filesystem is not corrupted – I can see that the content of my python files is intact. But the code that used to detect button clicks and print to console when it happens, simply doesn't print anything while other unrelated parts of the code continue to work as usual. When this happens, rebooting, re-uploading the files, the firmware, erasing the filesystem – none of that helps usually. This weird condition seems to go away on its own after I leave the board disconnected for some time (e.g. a few minutes).

The SAMD51 can get upset when pin voltages exceed VCC. I think this is an electrical noise problem, and doesn't have to do with anything CircuitPython is doing.

This does seem logical, and I'll probably end up incorporating this series resistor that I'm using in my project long term. I've also heard of TVS diodes that are supposed to prevent over-voltage, but I haven't tried them myself.

[dhalbert]

Closing because this appears to be a hardware problem.