I2sOut on UM BLING causes hard fault (8.2.9)

[scogswell]

CircuitPython version

Adafruit CircuitPython 8.2.9 on 2023-12-06; BLING! with ESP32S3

Code/REPL

# SPDX-FileCopyrightText: 2023 Kattni Rembor for Adafruit Industries
# SPDX-License-Identifier: MIT
"""
CircuitPython I2S Tone playback example.
"""
import time
import array
import math
import audiocore
import board
import audiobusio
import neopixel
import microcontroller


BLING = neopixel.NeoPixel(board.MATRIX_DATA,320,brightness=0.05,auto_write=False)

TONE_VOLUME = 0.1  # Increase this to increase the volume of the tone.
FREQUENCY = 600  # Set this to the Hz of the tone you want to generate. 

# Comment out this line and the hard fault goes away 
audio = audiobusio.I2SOut(microcontroller.pin.GPIO2, microcontroller.pin.GPIO1, microcontroller.pin.GPIO3) 

length = 8000 // FREQUENCY
sine_wave = array.array("h", [0] * length)
for i in range(length):
    sine_wave[i] = int((math.sin(math.pi * 2 * i / length)) * TONE_VOLUME * (2 ** 15 - 1))
sine_wave_sample = audiocore.RawSample(sine_wave)

while True:
    # audio.play(sine_wave_sample, loop=True)  # leave commented out
    BLING.fill(0x00FF00)
    BLING.show()
    time.sleep(0.1)
    # audio.stop()        # leave commented out 
    BLING.fill(0x0000FF)
    BLING.show()
    time.sleep(0.1)

Behavior

Running the above on BLING, which has a built-in i2s amp, will eventually hard fault circuitpython and drop into safe mode.
Hard fault: memory access or instruction error, sometimes Internal watchdog timer expired. Sometimes the CIRCUITPY drive becomes inaccessible after the reset OSError: [Errno 5] Input/Output error, sometimes everything disappears including the CIRCUITPY drive label:

$ ls -l /media/cogswell/35D5-F6A3/
total 1
-rw-r--r-- 1 cogswell cogswell 115 Dec 31  1999 boot_out.txt

After that I do a REPL

import storage
storage.erase_filesystem()

to reset things.

I have boot.py set to not autoreload:

import supervisor
supervisor.runtime.autoreload=False

No libraries present in /lib (all the required ones are built-ins with bling).

The process of having the hard fault happen can be exacerbated by doing filesystem access on CIRCUITPY while the program is running. I used this little python program on linux to automate it, just touches a file until it crashes, usually within 10-20 iterations:

import subprocess
import time

c=0
r=0
while(r == 0 ):
    result = subprocess.run(["touch","/media/cogswell/CIRCUITPY/foo.txt"]) 
    r=result.returncode
    print(c)
    c += 1 
    time.sleep(1)

Comment out the line audio = audiobusio.I2SOut(microcontroller.pin.GPIO2, microcontroller.pin.GPIO1, microcontroller.pin.GPIO3) and the touch program runs a long time without crashes (at least ten minutes for the test I did, 700+ iterations).

Note you don't have to be actually playing the audio (audio.play() are commented out), just creating the i2sOut seems to be enough. It takes longer but it will also eventually crash if I just have the audio calls active and comment out all the bling neopixel stuff but I perceive - right or wrong - that "more" i/o activity makes it more likely to hard-fault.

I can cut down the number of leds addressed in the neopixel call from 320 to 1 and it will still hard-fault.

Audio does play fine if the audio.play() calls are uncommented, but aren't seemingly required to make the fault.

Description

No response

Additional information

This is a boiled-down example to show this happens. I was running into this coding a longer program for bling which played audio and would hard-fault crash every now when when I saved (ctrl+s) code.py using vscode on linux (Ubuntu 22.04.3 LTS). In that program if I removed the audio calls it would appear run fine without hard-faulting.

I briefly tried using the Circuitpython 9.0.0-beta0 (from circuitpython.org) but that will throw an IDFerror on audio.play() so I didn't keep at that.

[dhalbert]

@UnexpectedMaker

[UnexpectedMaker]

Yup, I am aware - I asked the OP to post this issue - OP is using his own code, and HW works elsewhere - So this is 100% a CP related issue, and I've not really used I2S at all in CP.

[todbot]

One notable difference I see in comparing the port definitions of unexpectedmaker_bling and adafruit_qtpy_esp32s3_4mbflash_2mbpsram is:
OPTIMIZATION_FLAGS = -Os does not exist in unexpectedmaker_bling/mpconfigboard.mk

[romkey]

I've reproduced this issue overnight on a Bling but I haven't seen it happen yet on a different S3 board running the same code overnight.

[tannewt]

Does it crash without the neopixel updates?

It is possible the I2S interrupt isn't being loaded into RAM for when the flash is busy. The I2S code was changed in 9.0 to the new ESP IDF 5 API.

[scogswell]

I tried this now to check and make sure (I've tried so many variations of things to try and narrow this down I'll probably mix them up in my head). If I comment out all neopixel stuff, but let it make beeping sounds it willl also bomb with Hard fault: memory access or instruction error. I provoked it a little harder by copying a little directory of small .wav files instead of doing a lot of just touches while the program was running.

# SPDX-FileCopyrightText: 2023 Kattni Rembor for Adafruit Industries
# SPDX-License-Identifier: MIT
"""
CircuitPython I2S Tone playback example.
"""
import time
import array
import math
import audiocore
import board
import audiobusio
import neopixel
import microcontroller


# BLING = neopixel.NeoPixel(board.MATRIX_DATA,320,brightness=0.05,auto_write=False)

TONE_VOLUME = 0.1  # Increase this to increase the volume of the tone.
FREQUENCY = 600  # Set this to the Hz of the tone you want to generate. 

# Comment out this line and the hard fault goes away 
audio = audiobusio.I2SOut(microcontroller.pin.GPIO2, microcontroller.pin.GPIO1, microcontroller.pin.GPIO3) 

length = 8000 // FREQUENCY
sine_wave = array.array("h", [0] * length)
for i in range(length):
    sine_wave[i] = int((math.sin(math.pi * 2 * i / length)) * TONE_VOLUME * (2 ** 15 - 1))
sine_wave_sample = audiocore.RawSample(sine_wave)

while True:
    audio.play(sine_wave_sample, loop=True)  # leave commented out
    # BLING.fill(0x00FF00)
    # BLING.show()
    time.sleep(0.1)
    audio.stop()        # leave commented out 
    # BLING.fill(0x0000FF)
    # BLING.show()
    time.sleep(0.1)

[UnexpectedMaker]

I'm happy to add OPTIMIZATION_FLAGS = -Os to my board definitions, but I'd like to understand what it actually does and what side effects this might introduce.

[romkey]

-Os is a gcc option that tell the compiler to "optimize for size". In general it's not wrong to omit it, but it'll help compiled software occupy less RAM. In theory any program should behave identically whether you use it or not, but we all know how that goes...

I think this is set in the 9.0.0 builds though and not 8.2.9 and if I'm reading the issue right the problem is happening in 8.2.9, so this probably isn't the issue.

-Os enables all -O2 optimizations except
       those that often increase code size:

       -falign-functions  -falign-jumps -falign-labels
       -falign-loops -fprefetch-loop-arrays
       -freorder-blocks-algorithm=stc

       It also enables -finline-functions, causes the compiler to
       tune for code size rather than execution speed, and performs
       further optimizations designed to reduce code size.

-O2 Optimize even more.  GCC performs nearly all supported
       optimizations that do not involve a space-speed tradeoff.  As
       compared to -O, this option increases both compilation time
       and the performance of the generated code.

       -O2 turns on all optimization flags specified by -O.  It also
       turns on the following optimization flags:

       -falign-functions  -falign-jumps -falign-labels
       -falign-loops -fcaller-saves -fcode-hoisting -fcrossjumping
       -fcse-follow-jumps  -fcse-skip-blocks
       -fdelete-null-pointer-checks -fdevirtualize
       -fdevirtualize-speculatively -fexpensive-optimizations -fgcse
       -fgcse-lm -fhoist-adjacent-loads -finline-small-functions
       -findirect-inlining -fipa-bit-cp  -fipa-cp  -fipa-icf
       -fipa-ra  -fipa-sra  -fipa-vrp
       -fisolate-erroneous-paths-dereference -flra-remat
       -foptimize-sibling-calls -foptimize-strlen -fpartial-inlining
       -fpeephole2 -freorder-blocks-algorithm=stc
       -freorder-blocks-and-partition  -freorder-functions
       -frerun-cse-after-loop -fschedule-insns  -fschedule-insns2
       -fsched-interblock  -fsched-spec -fstore-merging
       -fstrict-aliasing -fthread-jumps -ftree-builtin-call-dce
       -ftree-pre -ftree-switch-conversion  -ftree-tail-merge
       -ftree-vrp

       Please note the warning under -fgcse about invoking -O2 on
       programs that use computed gotos.

[romkey]

I believe I've reproduced the crash with a Matrix Portal S3, so it's not BLING-specific

Ran the same code as originally posted, just changing the neopixel pin

Copying around 25 Python source files from a project to CIRCUITPY while it was running it hard crashed into the boot loader. MATRXS3BOOT came up. Reset got me back to CircuitPython and the system was able to mount CIRCUITPY. I'm on Sonoma but I'd manually done the remount workaround. I seem to be able to crash it around 50-75% of the time this way.

import time
import array
import math
import audiocore
import board
import audiobusio
import neopixel
import microcontroller
import supervisor

supervisor.runtime.autoreload = False

BLING = neopixel.NeoPixel(board.NEOPIXEL,320,brightness=0.05,auto_write=False)

TONE_VOLUME = 0.1  # Increase this to increase the volume of the tone.
FREQUENCY = 600  # Set this to the Hz of the tone you want to generate. 

# Comment out this line and the hard fault goes away 
audio = audiobusio.I2SOut(microcontroller.pin.GPIO2, microcontroller.pin.GPIO1, microcontroller.pin.GPIO3) 

length = 8000 // FREQUENCY
sine_wave = array.array("h", [0] * length)
for i in range(length):
    sine_wave[i] = int((math.sin(math.pi * 2 * i / length)) * TONE_VOLUME * (2 ** 15 - 1))
sine_wave_sample = audiocore.RawSample(sine_wave)

print("loop")

while True:
    # audio.play(sine_wave_sample, loop=True)  # leave commented out
    BLING.fill(0x00FF00)
    BLING.show()
    time.sleep(0.1)
    # audio.stop()        # leave commented out 
    BLING.fill(0x0000FF)
    BLING.show()
    time.sleep(0.1)

[scogswell]

So far, since I built and started trying Adafruit CircuitPython 9.0.0-beta.0-7-g96be941d83 on 2024-01-31; BLING! with ESP32S3 and working around the IDFerror stuff I haven't had a hard fault. Some quick testing with the code above wouldn't crash it (not extensive, just a few minutes testing), and I spent time coding my regular program which would bomb it every few saves in vscode and that didn't crash it.