Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing some libraries on 7.0.0 (FeatherS2) causes hard crash #5428

Closed
geudrik opened this issue Oct 4, 2021 · 15 comments
Closed

Importing some libraries on 7.0.0 (FeatherS2) causes hard crash #5428

geudrik opened this issue Oct 4, 2021 · 15 comments
Labels
bug crash espressif applies to multiple Espressif chips needs retest
Milestone

Comments

@geudrik
Copy link

geudrik commented Oct 4, 2021

CircuitPython version

Adafruit CircuitPython 7.0.0 on 2021-09-20; FeatherS2 with ESP32S2

Code/REPL

>>> from arcti.ntp import NTP
>>> 
^ Works fine (https://discord.com/channels/327254708534116352/537365702651150357/894552574345220158)

>>> from adafruit_gc_iot_core import MQTT_API, Cloud_Core
<< hard crash >>
Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Crash into the HardFault_Handler.
Please file an issue with the contents of your CIRCUITPY drive at 
https://github.com/adafruit/circuitpython/issues

Press any key to enter the REPL. Use CTRL-D to reload.

Behavior

Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Crash into the HardFault_Handler.
Please file an issue with the contents of your CIRCUITPY drive at
https://github.com/adafruit/circuitpython/issues

Press any key to enter the REPL. Use CTRL-D to reload.

Description

Importing some libraries causes a hard crash. I have my own little folder of helper libs that import fine. The adafruit_gc_Iot_core lib causes hard crashes for me (though it's intermittent - other times its a stack recursion limit issue, given the coupling, seems related)

Additional information

No response

@geudrik geudrik added the bug label Oct 4, 2021
@tannewt tannewt added crash espressif applies to multiple Espressif chips labels Oct 5, 2021
@tannewt tannewt added this to the 7.x.x milestone Oct 5, 2021
@tannewt
Copy link
Member

tannewt commented Oct 5, 2021

Please post a zip of all files on your CIRCUITPY drive so we can reproduce it.

@geudrik
Copy link
Author

geudrik commented Oct 7, 2021

circuit_python_feathers2_crashing.zip

Here's the zip that contains a single folder lib. In that folder, I've been able to reliably reproduce the crash by changing the name of the adafruit_binascii lib to adafruits_binascii (something that ensures it's not importable). Once I do that, I can from the REPL run from adafruit_gc_iot_core import MQTT_API, Cloud_Core and it will crash. Once it's crashed, I have to physically press the reset button to clear the crash state and allow code to run - hit the reset button, and then fix the name of the binascii file to what it's supposed to be, then run the same import from the REPL and you'll see a stack recursion exception (import depth too stronk, issue here )

@geudrik
Copy link
Author

geudrik commented Oct 8, 2021

@tannewt is there a doc somewhere to describe how you see the coredump (or whatever the equivalent is for a microcontroller)? I'd be interested to take a gander at what's happening under the covers!

@tannewt
Copy link
Member

tannewt commented Oct 8, 2021

@geudrik Your help would be awesome. ESP32-S2 is a bit trickier than the Cortex-Ms. Generally I build CircuitPython with DEBUG=1 on the make line. It'll then output debug serial to UART0. On a crash, this may include a Backtrace: line that can be used to get a call stack.

I don't know of a doc for ESP32-S2 debugging. I'd encourage you to join the Adafruit Discord and get help from us in #circuitpython-dev. We are happy to help you dig into things.

@geudrik
Copy link
Author

geudrik commented Oct 9, 2021

I'm getting make errors. I'm using a Mac (Apple Silicon), and followed the guild for getting going on OSX. One of the deps that was missing was Ninja (brew install ninja), but I'm not sure how this build chain works :S

circuitpython/ports/espressif on  main via △ v3.21.3 took 3s 
❯ make DEBUG=1 BOARD=unexpectedmaker_feathers2 -j8 
Use make V=1, make V=2 or set BUILD_VERBOSE similarly in your environment to increase build verbosity.
IDF_PATH=/Volumes/CircuitPython_Dev_Playground/circuitpython/ports/espressif/esp-idf cmake -S . -B build-unexpectedmaker_feathers2/esp-idf -DSDKCONFIG=build-unexpectedmaker_feathers2/esp-idf/sdkconfig -DSDKCONFIG_DEFAULTS="esp-idf-config/sdkconfig.defaults;esp-idf-config/sdkconfig-debug.defaults;esp-idf-config/sdkconfig-16MB.defaults;boards/unexpectedmaker_feathers2/sdkconfig" -DCMAKE_TOOLCHAIN_FILE=/Volumes/CircuitPython_Dev_Playground/circuitpython/ports/espressif/esp-idf/tools/cmake/toolchain-esp32s2.cmake -DIDF_TARGET=esp32s2 -GNinja
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- The ASM compiler identification is unknown
-- Found assembler: xtensa-esp32s2-elf-gcc
CMake Error at esp-idf/tools/cmake/project.cmake:311 (__project):
  The CMAKE_C_COMPILER:

    xtensa-esp32s2-elf-gcc

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
  the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
  CMakeLists.txt:12 (project)


CMake Error at esp-idf/tools/cmake/project.cmake:311 (__project):
  The CMAKE_CXX_COMPILER:

    xtensa-esp32s2-elf-g++

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
  CMakeLists.txt:12 (project)


CMake Error at esp-idf/tools/cmake/project.cmake:311 (__project):
  The CMAKE_ASM_COMPILER:

    xtensa-esp32s2-elf-gcc

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "ASM" or the CMake cache entry CMAKE_ASM_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
  CMakeLists.txt:12 (project)


-- Warning: Did not find file Compiler/-ASM
-- Configuring incomplete, errors occurred!
See also "/Volumes/CircuitPython_Dev_Playground/circuitpython/ports/espressif/build-unexpectedmaker_feathers2/esp-idf/CMakeFiles/CMakeOutput.log".
See also "/Volumes/CircuitPython_Dev_Playground/circuitpython/ports/espressif/build-unexpectedmaker_feathers2/esp-idf/CMakeFiles/CMakeError.log".
make: *** [build-unexpectedmaker_feathers2/esp-idf/config/sdkconfig.h] Error 1

@geudrik
Copy link
Author

geudrik commented Oct 9, 2021

Disregard, I've had a bad path line (and totally missed the S2 specific instructions!)

@geudrik
Copy link
Author

geudrik commented Oct 9, 2021

Build log attached. I wind up with build errors at the final stage :S Using Python 3.9, Latest OSX (Apple Silicon), and the build chain outlined in the S2 docs.

$ make DEBUG=1 BOARD=unexpectedmaker_feathers2 -j8 |tee ~/feathers2_build.log

feathers2_build.log

@dhalbert
Copy link
Collaborator

dhalbert commented Oct 9, 2021

I do not see an error in the feathers2_build.log above. Was there an error written to stderr?

@geudrik
Copy link
Author

geudrik commented Oct 9, 2021

Huh.. apparently the pipe to tee didn't capture the error... I should have checked before I uploaded.

This is the error

/Users/grodik/.espressif/tools/xtensa-esp32s2-elf/esp-2021r1-8.4.0/xtensa-esp32s2-elf/bin/../lib/gcc/xtensa-esp32s2-elf/8.4.0/../../../../xtensa-esp32s2-elf/bin/ld: build-unexpectedmaker_feathers2/firmware.elf section `.dram0.bss' will not fit in region `dram0_0_seg'
/Users/grodik/.espressif/tools/xtensa-esp32s2-elf/esp-2021r1-8.4.0/xtensa-esp32s2-elf/bin/../lib/gcc/xtensa-esp32s2-elf/8.4.0/../../../../xtensa-esp32s2-elf/bin/ld: region `dram0_0_seg' overflowed by 2176 bytes
collect2: error: ld returned 1 exit status
make: *** [build-unexpectedmaker_feathers2/firmware.elf] Error 1

@dhalbert
Copy link
Collaborator

The DEBUG=1 builds don't fit anymore on ESP32-S2. Your build is probalby fine otherwise. You can turn off several modules and make them fit. See #4984 (comment) as an example of what to turn off. The code region is not overflowing; it's something else, and we need to figure out the best way to make the DEBUG builds easier.

@geudrik
Copy link
Author

geudrik commented Oct 10, 2021

Oookiedoke, after getting the debug build set up and running, I no longer get hard crashes, but instead a permanent hang. Once I get here and it's hung, if I continue to wait, it eventually .. does something (it removes itself from connected to my laptop, and screen terminates since the device vanishes). I then am unable to reconnect to the device at all (no such file or directory) until I press the reset button. No output is generated when it crashes (logging from screen shows the last line as the import statement)

image

@geudrik
Copy link
Author

geudrik commented Oct 10, 2021

here's the diff of mpconfigport.mk in case I messed something up

diff --git a/ports/espressif/mpconfigport.mk b/ports/espressif/mpconfigport.mk
index 206de09ab..5adbd3163 100644
--- a/ports/espressif/mpconfigport.mk
+++ b/ports/espressif/mpconfigport.mk
@@ -12,25 +12,25 @@ LONGINT_IMPL = MPZ
 # These modules are implemented in ports/<port>/common-hal:
 CIRCUITPY_FULL_BUILD ?= 1
 CIRCUITPY_ALARM ?= 1
-CIRCUITPY_AUDIOCORE ?= 1
+CIRCUITPY_AUDIOCORE ?= 0
 CIRCUITPY_AUDIOMP3 ?= 0
-CIRCUITPY_AUDIOBUSIO ?= 1
+CIRCUITPY_AUDIOBUSIO ?= 0
 CIRCUITPY_AUDIOBUSIO_PDMIN ?= 0
 CIRCUITPY_AUDIOBUSIO_I2SOUT ?= 1
 CIRCUITPY_AUDIOIO ?= 0
-CIRCUITPY_AUDIOMIXER ?= 1
-CIRCUITPY_CANIO ?= 1
+CIRCUITPY_AUDIOMIXER ?= 0
+CIRCUITPY_CANIO ?= 0
 CIRCUITPY_COUNTIO ?= 1
 CIRCUITPY_DUALBANK ?= 1
 CIRCUITPY_FRAMEBUFFERIO ?= 1
 CIRCUITPY_FREQUENCYIO ?= 1
-CIRCUITPY_IMAGECAPTURE ?= 1
+CIRCUITPY_IMAGECAPTURE ?= 0
 CIRCUITPY_I2CPERIPHERAL ?= 0
-CIRCUITPY_RGBMATRIX ?= 1
+CIRCUITPY_RGBMATRIX ?= 0
 CIRCUITPY_ROTARYIO ?= 1
 CIRCUITPY_NVM ?= 1
-CIRCUITPY_PS2IO ?= 1
-CIRCUITPY_TOUCHIO_USE_NATIVE ?= 1
+CIRCUITPY_PS2IO ?= 0
+CIRCUITPY_TOUCHIO_USE_NATIVE ?= 0
 CIRCUITPY_WIFI ?= 1
 CIRCUITPY_WATCHDOG ?= 1

@dhalbert dhalbert modified the milestones: 7.x.x, 8.0.0 May 24, 2022
@DavePutz
Copy link
Collaborator

DavePutz commented Aug 6, 2022

After building 8.0.0-alpha-1 with the stack size modification from PR #6514 the crash/hang is resolved; CP now throws an error:

from adafruit_gc_iot_core import MQTT_API, Cloud_Core
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "adafruit_gc_iot_core.py", line 42, in <module>
  File "adafruit_jwt.py", line 39, in <module>
  File "adafruit_rsa/__init__.py", line 17, in <module>
  File "adafruit_rsa/pkcs1.py", line 20, in <module>
  File "adafruit_hashlib/__init__.py", line 32, in <module>
MemoryError: memory allocation failed, allocating 1210 bytes

The question of why importing standard libraries runs out of memory should be addressed separately.

@DavePutz
Copy link
Collaborator

DavePutz commented Aug 7, 2022

I was using a espressif Saola Wroom (smaller memory) for my previous test. I did some more testing using a MagTag (which has a lot more memory). It also hung without the patch from PR #6514. Compiling with that patch and running
"from adafruit_gc_iot_core import MQTT_API, Cloud_Core" gc.mem_free() reports that 73936 bytes
were used. This looks like mainly due to the import of adafruit_rsa, which took up 64080 bytes. So, a Feather S2
should have enough memory as well to work once that stack size patch is applied.

@geudrik
Copy link
Author

geudrik commented Aug 31, 2022

I haven't tested yet on 8.x firmware, but it sounds like this issue can be closed for now.

I had started doing some optimization of libraries a while back to reduce the stack size (tons of imports all over the place that can be minimized) and that made a significant improvement as well.

@geudrik geudrik closed this as completed Aug 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug crash espressif applies to multiple Espressif chips needs retest
Projects
None yet
Development

No branches or pull requests

4 participants