ability to predetermine if code.py is too big to run

[schaefer01]

Hi,
I wrote a program that fit in an rp2040 featherwiing, then added features, then got this error:

Running in safe mode! Not running saved code.
You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Hard fault: memory access or instruction error.

I then stopped calling the added code, but it still crashed, I then started commenting out the code
that wasn't called, and eventually I could again run the program (with limited features).

I'm thinking that some things are known pre-runtime that could be used to let me know ahead of time

whether the program would fit, and

It would be even better if I could know the code space my program took up and how much room

I had to add code in the future or leave a buffer for code growth

[carson-coder]

Could you include your code

[romkey]

This is a fascinating question that involves some of great questions of computer science! Which unfortunately isn't likely to be solved on an RP2040 :)

In formal computer science, this is called "the halting problem" - I'll paraphrase, but it's believed that there's no more efficient way to determine the behavior of a program than to actually run the program. While there are some trivial cases that are simple to recognize, we're not able to predict the behavior of an arbitrary program without actually executing it.

Add in the fact that on microcontrollers a program's behavior can vary dramatically depending on external inputs and the problem becomes even more difficult. And programs can free resources. Just adding up the size of strings or objects in a program isn't going to give you a meaningful result.

Also, the microcontroller environment is already very resource scarce, which is why your program ran out of space. Asking the microcontroller itself (which is where CircuitPython is running) to analyze your program is a very big ask.

If you can suggest some very specific simple heuristics to check, that might be possible at the cost of taking more resources away from the program itself. Those would almost certainly be better implemented in an IDE than CircuitPython itself.

There are tools called "static code analysis" programs. In general they're quite resource intensive and would need to be run on the host computer rather than the microcontroller itself. They're more focussed on code that's likely to cause errors than they are whether your code will fit on a microcontroller. Python tools like pylint are kept separate from the language implementation itself; I'm sure that would be preferable with CircuitPython as well. Due to the halting problem these tools are unable to tell you whether your program will fit or whether it will run out of memory during its execution.

[schaefer01]

HI,

My understanding is that the halting problem is the indeterminacy of whether a program will or will not halt,
not whether it will run out of resources.

There's a big difference between knowing the future over the long term and predicting what will happen on the next step.

Why aren't there runtime calls that can be used to check for heap and stack space before allocation, like C's malloc?

It's interesting that a warm boot doesn't find the problem but a cold boot (power cycle) does.

What is the difference in startup between cold and warm boot?

Could it be that running the over-large code runs into the executive/ciircuitpy?

I'm attaching two versions: mqtt_ether.txt and mqtt_ether-remagen.txt
The "remagen" version has lines of code too far (i.e. the movie a bridge too far)
The non-remagen version has lots of code commented out
mqtt_ether.txt
mqtt_ether-remagen.txt

[schaefer01]

If someone were going to fix this, the information needed is:

1. how much ram is on the feather?
2. where does circuitpy fit in the ram?
3. where does the app get loaded?
4. what is the loading process?
5. can the loading process do ram size pre-checks?

[anecdata]

What version of CircuitPython?

Does it make any difference which code is removed, or is it definitely related to the amount of code removed?

This could be an issue separate from pure RAM usage. If I have a code.py file importing every native module, then a few thousand lines of simple print statement, I get MemoryError: memory allocation failed (as expected) but not safe mode.

(Adafruit CircuitPython 9.0.0-alpha.6-dirty on 2023-12-14; Raspberry Pi Pico W with rp2040)

[schaefer01]

I tried with 8.2.4 and also 8.2.9

I'm not sure its responsible to recommend running code labeled "alpha.-dirty"

I tried 3 different levels of code excision because at first I thought it was code execution (me at fault) and not code loading (maintainers missing something), but only the last level, removal of unexecuted code prevented hard crashes
level 1. removing libraries, conditional imports (disable features)
level 2. conditional execution (more disable features)
level 3. commenting out blocks of unexecuted code (removal disabled features)

On hard crash, initial print statements aren't executed, so it fails before my program starts.

Then again, it could be a circuitypy code overrun after initial execution and not caught until soft reset,
i.e. it takes two soft starts before the hard crash.

1. I load and execute a smaller program.
2. I load and execute the larger program (no hard crash yet but)
3. I soft reset again and get the hard crash that I can only clear by loading a smaller program and sometimes I even have to run the boot loader to reload circuitypy.

What is the code path of loading code.py? Is this part of circuitypy's loader open source?

Is the source of the .uf2 file available? Debuggable?

[schaefer01]

here's my theory, circuitypy sets a sentinal (checksum result?) but only checks it on soft reset after a bad program load,
and regenerates the sentinel on a power cycle

scenario 1. reboot, loads good small code.py, everything is ok

scenario 2. bad large program is copied to code.py after previous ok program (scenario 1),
soft reboot, loads too big program but too-big program / sentinel is not checked,
program runs until power cycle and hard boot,
sentinel generated and now checked, hard fault error message as noted in earlier comment

scenario 3. attempting to load small/good program when large bad program was previously attempted.
soft reset: if small/good code.py is loaded circuitypy has memory that the last boot was bad will not recover
even though the program is small/good
hard reset: sentinel regenerated, small program is recognized as good and will run.

[deshipu]

@schaefer01 is that on MacOS Sonoma?

[schaefer01]

catalina. I just upgraded to sonoma this morning, I'll try again right now ...

[schaefer01]

wild, the remagen program generates an error on copy-and-paste:
The Finder can’t complete the operation because some data in “” can’t be read or written.
(Error code -36)

I'll re-expand the working shorter program to see what happens ...

[schaefer01]

I get the same error when I expand the small program into the too-big program.

deshipu, do you know what is going on?

How does the copy-and-paste operation know that a program is too big to run?

[schaefer01]

hi, I just talked to some knowledgeable people here (my boss) and there are 4 banks of 64K RAM, my program is just over that size in text and definitely smaller if I comment out code.
The question is, can anyone tell me about the paging mechanism, Can a program run across multiple banks of RAM?

[carson-coder]

The source of the .uf2 file is this github code.
Also maybe something doesn't get cleared right when soft rebooting causing the next program to crash?

[carson-coder]

Also try "compiling" the code that fails (convert to .mpy with mpy-cross) and see if it works then. I had to do this to use less memory on my device.