Avoid out of memory errors when querying globals in MicroPython

[aivarannamaa]

See https://groups.google.com/g/thonny/c/aXePExX2R0U

I randomly get this error:

THONNY FAILED TO EXECUTE COMMAND get_globals
SCRIPT:
__thonny_helper.print_mgmt_value({name : __thonny_helper.repr(value) for (name, value) in globals().items() if not name.startswith('__')})
STDOUT:

STDERR:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 21, in print_mgmt_value
MemoryError: memory allocation failed, allocating 4480 bytes

it occurs with different programs so I can't catch a common reason for it. After a reset the same programs work without errors.

• use shorter repr and/or make it configurable
• use gc before and/or after querying globals
• Make sure globals are not queried when Variables table is hidden.

[aivarannamaa]

now I get this error:
"MemoryError: memory a"
just as I reset the ESP8266 and press F5 to execute my program.

[pyforever]

Hello,
I've 300 lines of code for about 12KB. I checked that upython takes more or less 500KB, so I'd have more than 450KB free, right?
I haven't very large structures, I'm only developing now. I've this one:
timers = {"1a":[(3,11), [7,1,0,0,1000,1,"0000001111222233"]], "1b":[(3,12), [7,0,1,0,1000,1,"0000001111222233"]], "2a":[(3,34), [4,1,0,1,0,3,"0000004444666688"]], "3a":[(3,32), [9,1,0,1,1000,9,"0000002222333344"]], "2b":[(3,36), [4,0,1,0,0,3,"0000004444666688"]], "3b":[(3,35), [9,0,1,1,1000,9,"0000002222333344"]]}
and something like 10 strings long about 50 chars.
There should be plenty of space!

[aivarannamaa]

Does it work when you save your script as main.py and perform a soft-reboot (Ctrl+D) ?

[alexmojaki]

For shorter reprs of complex structures you might like https://github.com/alexmojaki/cheap_repr which is what birdseye uses, otherwise https://docs.python.org/3/library/reprlib.html

[aivarannamaa]

@alexmojaki, thanks for the suggestions, but I'm afraid cheap_repr is too heavy for MicroPython, and Python 3 library is simply not available.

[pyforever]

Does it work when you save your script as main.py and perform a soft-reboot (Ctrl+D) ?

Yes, it works as expected when saved as main.py.
Indeed I can't understand why it shouldn't run with so much free memory !

[pyforever]

For shorter reprs of complex structures you might like https://github.com/alexmojaki/cheap_repr which is what birdseye uses, otherwise https://docs.python.org/3/library/reprlib.html

Thanks alexmojaki, not feasible in my present project but interesting for others!

[pyforever]

Does it work when you save your script as main.py and perform a soft-reboot (Ctrl+D) ?

Yes, it works as expected when saved as main.py.
Indeed I can't understand why it shouldn't run with so much free memory !

Hi Aivar, no hint for this? I can't go on developing and I guess it's a Thonny problem otherwise my code wouldn't run at all also when flashed in the ESP8266. Please tell me how can I fix this.

Selecting different functions to be executed (in the same code as before) I get this other Thonny error:

>>> THONNY FAILED TO EXECUTE COMMAND get_globals

SCRIPT:
__thonny_helper.print_mgmt_value({name : __thonny_helper.repr(value) for (name, value) in globals().items() if not name.startswith('__')})

STDOUT:

STDERR:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 21, in print_mgmt_value
MemoryError: memory allocation failed, allocating 1816 bytes

[aivarannamaa]

@pyforever, did you try closing Variables table? Did this change something?

How many files do you have on the board's filesystem?

[pyforever]

@pyforever, did you try closing Variables table? Did this change something?

How many files do you have on the board's filesystem?

Sorry for late answer Aivar, yes i tried closing Variables table but it didn't seem to have relevant benefits. I gave a re-work to my code making it more clear and throwing some gc.collect here and there (after instancing large tables mostly) and now it seems more stable. I didn't quite change anything, only made some order in the functions libraries. However it never gave problems when run in the board, so I guess there must be something in Thonny which in some peculiar cases goes wrong. Maybe some objects in some case don't dealloc heap memory when destroyed? It's just a guess, you're the king of Thonny, of course :-)

[aivarannamaa]

@pyforever, I tweaked showing global variables, such that it should take less memory now.

Regarding memory problems when running code -- here we need to take into account that MicroPython also needs memory for parsing and compiling the code. When you are doing F5 in Thonny, it is less efficient.

You may want to try newest MicroPython release (1.14). With this, Thonny tries to use a new and more efficient mode for submitting code.

[pyforever]

@pyforever, I tweaked showing global variables, such that it should take less memory now.

Regarding memory problems when running code -- here we need to take into account that MicroPython also needs memory for parsing and compiling the code. When you are doing F5 in Thonny, it is less efficient.

You may want to try newest MicroPython release (1.14). With this, Thonny tries to use a new and more efficient mode for submitting code.

I installed 1.14 and trying it. I'll tell you if it goes better.

[aivarannamaa]

@pyforever, please also install the fresh Thonny 3.3.4, which tries to use less memory when querying local variables: https://github.com/thonny/thonny/releases/tag/v3.3.4

[aivarannamaa]

Querying globals is refactored in Thonny 4.0 and should work much better now