FYI: Memory allocation changes for ESP32

[jimmo]

Summary:

• Calling gc.mem_free() on ESP32 reports approximately how much total memory is available for MicroPython, but the real total may be higher or (rarely) lower.
• Many issues with SSL and memory errors, as well as slow GC collection times should now be resolved.
• The issue with only half the PSRAM being available is now resolved.
• ESP32S2 and ESP32S3 builds now automatically detect external PSRAM (see https://github.com/orgs/micropython/discussions/12294)

For most applications, there should be no changes required, other than hopefully the issues around using SSL should be improved. However one consequence is that you can no longer fully rely on gc.mem_free() to tell you how much more allocation can be done. When MicroPython runs out of memory, it will now request more RAM from the OS. The result of gc.mem_free() is the amount of free memory already assigned for MicroPython, plus the largest free block of memory that the OS has available to provide to MicroPython on demand. There will often be additional free blocks of OS memory that could be added in addition. If MicroPython keeps allocating memory then this "grow on demand" process continues until the OS is unable to provide any more memory.

You can use micropython.mem_info() to find out what specifically what the split is between free memory already allocated to MicroPython, and the largest free allocation that could be allocated to MicroPython on demand. This is a lower bound on how much more memory will be available.

(Thanks to @projectgus for all the work on this. Despite this small oddity with the reporting of the memory, overall this is a very important change for the ESP32 and should address a lot of problems we've seen reported).

Background:

On the ESP32 port, MicroPython runs as an application within the RTOS provided by the Espressif IDF. This is different to, e.g. the stm32 or rp2040 ports where we run "bare-metal" and MicroPython has complete control over everything. One long-standing issue is that MicroPython has to ask for memory from the RTOS, and this leads to two problems:

• Anything that MicroPython allocates is no longer available for OS functionality (e.g. SSL buffers, WiFi, etc).
• Even if the OS has (say) 160kiB total free, it might only be able to allocate a 90kiB chunk of contiguous RAM.

A further problem is on ESP32 boards with external PSRAM is that MicroPython has historically allocated a large chunk of this RAM (until recently, we allocated the whole amount, and more recently just half of it, to leave some for the OS). However, this means that when the garbage collector runs, even if your application only has a small working set, it has to do a lot of work to scan the entire RAM during the collection process. (In more detail, unless gc.threshold is configured, the GC only runs when it runs out of RAM, so it will fill up the entire allocation even if there are very few "live" objects, before doing a huge collection to free everything).

On non-PSRAM boards, we were limited to the size of the largest contiguous region (which varied significantly between IDF releases), but also there was a good chance that we left very little to the OS (this is where the SSL memory errors come from).

Now MicroPython tries to start by allocating a small region for the heap, and then only growing as necessary. The initial heap is only 64KiB but will grow on demand.

Details of what's changed:

• MicroPython now supports "split heap", which allows it to operate from multiple non-contiguous regions of RAM.
• On ESP32, MicroPython will request a small initial heap (64KiB) from the RTOS, and then grow it as necessary by adding new "splits".
• It will aim to grow the heap by doubling in size each time (but will grow by more if the current allocation requires it).
• gc.mem_free() returns how much RAM is free in all heap splits, plus the largest free block of RTOS memory that could be made into a new "split" on request. This is a reasonable heuristic for "total free memory".
• micropython.mem_info() prints separate values for how much memory is free in the existing MicroPython memory heap, and what size a "max new split" could be if added on demand. There might be still more available after that, in the next contiguous block.
• MicroPython now does additional bookkeeping on each heap split to track a high watermark (thanks @damz for the work on this!). This improves the performance of collection because the unused portion can be ignored. Combined with the split heap feature, this also allows it to return unused regions back to the OS.

Here is an example of a UM_TINYPICO board (which has 4MiB of external PSRAM).

>>> import gc
>>> import micropython
>>> gc.mem_alloc()
4976
>>> gc.mem_free()
4122064
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 5344, free: 58656, max new split: 4063232
 No. of 1-blocks: 44, 2-blocks: 7, max blk sz: 264, max free sz: 3654

At boot, it has allocated a 64kiB split, and used 4976 bytes. The total free is reported as 4122064 bytes, which is the total of 58656 bytes free in the existing split plus 4063232 bytes available in the PSRAM to be automatically added as a new "split" on demand.

Here is an example on a generic ESP32 module (without PSRAM).

>>> import gc
>>> import micropython
>>> gc.mem_alloc()
1712
>>> gc.mem_free()
172688
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 2080, free: 61920, max new split: 110592
 No. of 1-blocks: 52, 2-blocks: 8, max blk sz: 18, max free sz: 3858
>>> b = bytearray(100*1024)
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 166528, used: 103984, free: 62544, max new split: 14336
 No. of 1-blocks: 24, 2-blocks: 8, max blk sz: 6400, max free sz: 3791

Allocating 100KiB has automatically expanded the total memory used by MicroPython from 64000 bytes to 166528. Additional memory is still available in the RTOS to add a new "split" with more memory, if requested.

Other notes:

• As always, it's best if your program does as much of its large allocations up-front, to avoid memory fragmentation. There is also further work in-progress to allow pre-allocation of SSL buffers via the new SSLContext class.
• If your application works in a sort of cyclic fashion, i.e. some sort of top-level loop, a strategic gc.collect() at the end of each loop when there are as few "live" objects as possible can do a lot to prevent the heap growing unnecessarily.

Please let us know if you have questions or ideas.

(EDIT 2023-09-19: Updated for the newer behaviour of gc.mem_free(), merged after the initial change.)

[steveh127]

Typo on 4th summary point, ESP32S3 is mentioned twice in succession. Should one be EPS32C3?

[jimmo]

Thanks, fixed. It should have been S2. We do not currently support PSRAM on C3.

[bulletmark]

@jimmo , previously we could programmatically fetch the amount of free memory available by calling gc.mem_free() which returns an int value. Now (it seems?) we need to sum the "free" and "max new split" values from micropython.mem_info() but the function returns None because it just does a print() to the repl. How can we programmatically determine the total amount of available free memory?

[projectgus]

Why don't you change gc.mem_free() to return free + max_new_split, which at least is a bit more useful?

I wrestled with these kind of changes a bit when implementing this. There's a lot of fiddliness in the details - for example, the "max new split" value is the largest block of IDF heap memory that can be turned into new MicroPython heap space, but because of metadata overhead for Python heap accounting then you don't get all of this space available once it becomes MicroPython heap, it's at most a little over 97% of this size.

The total free space also doesn't account for fragmentation - both inside the existing heap(s), but then any new heap that gets added is going to be a new fragment compared to the existing heap. Then, as @jimmo also mentioned, there's the likelihood that the IDF heap already has some fragmentation, so after you add the "max new split" there is possibly another block of memory that could be added after this and it might be nearly as large again. One can make arguments that all these blocks should be added into gc.mem_free(), only the largest block should be added into gc.mem_free(), or that none of them should be added.

It's also fairly port-specific, especially as ESP32 memory management is complex in unique and unusual ways... Which is where @jimmo's suggestion of calling the esp32.idf_heap_info function comes in as another way around it.

some heuristic logic which dynamically sets the chunk size based on the available system free ram.

I don't know your application, so this might be way off. However I'm hopeful that now most of these workarounds are no longer needed. There previously wasn't any flexibility in how memory is divided between the "IDF side" and the "MicroPython side". Until now, the only way to try and balance both sides was to tweak the Micropython heap size in C and build new firmware (and even this has limits on the original ESP32, due to memory fragmentation, overcome by the new method.) So it was very common to run out of memory on one side but still have plenty of memory free on the other side.

So I'm hopeful that a larger OTA chunk size will work in your application for scenarios where it didn't before. 🤞 . You can use the gc.mem_free() and esp32.idf_heap_info() functions to try and get a picture of this after OTA has run (particularly the last value in each tuple returned by esp32.idf_heap_info(), which tells you the minimum free IDF heap space over time - this is memory that could have been used by either IDF or MicroPython, given demand.

If things don't turn out as nicely as this, then there's another option provided the OTA buffer is allocated up-front: optimistically allocate the larger chunk size, and fall back to a smaller one if you catch a MemoryError. However, as I said, hopefully this kind of fallback is no longer needed.

[bulletmark]

Angus, thanks for your work on this stuff. Anybody who has worked with any sizable app on MP has almost certainly experienced awkward heap management issues so improvements are very welcome!

In the last sentence of your 2nd paragraph you state 3 options and you have effectively chosen the last one. However I can't see how the value now returned by gc.mem_free() could be of use to anybody because it represents nothing that is physically meaningful. Can you suggest how anybody could use that? Certainly it is very meaningful to MP internally, just not to any user. So I was simply suggesting option 2 (and you point out that option 1 is another consideration) because, as I said above, it is "at least a bit more useful". At least option 2 or 1 give somewhat an approximation to the amount of free memory available, of course with caveats in the documentation stating how inaccurate the number is. I don't see how any case can be made for option 3.

Also, at the moment the documentation for gc.mem_free() says it returns "the number of bytes of available heap RAM" which probably should be updated/clarified.

[projectgus]

You make a good argument, @bulletmark . I'll put up a PR for this.

Can you suggest how anybody could use that?

Ultimately there isn't a 100% reliable way to use any of these metrics that isn't "try to allocate, see if you get a memory error". (This was true even before any of these changes, gc.mem_free() can return a high enough value but the heap is too fragmented for allocations close to that size to succeed.)

But I agree that adding the largest new block to the result will like make the metric more useful, not less.

[projectgus]

Now with PR: #12366

[projectgus]

PR has merged, and I've updated the OP to describe the new behaviour.

[projectgus]

There is another proposed tweak to this scheme, designed to not prematurely exhaust the system heap.

See the PR, here: #13035