ESP32 without psram. Socket and [Errno 105] ENOBUFS

[straga]

Checks

• I agree to follow the MicroPython Code of Conduct to ensure a safe and respectful space for everyone.

• I've searched for existing issues matching this bug, and didn't find any.

Port, board and/or hardware

ESP32 OEM

MicroPython version

Micropython 23.0-preview.346.g64f28dc1e on 2024-05-03

Reproduction

>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 112000, used: 71008, free: 40992, max new split: 21504
 No. of 1-blocks: 979, 2-blocks: 263, max blk sz: 142, max free sz: 133
>>> s1=socket.socket()
>>>
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 112000, used: 71040, free: 40960, max new split: 20480
 No. of 1-blocks: 977, 2-blocks: 265, max blk sz: 142, max free sz: 133
>>> s2=socket.socket()
>>>
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 112000, used: 71072, free: 40928, max new split: 20480
 No. of 1-blocks: 977, 2-blocks: 266, max blk sz: 142, max free sz: 133
>>> s3=socket.socket()
>>>
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 131968, used: 75584, free: 56384, max new split: 248
 No. of 1-blocks: 976, 2-blocks: 268, max blk sz: 282, max free sz: 966
>>> s4=socket.socket()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 105] ENOBUFS

Expected behaviour

No response

Observed behaviour

• Connected to WIFI.

• Already open 3 socket with FTP, TELNET, MQTT
  All works

However, when I attempt to create an additional socket for testing purposes, I encounter an error: OSError: [Errno 105] ENOBUFS. Following this error, the WiFi functionality ceases to work.

Additional Information

Any guidance on how to resolve this issue would be greatly appreciated.

[straga]

“It appears that the issue arises when an asynchronous function is invoked in a synchronous manner. This behavior is particularly noticeable when there’s a significant delay(some time.sleep) in the code execution or when a piece of code takes an extended period to complete its operation in synchronous mode. It seems that asyncio has an impact on these scenarios.”

[straga]

Also

Traceback (most recent call last):
  File "asyncio/core.py", line 1, in run_forever
  File "asyncio/core.py", line 1, in run_until_complete
  File "asyncio/core.py", line 1, in wait_io_event
OSError: [Errno 5] EIO

[straga]

I am try without thread same result. from gc and mcropython shows memory enought.

[straga]

When a call is executed within a thread, an error occurs - like that. However, when the call is executed outside of a thread, not that errors. But the WiFi functionality is compromised. Despite showing a connected status, data transmission is not occurring. Neither sending nor receiving of data is happening, and the ping operation is also failing.

[projectgus]

MicroPython is consuming all of the available memory in the ESP32 for its heap, and ESP-IDF is running out of memory for allocating new sockets. After memory has run out, it's likely the Wi-Fi will also stop working as it regularly allocates and frees buffers.

You can see this happening at the moment the "GC: total" value goes up in mem_info output, and "max new split" drops to a very low number. "max new split" is the largest free memory block that ESP-IDF can use to allocate buffers for sockets and Wi-Fi.

You can also call esp32.idf_heap_info() to confirm this.

MicroPython only grows its heap when it needs to avoid a MemoryError. It looks like your application's total memory usage is still pretty low, but maybe at some point your code allocates a large single buffer and fragmentation means it has to grow the heap for this. If you can find the places in your MicroPython code that do these allocations and either remove them, or allocate the large buffer early in your program and then reuse it, then probably you can prevent MicroPython from growing the heap and then the other errors will go away.

If you're not sure what is causing MicroPython to grow the heap, add some more micropython.mem_info() calls in your code and look for whatever makes the "GC: total" number go up.

[straga]

It turns out that ESP-IDF does not have enough memory for wifi operation. Wifi stops working, but sta.isconnected() == True. Ping timeout from PC to ESP32. Everyone thinks everything is fine. Asyncio stream keeps writing as there are no errors. Only if you try to create new sockets, asyncio crashes with Error 5 EIO.

Push Ctrl+c stop asyncio.

micropython.mem_info()
stack: 704 out of 15360
GC: total: 143936, used: 106624, free: 37312, max new split: 448
 No. of 1-blocks: 1550, 2-blocks: 407, max blk sz: 282, max free sz: 1111

gc.collect()

micropython.mem_info()
stack: 704 out of 15360
GC: total: 143936, used: 106752, free: 37184, max new split: 448
 No. of 1-blocks: 1556, 2-blocks: 408, max blk sz: 282, max free sz: 1111

That look same - (#12819)

Try build with CONFIG_LWIP_TCP_MSL=6000.

Result:

build with

LWIP

CONFIG_LWIP_TCP_MSL=6000
CONFIG_LWIP_SO_LINGER=y
and use awrite:

async def awrite(writer, data,  b=False):

    gc.collect()
    log.info(micropython.mem_info())

    try:
        if isinstance(data, str):
            data = data.encode('utf-8')
        await asyncio.wait_for(writer.awrite(data), timeout=1)
    except Exception as e:
        log.debug("Error: write: {}".format(e))
        pass

Stop frame: result -> ping timeout and ... .

[projectgus]

@straga It might be related to the linked issue, but the root cause of ESP-IDF running out of memory is that MicroPython has already moved all the memory into the "Python heap".

There are two separate heaps, Python heap and ESP-IDF heap. Even if you have free memory in the Python heap, ESP-IDF can't use it. So to prevent this issue, you need to stop the "GC: total: ..." number from ever increasing. Once this memory is added to the Python heap, it's no longer available for ESP-IDF even if it's free for Python...

[straga]

@projectgus I think it is necessary to return back that the buffer for the board without PS RAM is static, then we will have a little less RAM, but wi-fi will not fall off and everything will work as before. There will be 20 kilobytes less RAM but everything will work as before. Or it is necessary to realise this redistribution in some other way.

[projectgus]

@straga MicroPython is growing the heap automatically as your code is running, to prevent a MemoryError. From your logs:

GC: total: 112000,

This is early, there is enough other RAM for ESP-IDF.

GC: total: 143936,

This is later, not enough other RAM for ESP-IDF.

If you find the function in your code that causes this number to increase and refactor it, then ESP-IDF will start working again.

We might be able to add an option to MicroPython to make this simpler, as it's hard for MicroPython to know if it should choose to throw a MemoryError or to grow the heap.

[straga]

@projectgus Thanks for information. I am rewrite my code for use less usage ram as possible. Now works better. While free RAM enought for python and esp-idf, all right. Micropython got first all RAM or more agresive method. If micropython not shows not enought allocated RAM it ok for micropython. But when for esp-idf need more RAM and not free. ESP-IDF not print any just randomly can stop something (in my case wifi stack). Right ?

[projectgus]

@straga Yes, that's how it is at the moment. Glad that you got everything working.