RP2: Out of memory with simple "requests" script

[tacker66]

The following script causes OOM within 10 minutes on a Pico W whereas it runs without leaking memory on a Nano ESP32.
Both are using the same firmware 20230816-unstable-v1.20.0-379-ga18d62e06:

import gc, time
import urequests as requests
while True:
    gc.collect()
    print(gc.mem_free(), gc.mem_alloc())
    telegram = "https://google.com"
    requests.post(telegram)
    time.sleep(60)

[dpgeorge]

What exactly is the error that you receive?

Note that the code needs to close the connection from the request, eg:

r = requests.post(telegram)
r.close()

[tacker66]

With the original application (with slightly older firmware) I got an ENOMEM on the Pico W. In the case above I now get

79472 98960
56480 121952
33488 144944
Traceback (most recent call last):
  File "<stdin>", line 18, in <module>
  File "urequests.py", line 184, in post
  File "urequests.py", line 93, in request
OSError: (-29312, 'MBEDTLS_ERR_SSL_CONN_EOF')

but in all cases I see memory decreasing rapidly.

Using the return value and calling close on it does not leak memory anymore on the Pico W and on the Nano ESP32 it makes no difference.

The fact that you must not ignore the return value of requests.post and call close on it for certain platforms is really well hidden in the docs ;-)

[jimmo]

The fact that you must not ignore the return value of requests.post and call close on it for certain platforms

To explain this in more detail, you must close the response on all platforms.

What's going on here is that the GC will do this closing for you (via the socket's finalizer), but only when collection runs, and if you run out of ssl buffers or lwip sockets before MicroPython runs out of heap memory (triggering the collection), then you see the ENOMEM from the network/ssl stack.

The ratio between the size of the gc heap and the ram available for other things like ssl/lwip is different on esp32 vs rp2 which is why you see the difference.

[jimmo]

I'm not sure exactly how the requests library for CPython handles this specific case of posting without reading the response. The requests docs (unfortunately for us) say you don't usually need to call close, but the automatic close mechanism partly works by closing automatically at the end of the stream (which we also implement). It's possible they also rely on the underlying transport (e.g. urllib3) having a finaliser for this case where the stream doesn't get read to completion.
Either way of course it is much less likely to lead to a memory or resource exhaustion in CPython.

One thing we definitely should consider is adding a context manager (i.e. __enter__ and __exit__) to the response object.

[dpgeorge]

Whilst it is important to close requests/sockets/files when they are no longer needed, MicroPython does support finalisers to try and automatically clean up. And the rp2 port does not enable MICROPY_PY_SSL_FINALISER so the finalisers for SSL sockets are not enabled at all. They are enabled on esp32, so that's likely the reason there is a difference running the code on rp2 vs esp32.

[jimmo]

And the rp2 port does not enable MICROPY_PY_SSL_FINALISER so the finalisers for SSL sockets are not enabled at all.

😢 Raised #12286 to address this hopefully avoid this happening again on other ports.

[dpgeorge]

Finalisers are enabled on rp2 by f3eccb1