Starlette and pandas results in unreleased memory piling up when using sync endpoints #1573
Replies: 1 comment
-
I'll close this as outdated. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
The problem
The core problem I'm trying to solve here is that when you define sync endpoints in Starlette and there is a problem with an dependency where it won't properly release memory, the idle memory usage will grow unbounded. In my case pandas has a known issue of not releasing memory.
The problem doesn't occur with async endpoints, only with sync endpoints.
Reproducing
Results
The process goes from using around 50MiB to taking up roughly 3.5GiB during request time, and then once the requests are completed it settles around 800MiB, and it never decreases from this, and infact if you keep making the same requests then it increases to 1GiB, and then again to 1.2GiB and so on.
If I instead make requests to http://localhost:8080/async in the same HTTP load testing script, then the process goes from using 60MiB to taking up roughly 160MiB during request time, and then once the requests are completes it settles around 100MiB.
Thoughts
I've read that using the environment variable MALLOC_TRIM_THRESHOLD_=0 is supposed to help with memory behaviour in pandas, but the test above is already using that. If I remove that environment variable, then it goes from 50MiB to 4.3GiB, to settling on 3GiB, i.e. it uses even more memory so there is some truth to that but it doesn't address the underlying core problem.
When you make sufficient requests to the sync endpoints, Starlette spins up worker threads:
If however you use the async endpoints, it doesn't do that.
You can see here that if it's a coroutine, then it's run as-is, but if it's not then it runs everything in a threadpool that then creates all of these worker threads:
starlette/starlette/routing.py
Line 62 in 20d24a8
=>
starlette/starlette/concurrency.py
Line 35 in 20d24a8
=>
https://github.com/agronholm/anyio/blob/e23b44e171c71d44e57d0c103a7daec1aaa7ad57/src/anyio/to_thread.py#L10
But I don't think this is the underlying cause of the problem because these worker threads are killed on new requests being made after 10 seconds of being idle.
What I'm not sure about is if there is a way to avoid this ever-increasing memory usage, other than switching to entirely using async endpoints. Also if there is a way to make pandas better behaved w.r.t memory.
References:
Pandas memory leak issues:
pandas-dev/pandas#2659
pandas-dev/pandas#21353
Beta Was this translation helpful? Give feedback.
All reactions