-
Notifications
You must be signed in to change notification settings - Fork 15.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak in Python in version 3.15.0+ AND in 4.21.1 #10088
Comments
Thanks for the minimal repro. I've spent some time trying to debug this, but it's tricky because the relationship between malloc()/free() calls and RSS is not straightforward. I would still like to get to a root cause and fix here. But it will probably take some time. |
Hi, is there any update on this? We are also seeing this problem and are in a bit of a bind as our system is unable to scale to any reasonable capacity due to memory leaking rapidly on high volume use causing crashes and restarts. |
You could use the Python implementation by setting the environment variable: Though it might be too slow for your system, but it doesn't have this memory issue. |
After much debugging, I've gotten to a root cause for this. The problem is that the glibc The pymalloc allocator on the other hand returns memory to the OS as quickly as possible, using When you use the pure-Python protobuf library, all of the proto objects are allocated using pymalloc, so when they are collected pymalloc immediately returns the memory to the OS. But when you use an extension module, the proto objects are allocated at the C layer using There are two possible solutions to this problem. I have verified that both of these solutions successfully make the RSS go down in the test case attached to this issue:
There are pros and cons to both of these. It's not immediately clear which one is better. (1) has the benefit of being more portable, since it only relies on Python APIs that will always be available, whereas (2) introduces a conditional dependency on a glibc-specific function. On the other hand, the pymalloc functions for (1) require that the GIL is held, and it may be difficult to guarantee that if we are sharing messages between Python and another language. For example, if you want to call a C or C++ API that takes a proto, you may want to pass the Python proto to C or C++ without a copy. At the point it may be difficult to ensure that C or C++ will not hold a reference that is freed at an unpredictable time. But maybe this is not a concern, as it could be unsafe anyway for C or C++ to hold a reference, since Python message objects are mutable and this could introduce data races. There is also the question of performance of pymalloc vs glibc malloc. It is worth benchmarking which one of these is faster in a microbenchmark, just to see if there is a significant difference. |
Also got hit with this, latest 3.x versions didn't have this leak, but 4+ versions all seem to leak memory. |
…en memory is freed This partially fixes #10088. However we still have a global map that does not shrink, which can still create the appearance of leaking memory, as it will not be freed until the module is unloaded. PiperOrigin-RevId: 562976233
…en memory is freed This partially fixes #10088. The test case from that bug significantly improves with this change. However we still have a global map that does not shrink, which can still create the appearance of leaking memory, as it will not be freed until the module is unloaded. PiperOrigin-RevId: 562976233
#13865 uses
This isn't perfect (ie. it doesn't get back below 20) because the global object cache used internally by upb is still using memory which will not be released until process shutdown. We could address that separately in a follow-up change. But #13865 should address the majority of the issue here. |
@ericsalo @haberman were fixes here patched to 24.x ? I am still seeing a leak in |
i'm observing the same behavior as #10088 (comment) |
I just ran tests for both of the memory leak repros, and the results indicate that both of the reported fixes are present in 4.24.3. For the repro attached to this bug, I get:
For the repro attached to https://github.com/protocolbuffers/upb/issues/1243, I get:
If you are seeing memory leaks with 4.24.3, it must be another issue. If you have a repro, please open another issue and we will fix it. |
…en memory is freed This partially fixes protocolbuffers#10088. The test case from that bug significantly improves with this change. However we still have a global map that does not shrink, which can still create the appearance of leaking memory, as it will not be freed until the module is unloaded. PiperOrigin-RevId: 563124724
What version of protobuf and what language are you using?
Version: 3.14.0/v3.15.0/v4.21.1
Language: Python
What operating system (Linux, Windows, ...) and version?
Ubuntu Linux
What runtime / compiler are you using (e.g., python version or gcc version)
Python v3.9.5
Built protobuf schema with protoc 3.14.0
Built protobuf schema with protoc 3.21.1 too.
What did you do?
Steps to reproduce the behavior:
git clone https://github.com/Atheuz/proto_leak.git
.cd proto_leak
.virtualenv .venv --python=3.9
and activate it withsource .venv/bin/activate
.pip install -r requirements.txt
. (installs protobuf==3.14.0)python leak.py
pip install protobuf==3.15.0
python leak.py
pip install protobuf==4.21.1
from schema_pb2 import value_test_topic
withfrom schema_4_21_1_pb2 import value_test_topic
on line 4 inleak.py
python leak.py
What did you expect to see
In 3.15.0, I expected it to free basically all memory used for the list of pb items.
In 4.21.1, I expected it to free basically all memory used for the list of pb items.
What did you see instead?
In 3.15.0, it retains 500+ MB of memory that is not retained in 3.14.0.
In 4.21.1, it retains 1000+ MB of memory that is not retained in 3.14.0.
Previously reported in #9917, was closed as to be fixed in 4.21.1, but doesn't seem like it actually was.
The text was updated successfully, but these errors were encountered: