-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
Hunt memory allocations in addition to reference leaks #57599
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
This patch adds a counting of the number of allocated memory blocks (through the PyObject_Malloc API). Together with -R, it can help chase those memory leaks which aren't reference leaks (see c6dafa2e2594). The sys.getallocedblocks() function is also available in non-debug mode. This is meant to help 3rd party extension writers, who rarely have access to debug builds. To avoid too many false positives, bpo-13389 is a prerequisite (at least for the "test -R" part of the patch). Even after it, there are still a couple "test -R" failures; we'd have to investigate them. |
I added some review comments to the patch, but I'm not sure how usable this is going to be in practice. References generally stay fairly stable while using the interactive interpreter, but the new block accounting jumps around all over the place due to the internal free lists (which *don't* count in for 'gettotalrefcounts', but *do* count in the new block accounting). The following interpreter session has both this patch and the bpo-13389 patch applied: >>> a = sys.getallocedblocks()
[76652 refs, 21773 blocks]
>>> a
21779
[76652 refs, 21774 blocks]
>>> x = [None]*10000
[76652 refs, 21776 blocks]
>>> del x
[66650 refs, 21775 blocks]
>>> gc.collect(); gc.collect(); gc.collect()
0
0
0
[66650 refs, 21756 blocks]
>>> b = sys.getallocedblocks()
[66652 refs, 21772 blocks]
>>> b - a
-2
[66652 refs, 21772 blocks] So, generally +1 on the idea, but I think we should hide at least the initial implementation behind PY_REF_DEBUG until we're sure we've worked the kinks out of it. |
Thanks for the comments, here is a new patch addressing them. As for the memory leak run results, I think we may have to devise a heuristic where results like [1,0,1] (or similar variation of 0s, 1s, and -1s) are considered a success, but others like [1,1,1] are a failure. |
And the latest patch (debugblocks3.patch) adds said heuristic. |
Valgrind does a much better job at this: it will also show you where the leaked blocks were allocated. |
This will likely be a decent "you have a problem" indicator, but you may still need tools like Valgrind to actually track down the *cause* of that problem. |
Yes, Valgrind is much more exhaustive and precise, but you have to build ==20599== Conditional jump or move depends on uninitialised value(s) Right now this patch will allow to enrich the daily refleak runs (those |
Did you use the valgrind suppression file (Misc/valgrind-python.supp)?
Daily, or rather weakly, since running under valgrind is so much slower ;-) |
Ah, I hadn't. But using it doesn't seem to make much of a difference. |
The overhead on PyObject_Malloc() is just an increment on an integer, so it is very low (or null). The feature is interesting, but I'm not convinced that a very simple counter is enough to track memory leaks. It may help the CPython test suite, but what about real world application?
Programs not always behave exactly the same in debug or in release mode. Sometimes, bugs disappear in debug mode :-( If the feature is written to track memory leaks in the CPython test suite, it's ok to only expose the function in debug mode.
Did you already found real leaks using your hack^Wpatch? (was c6dafa2e2594 found by your tool?) |
Good question. A simple counter is the only thing we can enable by default, though. Anything else would require recompiling Python, which is probably a barrier for most users.
yes, c6dafa2e2594 was found with this patch. It's the only one, though (there's also a leak in test_ctypes but I don't want to investigate :-)). |
On Fri, Dec 2, 2011 at 12:12 PM, Antoine Pitrou <report@bugs.python.org> wrote:
I'll take a look at the ctypes leak. |
I looked at the 'ctypes' "leak" a bit. I haven't determined exactly what import sys, ctypes, gc
ctypes._reset_cache()
gc.collect()
for i in range(0, 5):
ctypes._reset_cache()
gc.collect()
print("%d: start refs = %s" % (i, sys.gettotalrefcount()))
proto = ctypes.CFUNCTYPE(ctypes.POINTER(ctypes.c_char))
ctypes._reset_cache()
gc.collect()
print("%d: after refs = %s" % (i, sys.gettotalrefcount())) which prints: 0: start refs = 71395 Note that the start/after refs converge on a difference of 29 references. The existing version 'regrtest.py' does something like: import sys, ctypes, gc
ctypes._reset_cache()
gc.collect()
for i in range(0, 5):
print("%d: start refs = %s" % (i, sys.gettotalrefcount()))
proto = ctypes.CFUNCTYPE(ctypes.POINTER(ctypes.c_char))
ctypes._reset_cache()
gc.collect()
print("%d: after refs = %s" % (i, sys.gettotalrefcount())) which prints: 0: start refs = 71391 This one converges on a difference of zero. So, I am not sure whether there really is a leak, if this is just |
Well, test_ctypes seems to be the only test exhibiting that behaviour. |
How different is the performance cost of this solution compared to inserting DTrace probe for the same purpose? |
DTrace is only available on some platforms (Solaris and maybe FreeBSD?). |
On Mon, Dec 12, 2011 at 6:11 PM, STINNER Victor <report@bugs.python.org>wrote:
Solaris <http://en.wikipedia.org/wiki/Solaris_(operating_system)\>, Mac |
Here is an updated patch. $ ./python -m test -R 3:8 test_ctypes
[1/1] test_ctypes
beginning 11 repetitions
12345678901
...........
test_ctypes leaked [2, 2, 1, 1, 1, 1, 1, 1] memory blocks, sum=10 |
As it turns out, ctypes does leak memory: see bpo-16628. |
Updated patch with doc. If noone objects, I will commit soon. |
Minor bikeshedding/spelling nit: should (and s/_Py_GetAllocedBlocks/_Py_GetAllocatedBlocks/)? |
Right indeed. I'll do the change. |
New changeset c40f4c19d20b by Antoine Pitrou in branch 'default': |
Committed and pushed! |
``./configure --without-pymalloc'' fails here: gcc -pthread -Xlinker -export-dynamic -o python Modules/python.o libpython3.4.a -lpthread -ldl -lutil -lm |
Hmm, interesting. When built --without-pymalloc, we could make |
sys.gettotalrefcount() is only defined when Python is compiled in |
Memory control over the stuff that Python creates is a practical feature that compensates OS disability for tracking memory usage. If all Python scripts could measure their memory usage, we could see more memory effective and adaptive programs around. |
On the contrary, the aim is precisely to provide a memory statistics |
Antoine Pitrou <report@bugs.python.org> wrote:
Given the name getallocatedblocks(), it would seem reasonable to return I'm using the option only for Valgrind testing, that's how I found |
New changeset a85673b55177 by Antoine Pitrou in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: