-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
Empty __slots__ can create untracked reference cycles #86150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Currently, we don't track instances of certain heap types based on the assumption that "no members" == "no reference cycles". Unfortunately, it's still possible to create untracked reference cycles with one's parents. The following program leaks memory: while True:
class C:
__slots__ = ()
C.i = C()
del C The fix is simple: track all instances of user-defined classes, no exceptions. I'm not sure we were actually getting any real wins from the old behavior anyways. |
Thanks for noticing this, Brandt!
I have the fear that this may increase the time of collections substantially because even if many of them won't have references at all, they still net to be visited and they count towards the time complexity of the algorithm that detects isolated sub-cycles. I mention this because this could affect an unbounded number of instances per type. For example, some of the types that are untracked. Maybe I am missing something but we could mark them as having GC support unconditionally but still leave them untracking and unconditionally add a tracking call on setattribute. |
Hm, I’m not sure that would be enough. Consider the case of a class that registers its instances in a collection of some sort: while True:
class D:
__slots__ = ()
registry = []
def __init__(self):
self.registry.append(self)
for i in range(100):
D() and None # Suppress REPL output.
del D This is probably more common (and problematic) than my example above. At the same time, I agree that it’s not *ideal* to track all of these objects automatically. Anyone setting __slots__ is probably planning on creating lots of “cheap” instances. If they do accidentally create cycles, though, I feel the memory hit then would be worse than any collection overhead. I’m just not sure I see a way to fix this without tracking them all. |
IIRC we do skip the GC flags for user-created types only when the subtype is not adding new variables *and* the base class is not a GC class by itself. This includes the case with __slots__ but is not limited to. If I recall correctly, there is a bunch of metaclasses that fall into this category and some other minor things so maybe is not that bad to unconditionally make all user objects tracked. In any case I think is prudent to run the performance test suite with PGO/LTO + CPU isolation to get an idea. Unfortinately we already have some unwanted 3.9 performance regressions of unknown origin and I would like to not add to it if we can. |
Using the following patch: master...brandtbucher:track-all-heap-types I got the following pyperformance results (with PGO/LTO and CPU isolation, insignificant rows omitted): 2020-10-13_20-04-master-7992579cd27f.json.gz Performance version: 1.0.0 2020-10-13_20-37-track-all-heap-types-63d8d25867b5.json.gz Performance version: 1.0.0 +-------------------------+----------------------------------------------+------------------------------------------------------------+--------------+-----------------------+ Note that I used pyperformance==1.0.0, since 1.0.1 won't run on 3.10. In general, it looks like the patch has no real performance impact. The result for unpack_sequence is *extremely* surprising... I'm quite skeptical of it, although it looks like it is one of the more "micro" benchmarks in the suite. All things considered, I believe we should fix this... I personally view memory leaks as second only to crashes and incorrect behavior in terms of severity. |
Thanks a lot, Brandt for doing the benchmarking! I think that unpack_sequence is **very** affected by code locality. I was debugging it recently with Linux perf (https://perf.wiki.kernel.org/index.php/Main_Page) and it shows that is extremely affected by L1-L2 cache performance. I still need to understand why, but yeah, I am fine ignoring it.
Yeah, I very much agree with you. Could you convert your patch into a PR? |
No problem. I just need to rework some hacks in test_finalization where we use empty __slots__ to create non-GC types. I think I can just create a simple utility in _testcapi to untrack instances for this purpose. |
I have the feeling that I am misunderstanding what you refer to, but if is **untrack** what you need, there is gc.untrack() |
I'm not sure that's true... :) Although perhaps track()/untrack() functions *could* be useful to add to the gc module... but that's a separate conversation. |
Oh boy, I need to....ehem....yeah I will be back in....one second... |
In any case, you can expose: https://docs.python.org/3/c-api/gcsupport.html#c.PyObject_GC_UnTrack in the testcapi module |
I'll hold off until https://bugs.python.org/issue42039 is resolved. |
Actually, never mind. Because of the way finalization works, we need to create untracked *types* for these tests, not untracked *instances*. PR up soon. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: