Reference count variations after getting __dict__, getattr, and gc.collect

[encukou]

The following program accesses __dict__ (!) and some attributes of two different classes, in a loop.
After some of the loops, the total reference count is exactly 2 less than usual. In this case, it's every 5th run of the outer loop:

import gc
import sys

class A:
    class_attr = None

class B:
    class_attr = None

a = A()
b = B()

for i in range(100):

    for obj in a, b:
        obj.__dict__
        for i in range(20):
            obj.class_attr

    gc.collect()
    print(sys.gettotalrefcount())

24189
24189
24187
24189
24189
24189
24189
24187
24189
24189
24189
24189
24187
...

The code is a highly simplified version running test_typing in refleak mode (-R:), where a and b correspond to typing.Tuple and typing.List. In the actual test, this happens with more classes, and more overlapping “periods”. Sometimes, the test doesn't meet our (currently rather generous) heuristic, and fails.

The behaviour is ultimately benign, but it interferes with reference leak testing. We should find out what's happening and make regrtest clear any caches and revert to a known state.

[encukou]

I'll bisect this.
Sharing in case it rings a bell for someone :)

[AlexWaygood]

I'll bisect this. Sharing in case it rings a bell for someone :)

Could look at a8b9350

[encukou]

Bisected to #106539 -- “Dematerialize instance dictionaries when possible”. That makes sense :)

Don't know if we can meaningfully control dict (de)materialization.
But it seems that it would be best to use attributes, and avoid accessing __dict__ (which needs to create a dict object).

[encukou]

I'll prepare a PR.

[encukou]

Ah, not that easy: getattr searches more than just the dict.
Maybe we need a function in operator that's like getattr but doesn't search the object's class.

@brandtbucher, any ideas would be welcome :)

[encukou]

I've spent some time reading up on the dict dematerialization implementation, and the public discussion. Now, it seems there's not much to be done about the refcount oscillation. The original proposal about avoiding __dict__ -- __dict_copy__ and __dict_update__ -- would probably need to be extended with __dict_get__ and __dict_contains__ to be useful in pickle, enum and typing (and third-party code that does similar things and wants to be fast). Sort of like an alternate Mapping API for non-materialized instance dicts, which sounds rather ridiculous.

Perhaps it'll be better to teach the tests about what kind of “leaks” are expected.
I've merged #115726 to make the behaviour easier to see in the test output.
I plan to focus on other things for a while, and return to this issue when the buildbot fails again, or in a few weeks, with a fresh brain :)

[brandtbucher]

There's a chance we may ditch dict dematerialization for a simpler scheme in the future. So this problem may resolve itself for 3.13 or 3.14.