Circular weakref callbacks never called (self._x = weakref.ref(self, callback))

[gitlab-importer]

In Heptapod by bitbucket_importer on Apr 14, 2015, 19:18

Created originally on Bitbucket by jamadden (Jason Madden)

It appears that if an object holds a weak reference to itself, the callback for that reference is never called. It's as if when both the object and the reference to it become garbage at the same time, the callback isn't called.

I ran into this testing ZODB under PyPy (it used this pattern in ZODB.blob.Blob).

Here's an example of what happens under CPython:

$ /opt/local/bin/python2.7
Python 2.7.9 (default, Dec 13 2014, 15:13:49)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import weakref, sys
>>> class X(object):
...     pass
...
>>> x = X()
>>> x._ref = weakref.ref(x, lambda r: sys.stderr.write('callback\n'))
>>> del x
callback
>>>

Now, under PyPy (both 2.5.1 and a relatively recent nightly), the callback doesn't get called when the reference is deleted (but we expect that):

$ pypy
Python 2.7.9 (9c4588d731b7, Mar 23 2015, 16:20:40)
[PyPy 2.5.1 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> import weakref, sys
>>>> class X(object):
....     pass
....
>>>> x = X()
>>>> x._ref = weakref.ref(x, lambda r: sys.stderr.write('callback\n'))
>>>> del x
>>>>

No problem, we need to ask GC to do a collection. We've seen this before.

>>>> import gc
>>>> gc.collect()
0

Hmm, maybe a few collections?

>>>> for _ in range(100): _ = gc.collect()
>>>>

Nothing.

If we use a separate reference object with a different lifetime, then it is called immediately after the first GC:

>>>> x = X()
>>>> ref = weakref.ref(x, lambda r: sys.stderr.write('direct callback\n'))
>>>> del x
>>>> gc.collect()
direct callback
0
>>>>

[gitlab-importer]

In Heptapod by @arigo on Apr 14, 2015, 22:07

This works as documented by the line in the Python documentation that introduces callbacks (https://docs.python.org/2/library/weakref.html):

""" If callback is provided and not None, and the returned weakref object is still alive, the callback will be called when the object is about to be finalized"""

In other words, because the weakref object dies at the same time as the object it is a weak reference to, then the callback is not invoked. You can see the same effect in CPython by doing del x._ref; del x in this order, which does not invoke the callback.

We should investigate more in-depth what the real reason for this restriction is, and if it makes sense to weaken it somehow in PyPy...

[gitlab-importer]

In Heptapod by @arigo on Jun 30, 2015, 07:52

Added documentation in d3d07e983854.

[gitlab-importer]

In Heptapod by @cfbolz on Mar 5, 2016, 23:28

this is very hard to fix, and documented.