-
-
Notifications
You must be signed in to change notification settings - Fork 30.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Better support for finalization with weakrefs #59733
Comments
A patch with docs and tests for the idea I suggested on python-ideas:
To repeat what I wrote there, the current issues with weakref callbacks include:
(Note that problems 2-4 also apply to using __del__ methods.) Trivial usage of the class looks like >>> class Kenny: pass
...
>>> kenny = Kenny()
>>> finalize(kenny, print, "you killed kenny!")
<finalize.finalize object at 0xffed4f2c>
>>> del kenny
you killed kenny! |
I don't quite understand the purpose of your suggestions. What can you do with it help, what you can not do with contextlib.ExitStack, atexit, __del__ method, weakref.WeakKeyDictionary or weakref.ref? I read the documentation, but the meaning eludes me. |
finalize does not "compete" with contextlib.ExitStack, atexit and Points 1 and 2 in my first message are the main points. Also, read the http://docs.python.org/py3k/reference/datamodel.html#object.\_\_del__ which also applies to weakref callbacks. Other problems with __del__:
There was actually a proposal to remove or replace __del__ methods in http://mail.python.org/pipermail/python-3000/2006-September/thread.html#3797 As for weakref callbacks, I think they are just too difficult to use correctly |
What kind of problems you solve with __del__ and weakref callbacks? For most
For point 1: global weakref.WeakKeyDictionary is good store for weak refs with global weakdict If you do not want to work at a low level, in most cases fit the above- For point 2 Antoine has noted that it is a known issue and there is a solution
Is this your point 2?
It is use case for weakrefs which break reference loops.
What you are meaning? Relying on GC is dangerous thing for resource releasing. And it even more Can you give examples, where the use of finalize necessary or more convenient |
That depends on kenny being hashable. It also surprises me a bit that it works. It seems to depend on unguaranteed I think users should have a documented way to schedule callbacks without having to
That has been languishing for 9 years. I would not hold your breath.
Yes.
I agree that explicit clean up using context managers to be strongly encouraged.
multiprocessing (which I orginally wrote) makes extensive use of finalizers, But there are various things there that depend on finalizers, and on being able |
Updated patch. get() and put() replaced by peek() and detach(). Added isalive(). Now finalizer class has no state, i.e. __slots__ == (). |
I don't understand why you have two names: finalize and Finalizer. Let's keep only one of them. isalive() could be a read-only property named "alive" instead. In _make_callback, I still think the default error reporting mechanism should be kept. It can be improved separately. |
New patch. This time I have got rid of _make_callback, and just given __call__ an ignored optional argument. |
In the latest patch, what are "broken" and "priority" for? Also, I would question why atexit is false by default. I would find it personally less surprising to be true. |
They are for a subclass used by multiprocessing. But it is premature to worry about subclassing, so I will take them out.
OK. One thing I do worry about is having the loop in the exit function to run any finalizers created during the exit function. The current implementation will run these extra finalizers at the wrong time. Fixing this could probably be done by using a dirty flag and disabling gc while running the finalizers. I wonder if it would be better to not call finalizers created during the exit function. We cannot guarantee that every finalizer created during shutdown is run, so is a best effort attempt really worth the effort? |
Updated patch. |
New patch. |
In: + except: I would write "except Exception" instead. You don't want to trap e.g. KeyboardInterrupt. For clarity, I would also add "_dirty = False" at the finalize top-level. |
Richard, do you still want to push this forward? Otherwise I'd like to finalize the patch (in the other sense ;-). |
I started to worry a bit about daemon threads. I think they can still run while atexit functions are being run.* So if a daemon thread creates an atexit finalizer during shutdown it may never be run. I am not sure how much to worry about this potential race. Maybe a lock could be used to cause any daemon threads which try to create finalizers to block.
|
Mmmh... thread switching is already blocked at shutdown: |
On 14/02/2013 3:16pm, Antoine Pitrou wrote:
But in Py_Finalize(), call_py_exitfuncs() is called *before* _Py_Finalizing is set to a non-NULL value.
|
Ah, right. But is it any different from, e.g., registering an atexit |
I agree. |
Here is an updated patch. It is only really the example in the docs which is different, plus a note about daemon threads. Antoine, do think this is ready to be committed? |
Well, assuming there are no significant changes in the code (I haven't checked), +1 for committing. |
The only (non-doc, non-comment) changes were the two one-liners you suggested in msg172077. So I will commit. |
New changeset 2e446e87ac5b by Richard Oudkerk in branch 'default': |
The changeset 2e446e87ac5b broke some buildbots at compile step. ./python -E -S -m sysconfig --generate-posix-vars
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/locale.py", line 17, in <module>
import re
File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/re.py", line 124, in <module>
import functools
File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/functools.py", line 18, in <module>
from collections import namedtuple
File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/collections/__init__.py", line 8, in <module>
__all__ += collections.abc.__all__
AttributeError: 'module' object has no attribute 'abc' http://buildbot.python.org/all/builders/x86%20RHEL%206%203.x/builds/1963 |
New changeset 186cf551dae5 by Richard Oudkerk in branch 'default': |
Your patch leaks references with subinterpreters, which was exposed by functools.singledispatch using weakref. While the actual bug is in atexit (which doesn't properly unregister on subinterpreter termination), now all uses of weakref leak. Context: http://mail.python.org/pipermail/python-dev/2013-June/126755.html PJE suggests importing atexit and registering finalize only when it's actually used. I guess this would be the easiest workaround. |
New changeset d6ad9d7468f7 by Richard Oudkerk in branch 'default': |
Done. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: