module shutdown procedure based on GC

[arigo]

BPO	812369
Nosy	@loewis, @brettcannon, @arigo, @nascheme, @gpshead, @ncoghlan, @abalkin, @pitrou, @tiran, @asvetlov, @florentx, @cburroughs, @ericsnowcurrently
Superseder	bpo-18214: Stop purging modules which are garbage collected before shutdown
Files	x.txt: patch 0001-update-GC-shutdown-patch.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-07-30.18:34:23.546>
created_at = <Date 2003-09-25.10:49:56.000>
labels = ['interpreter-core', 'type-bug']
title = 'module shutdown procedure based on GC'
updated_at = <Date 2013-08-21.06:13:40.226>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2013-08-21.06:13:40.226>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2013-07-30.18:34:23.546>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2003-09-25.10:49:56.000>
creator = 'arigo'
dependencies = []
files = ['5605', '13091']
hgrepos = []
issue_num = 812369
keywords = ['patch']
message_count = 24.0
messages = ['44689', '44690', '44691', '44692', '59257', '79671', '82136', '84775', '94027', '94033', '110337', '114285', '118245', '118259', '153530', '153576', '153603', '154094', '154122', '172099', '172106', '172165', '179098', '193946']
nosy_count = 17.0
nosy_names = ['loewis', 'brett.cannon', 'arigo', 'nascheme', 'glchapman', 'gregory.p.smith', 'ncoghlan', 'belopolsky', 'pitrou', 'christian.heimes', 'andrea.corbellini', 'asvetlov', 'flox', 'cburroughs', 'BreamoreBoy', 'eric.snow', 'Stefan.Friesel']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'patch review'
status = 'closed'
superseder = '18214'
type = 'behavior'
url = 'https://bugs.python.org/issue812369'
versions = ['Python 3.2']

[arigo]

This patches changes PyImport_Cleanup() in an attempt
to make the module shutdown order more predictable in
the presence of modules that import each other. Before
it explicitely clears the globals of the modules, it
relies on the GC to try to do it more cleanly.

http://mail.python.org/pipermail/python-dev/2003-September/038238.html

To prevent objects with __del__ methods from keeping
whole modules alive I actually replace each module with
a weak reference to it in sys.modules. This allows me
to find modules that remain alive after a call to
PyGC_Collect(), and then go back to the old technique
of clearing their globals.

Note that weak references to dead cycles containing
objects with finalizers have a strange property in
Python: if you use the weak reference again you can
break the cycles, but the objects with finalizers still
won't be collected because they are in gc.garbage. As
this is exactly what occurs above, I clear the
gc.garbage list explicitely before the final
PyGC_Collect() call. I'm not sure exactly what this
might do; could it release older objects that were
never put in a module but that at some time were put in
gc.garbage and whose cycles were later broken? If so,
is it a good thing to release them now? (Would it make
sense to clear gc.garbage and call gc.collect again
from time to time to check if the objects are still in
a cycle?)

This patch does not change the behavior of module
objects clearing their globals dictionary as soon as
they are deallocated. This could be work investigating.

This patch puts weak references (actually proxies) in
sys.modules but only at shutdown time. Moure thoughts
could be given towards allowing weak references during
normal program execution as well. To do so we must
ensure that 'import' statements return the real module,
not the weak proxy object, which is more difficult than
it first seems in the presence of packages.

And finally -- this patch has not really been tested,
apart from the fact that it passes the test suite.

[loewis]

Logged In: YES
user_id=21627

I think clearing gc.garbage at shutdown time is a good idea;
gc.garbage is mostly a debugging aid, and should be empty in
production code. If it still contains objects at shutdown
time, it is IMO reasonable to give them a chance to become
collected, in case somebody broke their cycles.

[glchapman]

Logged In: YES
user_id=86307

Not sure if this is a good idea, but I wonder if the extensions
dictionary should be cleared (ie _PyImport_Fini called)
sooner, possibly even before PyImport_Cleanup. This would
allow garbage collection during PyImport_Cleanup to catch
anything a C module might have created which is in a cycle
with its module (through a bad design on my part, I recently
discovered I had created just such a cycle).

I suppose _PyImport_Fini is called when it is called because
some code during PyImport_Cleanup might import a
dynamic module, which would then create a new extensions
dictionary if _PyImport_Fini had already been run. Perhaps
a flag could be added so that _PyImport_FixupExtension
would not try to add a module's dict to extensions if Python
is currently shutting down.

[brettcannon]

Logged In: YES
user_id=357491

Just so you know, Armin, the patch did not apply cleanly; the comment
for pythonrun.c did not apply. I also need to add an extern declaration in
import.c for _PyGC_garbage for the code to compile (OS X 10.3, gcc
3.3).

[tiran]

Consider this patch for 2.6 and discuss it at the bug day.

[pitrou]

Looking at the patch, is there any reason it doesn't get rid of the
current _PyModule_Clear() implementation to replace it by a call to
PyDict_Clear() followed by PyGC_Collect()?
(the call to PyGC_Collect could be disabled while finalizing, because
there's no use calling it as many times as there are modules to be
disbanded)

The major annoyance with the current scheme is that, at interpreter
shutdown, some globals you want to rely on in your destructors suddenly
become None.

About what to do of gc.garbage at shutdown, there was another proposal
in bpo-477863.

[nascheme]

This sounds like a nice idea. The current cleanup procedure in
pythonrun.c is pretty lame since it can play havoc with __del__ methods
(e.g. if they run after globals have been cleared).

I've updated the patch to work with the current SVN head. Probably this
should get tested with big applications based on Zope, Django, etc.

[pitrou]

Retargetting, and I hope someone can take a look at the patch and give
it the green light :-)

[abalkin]

Does this patch fix bpo-1545463 by any chance? I am away from a
development box ATM and cannot test the patch myself.

[nascheme]

It should fix bpo-1545463 and running a quick test seems to show that
it does.

[BreamoreBoy]

bpo-1545463 has been closed as "won't fix", so wouldn't implementing this patch kill two birds with one stone?

[BreamoreBoy]

bpo-1545463 has been reopened with comments about being used as a stop gap. Possibly review and implementation of the patch here would be a better option, sorry it's over my head.