Mysterious atexit fail

[techtonik]

BPO	16718
Nosy	@amauryfa, @iritkatriel
Files	wow.py wy.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-16.22:45:12.996>
created_at = <Date 2012-12-18.20:35:40.650>
labels = ['interpreter-core', 'library']
title = 'Mysterious atexit fail'
updated_at = <Date 2021-04-16.22:45:12.996>
user = 'https://bugs.python.org/techtonik'

bugs.python.org fields:

activity = <Date 2021-04-16.22:45:12.996>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-16.22:45:12.996>
closer = 'iritkatriel'
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2012-12-18.20:35:40.650>
creator = 'techtonik'
dependencies = []
files = ['28353', '28354']
hgrepos = []
issue_num = 16718
keywords = []
message_count = 15.0
messages = ['177707', '177708', '177709', '177710', '177716', '177724', '177768', '177773', '177777', '177785', '177790', '177791', '177802', '177858', '387053']
nosy_count = 4.0
nosy_names = ['amaury.forgeotdarc', 'techtonik', 'sbt', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue16718'
versions = ['Python 2.6', 'Python 2.7']

[techtonik]

Weird bug. Attached are two files: wow.py and wy.py
When wow.py is executed - it fails with single ImportError.
But when wy.py is executed (which is a simple "import wow" statement) it fails with one ImportError and TypeError exceptions.

[techtonik]

...one ImportError and *two* TypeError exceptions.

[techtonik]

>py wy.py
Traceback (most recent call last):
  File "wy.py", line 1, in <module>
    import wow
  File "E:\scons\wow.py", line 12, in <module>
    import fail
ImportError: No module named fail
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "E:\scons\wow.py", line 8, in _clean
    cleanlist = [ c for c in _Cleanup if c ]
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "E:\scons\wow.py", line 8, in _clean
    cleanlist = [ c for c in _Cleanup if c ]
TypeError: 'NoneType' object is not iterable

[serhiy-storchaka]

It's because _Cleanup is None (as all global variables) when atexit registered functions executed. Don't use globals in such code.

[techtonik]

But why _Cleanup is not None when wow.py is executed standalone?

[sbt]

When you run wy.py the wow module gets partially imported, and then garbage collected because it fails to import successfully. The destructor for the module replaces values in the module's __dict__ with None. So when the cleanup function runs you get the unexpected error.

When you run wow.py directly, wow (i.e. the main module) will not be garbage collected, so _Cleanup is never replaced by None.

[techtonik]

This is not repeatable in Python 3. Is it possible to fix it for Python 2 as well?

Is it possible to postpone registration until the import is finished successfully? Or at least give atexit handler a chance to run before global variable stack is purged? Or destroy atexit handler if its namespace is purged?

[amauryfa]

Infortunately, the behaviour is the same in 2.7 and 3.2, and only changes with 3.3, which means that this is probably related to the new implementation of the import system.
This won't be backported.

[It would be interesting to understand why there is a difference, btw... is a reference to the module held somewhere?]

A possible workaround for such module-level calls atexit.register() is to ensure that used globals are captured in the function namespace, a bit like some __del__ methods:

def _clean(_Cleanup=_Cleanup):
    ....

[amauryfa]

OK, found the difference between 3.2 and 3.3:
The ImportError exception is still alive when atexit handlers are called, with its __traceback__, and all local variables in Python frames.
And since 3.3 the import system is built with Python functions...

More exactly, I could find the incomplete 'wow' module in sys.last_value.__traceback__.tb_next.tb_frame.f_back.f_back.f_back.f_locals['module']

But all this is not really related to the current issue.
Things should be better in the future, when modules are cleared with true garbage collection.

[sbt]

Things should be better in the future, when modules are cleared with true
garbage collection.

When is this future of which you speak?

I am not sure whether it would affect performance, but a weakrefable subclass of dict could be used for module dicts. Then the module destructor could just save the module's dict in a WeakValueDictionary keyed by the id (assuming we are not yet shutting down). At shutdown the saved module dicts could be purged by replacing all values with None.

Or maybe something similar is possible without using a dict subclass.

[amauryfa]

See bpo-812369 for the shutdown procedure and modules cleanup.

[sbt]

See bpo-812369 for the shutdown procedure and modules cleanup.

I am aware of that issue, but the original patch is 9 years old. Which is why I ask if/when it will actually happen.

[amauryfa]

I don't know, you should probably ask there. One blocker is to make builtin and extension modules participate in GC, search "PEP-3121" to see the many intermediate issues.

[sbt]

Perhaps the simplest thing would be to stop doing anything special when a module is garbage collected: the garbage collector can take care of any orphaned ref-cycles involving the module dict. Then at shutdown the remaining modules in sys.modules could have their dicts "purged" in the old way.

This would be orthogonal to bpo-812369. In fact Armin's original post says that this is a change worth investigating, though his patch does not do it.

[iritkatriel]

Since this was fixed since 3.3, is there a reason why this issue is still open?

I don't get the error anymore:

C:\Users\User\src\cpython-dev>python.bat wow.py
Running Release|x64 interpreter...
Traceback (most recent call last):
  File "C:\Users\User\src\cpython-dev\wow.py", line 12, in <module>
    import fail
ModuleNotFoundError: No module named 'fail'

C:\Users\User\src\cpython-dev>python.bat wy.py
Running Release|x64 interpreter...
Traceback (most recent call last):
  File "C:\Users\User\src\cpython-dev\wy.py", line 1, in <module>
    import wow
  File "C:\Users\User\src\cpython-dev\wow.py", line 12, in <module>
    import fail
ModuleNotFoundError: No module named 'fail'