execv (et al.) should invoke atexit handlers before executing new code

[nedbat]

BPO	16822
Nosy	@birkenfeld, @pitrou, @taleinat, @nedbat, @ericsnowcurrently, @emilyemorehouse

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-12-30.22:38:28.866>
labels = ['type-feature']
title = 'execv (et al.) should invoke atexit handlers before executing new code'
updated_at = <Date 2018-07-08.07:15:25.829>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2018-07-08.07:15:25.829>
actor = 'taleinat'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2012-12-30.22:38:28.866>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 16822
keywords = []
message_count = 8.0
messages = ['178623', '178668', '178669', '178670', '180337', '321155', '321167', '321264']
nosy_count = 7.0
nosy_names = ['georg.brandl', 'pitrou', 'taleinat', 'nedbat', 'neologix', 'eric.snow', 'emilyemorehouse']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16822'
versions = ['Python 3.4']

[nedbat]

If I register an atexit handler, and then call os.execv, the handler is not invoked before my process changes over to the new program. Shouldn't it be? My program is ending, so my atexit handlers should be invoked.

This is based on this coverage.py bug: https://bitbucket.org/ned/coveragepy/issue/43/coverage-measurement-fails-on-code If the atexit handlers were invoked as part of os.execv, it would work properly.

[pitrou]

That's a good question. Conceptually it makes sense, but I wonder if programs currently rely on os.execv not cleaning up anything: not only it doesn't call atexit handlers, but it also doesn't try to shutdown the interpreter. Which can be handy if you are using exec() in a fork() + exec() context (I think it is generally recommended to use os._exit(), not sys.exit() in a forked child).

[birkenfeld]

FTR, with C's atexit(3), the handlers are not called either on exec().

[neologix]

The first reason for not calling atexit handlers upon exec() is that
it wouldn't be async-safe anymore, and could result in deadlocks.
Also, since atexit handlers are inherited upon fork(), running atexit
handlers upon exec() could result in such handlers being called
several times - something which should definitely be avoided.

Note that the atexit documentation states that handlers will only be
called in case of "normal interpreter termination".

So I'm -1 on the change, the chance of breaking existing applications
is way too high.

[pitrou]

I agree with Charles-François, this is a too risky change.
However, we could definitely have a separate "atexec" handler, like the "atfork" handlers which are proposed in bpo-16500.

[taleinat]

we could definitely have a separate "atexec" handler

Couldn't coverage.py and similar apps can just invoke the atexit handlers before calling os.execv() or similar? If so, perhaps a mention of this in the docs would suffice?

[nedbat]

Coverage.py is registering a handler to save data before the program ends. The execv call is not in the coverage.py code, it's in the program that coverage.py is running.

[taleinat]

Seems like a decision needs to be made: Add support for atexec handlers, close this issue, or something else?

[albertz]

Note that a common use case is that you actually want to replace the current process intentionally without any atexit handlers. So changing this now would break such cases.

However, calling the atexit handlers explicitly when this is wanted should also be possible, so there should be a separate API for that. Also, this should ideally cover native atexit handlers, also uninit libraries etc, i.e. doing a proper shutdown of the process. Maybe this API can be combined with execv (et al), if the process would otherwise be in an invalid state afterwards. So sth like exit_and_execv.

I have a use case for the latter: I want to restart my app. It uses CUDA. Just doing execv keeps the GPU memory allocated, and all the CUDA state as-is, so then after the execv, I cannot use CUDA anymore. I think if it would do a proper exit, uninit all libraries including CUDA, then this should be fine. (rwth-i6/returnn#1489)