threading, signals, atexit: different execution with different versions

[cbertram]

BPO	9863
Nosy	@pitrou, @ned-deily, @bitdancer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-09-16.01:53:22.430>
created_at = <Date 2010-09-15.19:00:15.401>
labels = ['type-bug', 'invalid']
title = 'threading, signals, atexit: different execution with different versions'
updated_at = <Date 2010-09-16.01:53:22.429>
user = 'https://bugs.python.org/cbertram'

bugs.python.org fields:

activity = <Date 2010-09-16.01:53:22.429>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2010-09-16.01:53:22.430>
closer = 'r.david.murray'
components = []
creation = <Date 2010-09-15.19:00:15.401>
creator = 'cbertram'
dependencies = []
files = []
hgrepos = []
issue_num = 9863
keywords = []
message_count = 7.0
messages = ['116472', '116480', '116489', '116495', '116496', '116497', '116503']
nosy_count = 4.0
nosy_names = ['pitrou', 'ned.deily', 'r.david.murray', 'cbertram']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9863'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[cbertram]

my code works on python 2.6.2 and fails to work on python 2.6.5. What's going on here?

import atexit, sys, signal, time, threading

terminate = False
threads = []

def test_loop():
    while True:
        if terminate:
            print('stopping thread')
            break
        else:
            print('looping')
            time.sleep(1)

@atexit.register
def shutdown():
    global terminate
    print('shutdown detected')
    terminate = True
    for thread in threads:
        thread.join()

def close_handler(signum, frame):
    print('caught signal')
    sys.exit(0)

def run():
    global threads
    thread = threading.Thread(target=test_loop)
    thread.start()
    threads.append(thread)

    while True:
        time.sleep(2)
        print('main')

signal.signal(signal.SIGINT, close_handler)

if __name__ == "__main__":
    run()

python 2.6.2:
$ python halp.py
looping
looping
looping
main
looping
main
looping
looping
looping
main
looping
^Ccaught signal
shutdown detected
stopping thread

python 2.6.5:
$ python halp.py
looping
looping
looping
main
looping
looping
main
looping
looping
main
^Ccaught signal
looping
looping
looping
looping
...
looping
looping
Killed <- kill -9 process at this point

The main thread on 2.6.5 appears to never execute the atexit functions.

[bitdancer]

I can confirm this, and that it works on python2.5 (I don't have a 2.6.2 around to test). It also has the looping behavior on 2.7 and 3.2.

Given the timing, I wonder if this is the result of the fix to bpo-1722344.

(Note, I removed 2.6 because it doesn't receive bug fixes any more, only security fixes.)

[ned-deily]

I can verify that the test case terminates with 2.6.2 and 2.5.1 but does not for 2.6.6, 2.7, and 3.2.

This is indeed almost certainly due to bpo-1722344. r75749 moved wait_for_thread_shutdown into Py_Finalize just before the atexit functions are called. The previous behavior was considered a bug (see discussion at Msg66054 so, by that standard, the test program here is incorrect and the fact that it worked prior to 2.6.2 was an accident. The test can be fixed trivially of course by setting terminate in close_handler instead of the atexit handler.

[ned-deily]

A better fix for the program is to mark the test_loop thread as a daemon thread. As the threading module documentation says, by default threads started from the main thread are non-daemon threads and the "entire Python program exits when no alive non-daemon threads are left." The test case terminates properly by adding a thread.daemon = True after the threading.Thread call in run().

[cbertram]

let me preface this: I'm no expert on how this should be done ect...

The goal here was to specifically not set daemon on the test_loop thread. In an actual app test_loop would perhaps need to shutdown cleanly (save state or what have you). That said, your previous suggestion of moving the setting of terminate to the close handler (or maybe just call shutdown) works, but defeats the whole purpose of using atexit at all.

My understanding is that the main thread has ended execution due to the signal. If the signal handler called shutdown, when the test_loop thread exits, all of the atexit registered functions would execute (in this example it would mean calling the shutdown function twice). It just seems ugly now no matter what.

[cbertram]

On second thought, i'll just move forward with this new expected behavior. I understand the design decision and why it was made here in regards to waiting for threads to shutdown before atexit kicks in. My intended use for atexit just no longer works in this particular case.

[bitdancer]

I'm closing this as invalid, but I note that this kind of behavior change is not something we would normally have done in a point release if we had realized the consequences.