Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows

[rebeccafair]

BPO	38428
Nosy	@pfmoore, @pitrou, @tjguk, @zware, @eryksun, @zooba, @applio, @rebeccafair
Files	bug.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-10-10.08:35:51.894>
labels = ['3.10', 'type-bug', '3.8', '3.9', 'OS-windows']
title = "Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows"
updated_at = <Date 2021-03-22.03:12:37.354>
user = 'https://github.com/rebeccafair'

bugs.python.org fields:

activity = <Date 2021-03-22.03:12:37.354>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2019-10-10.08:35:51.894>
creator = 'rebeccafair'
dependencies = []
files = ['48653']
hgrepos = []
issue_num = 38428
keywords = []
message_count = 5.0
messages = ['354344', '354346', '354393', '354413', '389273']
nosy_count = 8.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'davin', 'rebeccafair']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38428'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[rebeccafair]

I want to be able to Ctrl+C to exit a multiprocessing Pool.map gracefully, and have made a solution based on this Stack Overflow answer: https://stackoverflow.com/questions/11312525/catch-ctrlc-sigint-and-exit-multiprocesses-gracefully-in-python

However, this solution works on Linux, and on Windows 10 with Python 2.7, but not on Windows 10 with Python 3.7. The Ctrl+C is just ignored and I have to kill the processes manually. I've attached the minimum code required to reproduce the problem.

Is this a bug, expected, or is there a workaround? I believe it might be caused by the behaviour of threading.Condition.wait() being changed in commit 7c3e577 but I don't know enough about signalling to say any more than that

[rebeccafair]

Sorry, I didn't realise it wouldn't link to a commit automatically, the commit is here: 7c3e577

[zooba]

I think you're right about that change touching the right code, but I'm pretty sure it wasn't working before either. (Python 2.7 had totally different code, IIRC.)

But the good news is that we should be able to use code similar to what's in Modules/timemodule.c#L1878 (the pysleep() function) to also wait on the event we use to emulate signals on Windows. But perhaps there's a reason Antoine (nosied) didn't do that originally?

[eryksun]

Python 2.7 had totally different code, IIRC.

In 2.7 the wait() method of threading._Condition implements the timeout with a loop the calls time.sleep() with an increasing delay from 500 microseconds up to 50 ms, and no more than the remaining time.

But the good news is that we should be able to use code similar to
what's in Modules/timemodule.c#L1878 (the pysleep() function) to also
wait on the event we use to emulate signals on Windows.

If Python continues to use emulated condition variables, then the kernel wait in _PyCOND_WAIT_MS can easily be adapted to include the SIGINT event. However, isn't it planned at some point to remove the emulated condition-variable implementation in favor of native condition variables? If so and PyCOND_WAIT and PyCOND_TIMEDWAIT end up calling SleepConditionVariableSRW, there's no obvious way to interrupt this to support Ctrl+C and Ctrl+Break (given a custom SIGBREAK handler) in the main thread.

In particular, in Windows 8+, SleepConditionVariableSRW begins by spinning in user mode for a fixed number of cycles. Then it calls NtWaitForAlertByThreadId to block until alerted by NtAlertThreadByThreadId, which gets called internally by Wake[All]ConditionVariable. It wouldn't be kosher to call NtAlertThreadByThreadId directly in order to alert the main thread because it's an undocumented system call, plus it's poking into the internal implementation details of SleepConditionVariableSRW and WakeConditionVariable, which could leave the variable in a bad state.

[eryksun]

In bpo-29971 it was suggested to split the PyThread lock API in Windows into an "interruptible lock" API that's based on emulated condition variables and a "fast lock" API that's based on native condition variables and SRW locks. Maybe the only API change that's needed in that regard is to add PyThread_allocate_lock_ex(int interruptible), and otherwise dispatch on the lock type that's set in the PNRMUTEX.