Processes in Python 3.9 exiting with code 1 when It's created inside a ThreadPoolExecutor

[Genarito]

BPO	43944
Nosy	@pitrou, @graingert, @applio, @Genarito, @jdevries3133

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-26.20:59:45.367>
labels = ['type-bug', '3.9']
title = "Processes in Python 3.9 exiting with code 1 when It's created inside a ThreadPoolExecutor"
updated_at = <Date 2022-04-08.09:35:24.244>
user = 'https://github.com/Genarito'

bugs.python.org fields:

activity = <Date 2022-04-08.09:35:24.244>
actor = 'graingert'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-04-26.20:59:45.367>
creator = 'Genarito'
dependencies = []
files = []
hgrepos = []
issue_num = 43944
keywords = []
message_count = 8.0
messages = ['391985', '396522', '398157', '398159', '398175', '398202', '398203', '416966']
nosy_count = 6.0
nosy_names = ['pitrou', 'graingert', 'davin', 'Genarito', 'jack__d', 'sicard_elda']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43944'
versions = ['Python 3.9']

Linked PRs

• gh-88110: clear concurrent.futures.thread._threads_queues after fork #101940
• gh-88110: clear concurrent.futures.thread._threads_queues after fork to avoid joining parent process' threads #126098
• [3.13] gh-88110: Clear concurrent.futures.thread._threads_queues after fork to avoid joining parent process' threads (GH-126098) #127163
• [3.12] gh-88110: Clear concurrent.futures.thread._threads_queues after fork to avoid joining parent process' threads (GH-126098) #127164

[Genarito]

I've a piece of code that submits a task to a ThreadPoolExecutor which starts a Process. I've realised that in Python 3.8 that Process finished with exit code 0. But I've updated Python to the 3.9 version and this started to finishing with exit code 1! Even when the Process executes an empty task.

Here's a minimal example:

from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor


def some_task():
    pass


def execute_error():
    p = Process(target=some_task)
    p.start()
    p.join()
    print(p.exitcode)  # This is always 1 on a ThreadPoolExecutor!!!


executor = ThreadPoolExecutor(max_workers=4)
executor.submit(execute_error)
# execute_error()  # IMPORTANT: this works correctly (exit 0)

My versions:

Ubuntu 21.04
Python 3.9.4

Note that if __execute_error is called outside the ThreadPoolExecutor it works correctly.
Running on Python 3.8.6 exitcode = 0 too.

[sicardelda]

Thank you very much for this report, Genaro. I encountered the same bug with a Process running in the context of a Django view. Downgrading to Python 3.8 also fixed the issue for me.

Versions:
python:3.9-alpine Docker image, running Python 3.9.5 and Alpine 3.13. Can also reproduce on the standard (Debian Buster based) python:3.9 image, and on Arch Linux (bare metal), all also running Python 3.9.5.

[jdevries3133]

I am working on a fix for this bug. I'm a beginner cpython contributor, so if anyone recognizes this as a fool's errand, please let me know!

[jdevries3133]

Ah never mind. @Genarito, the ThreadPoolExecutor is supposed to be used as a context manager. In your current code, the script ends and Python starts tearing itself down while execute_error is still running in a subprocess.

If you simply use the ThreadPoolExecutor to a context manager, the error goes away::

from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor


def some_task():
    pass


def execute_error():
    p = Process(target=some_task)
    p.start()
    p.join()
    print(p.exitcode)  # This is always 1 on a ThreadPoolExecutor!!!


# THIS IS THE IMPORTANT CHANGE
with ThreadPoolExecutor(max_workers=4) as executor:
    executor.submit(execute_error)

[Genarito]

Hi @jack__d, thanks for your answer and time. Unfortunately, It's still a regression, as in Python < 3.9 my example works as expected

[jdevries3133]

I've identified the first bad commit with git-bisect:

commit b61b818
Author: Kyle Stanley <aeros167@gmail.com>
Date: Fri Mar 27 15:31:22 2020 -0400

bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)

Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.

[jdevries3133]

The first bad commit was a fix for bpo-39812.

[graingert]

the problem is multiprocessing/process is calling threading._shutdown which tries to join its own thread, because concurrent.futures.thread._threads_queues contains the main thread in the subprocess

File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/multiprocessing/process.py", line 333, in _bootstrap
threading._shutdown()
File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1530, in _shutdown
atexit_call()
File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit
t.join()
File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1086, in join
raise RuntimeError("cannot join current thread")

[marmarek]

I'm hitting exactly the same issue without ThreadPoolExecutor, but with a PyQt application (that does use QThread), and also tests running under nose2. I think the issue is that concurrent.futures.thread doesn't clear _threads_queues in atfork callback. Adding this fixes the issue for me. PR coming soon.