OSError: handle is closed for ProcessPoolExecutor and run_in_executor

[basnijholt]

BPO	36281
Nosy	@asvetlov, @1st1, @basnijholt, @iritkatriel, @JoeKun
Superseder	bpo-39104: ProcessPoolExecutor hangs on shutdown nowait with pickling failure

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-11-05.19:48:49.526>
created_at = <Date 2019-03-13.17:36:22.212>
labels = ['3.7', '3.8', 'type-crash', 'expert-asyncio']
title = 'OSError: handle is closed for ProcessPoolExecutor and run_in_executor'
updated_at = <Date 2021-11-05.19:48:49.526>
user = 'https://github.com/basnijholt'

bugs.python.org fields:

activity = <Date 2021-11-05.19:48:49.526>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-11-05.19:48:49.526>
closer = 'iritkatriel'
components = ['asyncio']
creation = <Date 2019-03-13.17:36:22.212>
creator = 'basnijholt'
dependencies = []
files = []
hgrepos = []
issue_num = 36281
keywords = []
message_count = 3.0
messages = ['337868', '366116', '404196']
nosy_count = 5.0
nosy_names = ['asvetlov', 'yselivanov', 'basnijholt', 'iritkatriel', 'JoeKun']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '39104'
type = 'crash'
url = 'https://bugs.python.org/issue36281'
versions = ['Python 3.7', 'Python 3.8']

[basnijholt]

The following code in Python 3.7.1

import random
import concurrent.futures
import asyncio

executor = concurrent.futures.ProcessPoolExecutor()
ioloop = asyncio.get_event_loop()

async def func():
    result = await ioloop.run_in_executor(executor, random.random)
    executor.shutdown(wait=False)  # bug doesn't occur when `wait=True`

task = ioloop.create_task(func())

prints the following error:

Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.7/concurrent/futures/process.py", line 368, in _queue_management_worker
    thread_wakeup.clear()
  File "/opt/conda/lib/python3.7/concurrent/futures/process.py", line 92, in clear
    while self._reader.poll():
  File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 255, in poll
    self._check_closed()
  File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

I think this is related to https://bugs.python.org/issue34073 and https://bugs.python.org/issue34075

This happens in the Adaptive package https://adaptive.readthedocs.io/en/latest/docs.html#examples and the related issue is python-adaptive/adaptive#156

[basnijholt]

Using git bisect I've discovered the commit (b713adf) (b713adf27a) that broke the code.

I've used one script:

import sys
sys.path.append("/Users/basnijholt/Downloads/cpython/Lib/concurrent/futures/")
from random import random
from process import ProcessPoolExecutor
import asyncio

ioloop = asyncio.get_event_loop()

async def func(ioloop, executor):
    result = await ioloop.run_in_executor(executor, random)
    executor.shutdown(wait=False)  # bug doesn't occur when `wait=True`

if __name__ == "__main__":
    executor = ProcessPoolExecutor()
    task = ioloop.run_until_complete(func(ioloop, executor))

and test2.py

import pexpect
import sys

child = pexpect.spawn("python /Users/basnijholt/Downloads/cpython/test.py")
try:
    child.expect(["OSError", "AssertionError"], timeout=1)
    raise Exception
except pexpect.EOF as e:
    sys.exit(0)

Then did

git checkout master
git reset --hard 9b6c60cbce  # bad commit
git bisect start
git bisect bad
git bisect good ad2c2d380e  # good commit
git bisect run python test2.py

I will see if I can fix it.

[iritkatriel]

I don't get the error on 3.11, and I think it may have been fixed under bpo-39104.