RuntimeError after closing loop that used run_in_executor

[hniksic]

BPO	33469
Nosy	@hniksic, @asvetlov, @1st1, @miss-islington
PRs	bpo-33469: RuntimeError after closing loop that used run_in_executor #7171 [3.7] bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171) #7178 [3.6] bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171) #7179
Files	executor-cancel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-29.15:17:35.485>
created_at = <Date 2018-05-12.07:53:08.629>
labels = ['3.7', 'expert-asyncio']
title = 'RuntimeError after closing loop that used run_in_executor'
updated_at = <Date 2018-05-29.15:17:35.484>
user = 'https://github.com/hniksic'

bugs.python.org fields:

activity = <Date 2018-05-29.15:17:35.484>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-29.15:17:35.485>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2018-05-12.07:53:08.629>
creator = 'hniksic'
dependencies = []
files = ['47583']
hgrepos = []
issue_num = 33469
keywords = ['patch']
message_count = 6.0
messages = ['316420', '317888', '317889', '317925', '317943', '317972']
nosy_count = 4.0
nosy_names = ['hniksic', 'asvetlov', 'yselivanov', 'miss-islington']
pr_nums = ['7171', '7178', '7179']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33469'
versions = ['Python 3.6', 'Python 3.7']

[hniksic]

Looking at a StackOverflow question[1], I was unable to find a way to correctly close an event loop that uses run_in_executor() for long-running tasks.

The question author tried to implement the following scenario:

1. create some tasks that use run_in_executor
2. run asyncio.wait(tasks, return_when=FIRST_EXCEPTION)
3. cancel pending tasks, if any
4. close the loop and continue with non-async work

However, when there are pending tasks from wait(), a RuntimeError is raised some time after step #4. In the test programs, it happens while waiting for the program to finish. I have attached a minimal example to reproduce the issue.

The immediate cause is that a callback installed by wrap_future() notices that the underlying concurrent.futures.Future is done and calls loop.call_soon_threadsafe() to copy the result to the asyncio.Future. call_soon_threadsafe() fails when the loop is closed.

This would be reasonable behavior if not for the fact that the code explicitly cancelled the asyncio future, and awaited it to ensure that the cancellation took effect. While it is clear that asyncio cannot interrupt a function already running in an executor, it should probably detach the connection between the concurrent future and the asyncio future, to prevent this kind of error (and possibly other problems).

For example, the _call_check_cancel closure in _chain_future could remove (or disable) the done_callback installed on source after the call to source.cancel(). Since at that point we know that destination (asyncio.Future) is already canceled, there is no longer value in invoking the done callback for source (concurrent.futures.Future).

[1]
https://stackoverflow.com/q/50279522/1600898

[1st1]

Hopefully asyncio.run() in Python 3.7 will handle this case correctly.

[1st1]

Ah, I see, the callback that tracks the state of the wrapped concurrent.Future doesn't check that the loop is closed and its future has been cancelled. I think this is a bug.

[1st1]

New changeset fdccfe0 by Yury Selivanov in branch 'master':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
fdccfe0

[miss-islington]

New changeset 8d8b861 by Miss Islington (bot) in branch '3.7':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
8d8b861

[miss-islington]

New changeset a6d6bd7 by Miss Islington (bot) in branch '3.6':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
a6d6bd7