Exceptions raised by EventLoop.call_soon_threadsafe

[bdarnell]

BPO	39651
Nosy	@vstinner, @asvetlov, @ambv, @bdarnell, @1st1, @cmeyer, @miss-islington, @mikeshardmind
PRs	bpo-39651: Fix asyncio proactor _write_to_self() #22197 [3.8] bpo-39651: Fix asyncio proactor _write_to_self() (GH-22197) #22215 [3.9] bpo-39651: Fix asyncio proactor _write_to_self() (GH-22197) #22216

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-09-12.06:53:41.452>
created_at = <Date 2020-02-16.16:41:58.945>
labels = ['release-blocker', '3.8', '3.9', '3.10', 'expert-asyncio']
title = 'Exceptions raised by EventLoop.call_soon_threadsafe'
updated_at = <Date 2020-09-12.07:11:42.380>
user = 'https://github.com/bdarnell'

bugs.python.org fields:

activity = <Date 2020-09-12.07:11:42.380>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2020-09-12.06:53:41.452>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2020-02-16.16:41:58.945>
creator = 'Ben.Darnell'
dependencies = []
files = []
hgrepos = []
issue_num = 39651
keywords = ['patch']
message_count = 10.0
messages = ['362078', '366793', '366839', '366840', '368791', '373594', '376784', '376785', '376790', '376791']
nosy_count = 8.0
nosy_names = ['vstinner', 'asvetlov', 'lukasz.langa', 'Ben.Darnell', 'yselivanov', 'cmeyer', 'miss-islington', 'mikeshardmind']
pr_nums = ['22197', '22215', '22216']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39651'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[bdarnell]

Proactor and selector event loops behave differently when call_soon_threadsafe races with a concurrent call to loop.close(). In a selector event loop, call_soon_threadsafe will either succeed or raise a RuntimeError("Event loop is closed"). In a proactor event loop, it could raise this RuntimeError, but it can also raise an AttributeError due to an unguarded access to self._csock.

cpython/Lib/asyncio/proactor_events.py

Lines 785 to 787 in 1ed6161

	def _write_to_self(self):
	try:
	self._csock.send(b'\0')

Comments in BaseSelectorEventLoop._write_to_self indicate that this is deliberate, so the csock is not None check here should probably be copied to the proactor event loop version.

cpython/Lib/asyncio/selector_events.py

Lines 129 to 136 in 1ed6161

	def _write_to_self(self):
	# This may be called from a different thread, possibly after
	# _close_self_pipe() has been called or even while it is
	# running. Guard for self._csock being None or closed. When
	# a socket is closed, send() raises OSError (with errno set to
	# EBADF, but let's not rely on the exact error code).
	csock = self._csock
	if csock is not None:

I'd also accept an answer that the exact behavior of this race is undefined and it's up to the application to either arrange for all calls to call_soon_threadsafe to stop before closing the loop. However, I've had users of Tornado argue that they use the equivalent of call_soon_threadsafe in contexts where this coordination would be difficult, and I've decided that tornado's version of this method would never raise, even if there is a concurrent close. So if asyncio declines to specify which exceptions are allowed in this case, tornado will need to add a blanket `except Exception:` around calls to call_soon_threadsafe.

[ambv]

Good catch. We should fix this for Python 3.8.3.

[cmeyer]

Is this related to bpo-39010 too?

[bdarnell]

No, this is unrelated to bpo-39010.

[ambv]

This sadly missed 3.8.3 but I want this addressed for 3.8.4.

[ambv]

We have trouble finding a Windows expert with available time to address this :/ This is missing 3.8.4 as well.

[vstinner]

New changeset 1b0f0e3 by Victor Stinner in branch 'master':
bpo-39651: Fix asyncio proactor _write_to_self() (GH-22197)
1b0f0e3

[vstinner]

Thanks for the bug report Ben Darnell, it's now fixed.

[miss-islington]

New changeset 530d110 by Miss Islington (bot) in branch '3.8':
bpo-39651: Fix asyncio proactor _write_to_self() (GH-22197)
530d110