-
-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-33694: Fix race condition on proactor recv() #7498
Conversation
The cancellation of an overlapped WSARecv() has a race condition which causes data loss because of the current implementation of proactor in asyncio. No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport to work around the race condition. Remove the optimized recv_into() implementation to get simple implementation of pause_reading() using the single _pending_data attribute.
I ran manually race.py of https://bugs.python.org/issue33694:
This script didn't detect any regression, whereas these two tests always reproduced the race condition in a reliable way. |
Lib/asyncio/proactor_events.py
Outdated
nbytes = len(data) | ||
if nbytes: | ||
try: | ||
buf = self._protocol.get_buffer(-1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should pass nbytes
to get_buffer()
. Also, get_buffer()
can return a smaller buffer than requested. So you better should use sslproto._feed_data_to_bufferred_proto
helper to make sure that all of the received data is passed to the BufferedProtocol
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you better should use sslproto._feed_data_to_bufferred_proto helper
Ok, done.
Lib/asyncio/proactor_events.py
Outdated
return | ||
|
||
if isinstance(self._protocol, protocols.BufferedProtocol): | ||
try: | ||
sslproto._feed_data_to_bufferred_proto(self._protocol, data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd also move _feed_data_to_bufferred_proto
helper to asyncio/protocols.py
so that this is less confusing for whoever reads the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, done.
@@ -459,6 +459,8 @@ def test_dont_pause_writing(self): | |||
self.assertFalse(self.protocol.pause_writing.called) | |||
|
|||
|
|||
@unittest.skip('FIXME: bpo-33694: these tests are too close ' | |||
'to the implementation and should be refactored or removed') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep.
I tested manually the latest change ("Move _feed_data_to_bufferred_proto() to protocols") on my Windows VM:
Each of these tests always reproduced the race condition with the fix. Note: test_asyncio pass as well ;-) |
Here, a tired but happy smiley face from me: 😍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Proactor requires an alternative way of thinking.
Looks good, the implementation is cleaner than before.
Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.7. |
The cancellation of an overlapped WSARecv() has a race condition which causes data loss because of the current implementation of proactor in asyncio. No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport to work around the race condition. Remove the optimized recv_into() implementation to get simple implementation of pause_reading() using the single _pending_data attribute. Move _feed_data_to_bufferred_proto() to protocols.py. Remove set_protocol() method which became useless. (cherry picked from commit 79790bc) Co-authored-by: Victor Stinner <vstinner@redhat.com>
GH-7499 is a backport of this pull request to the 3.7 branch. |
The cancellation of an overlapped WSARecv() has a race condition which causes data loss because of the current implementation of proactor in asyncio. No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport to work around the race condition. Remove the optimized recv_into() implementation to get simple implementation of pause_reading() using the single _pending_data attribute. Move _feed_data_to_bufferred_proto() to protocols.py. Remove set_protocol() method which became useless. (cherry picked from commit 79790bc) Co-authored-by: Victor Stinner <vstinner@redhat.com>
I didn't look at the code but fyi in passing: "bufferred" is misspelled |
The cancellation of an overlapped WSARecv() has a race condition
which causes data loss because of the current implementation of
proactor in asyncio.
No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport
to work around the race condition.
Remove the optimized recv_into() implementation to get simple
implementation of pause_reading() using the single _pending_data
attribute.
https://bugs.python.org/issue33694