GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout` #98518

kumaraditya303 · 2022-10-21T12:59:07Z

This PR changes asyncio.wait_for to use asyncio.timeout as its underlying implementation. It simplifies the code and makes it easy to understand the cancellation semantics as both asyncio.timeout and asyncio.wait_for behaves similarly.

Fixes #86296
Fixes #81839
Fixes #96764

Issue: Rewrite asyncio.wait_for using asyncio.timeout #96764

kumaraditya303 · 2022-10-21T18:09:51Z

This is more of a POC at this point. I think it is worth changing wait_for to use timeout in 3.12+. This seems to fix some of the open bugs of wait_for, I'll add those tests to make sure.

cc @gvanrossum

gvanrossum

You got a bit farther than I did when I tried this, but I'm still concerned why you had to modify several tests. (Also, there's one test that fails in CI, but doesn't appear to fail when I run it locally.)

Lib/asyncio/tasks.py

Lib/test/test_asyncio/test_futures2.py

Lib/test/test_asyncio/test_waitfor.py

Lib/asyncio/tasks.py

Misc/NEWS.d/next/Library/2022-10-22-09-26-43.gh-issue-96764.Dh9Y5L.rst

Lib/test/test_asyncio/test_waitfor.py

gvanrossum

LG (one naming nit).

Lib/test/test_asyncio/test_waitfor.py

gvanrossum · 2022-10-25T18:34:13Z

I had hoped to first merge one of the alternatives that can be backported to 3.10, so we can declare this fixed in 3.10. Then we would merge this one on top of that, but only in 3.11 and main.

But we could also just do a custom fix for 3.10 based on one of @twisteroidambassador's PRs. (I wish you had linked this PR to the same issue rather than creating a new issue, since it's all related.)

gvanrossum · 2022-10-25T18:36:20Z

In particular, @twisteroidambassador has this PR: #98607

Lib/test/test_asyncio/test_futures2.py

a new task so no sleep is required

* Switch to using asyncio.timeout instead of asyncio.wait_for `asyncio.wait_for` creates a task whereas `asyncio.timeout` avoids doing this. Fallback to using `async_timeout` when the python version is too old (<3.11) `asyncio.timeout` will become the underlying implementation for `async.wait_for` in cpython 3.12 python/cpython#98518

asyncio.wait_for creates a task whereas asyncio.timeout doesn't. Fallback to a vendored version of async_timeout on Python < 3.11. async.timeout will become the underlying implementation for async.wait_for in Python 3.12: python/cpython#98518

`asyncio.wait_for` creates another tasks which leads to some race conditions in cancelation and a performance hit cpython 3.12 will change the underlying implementation of `asyncio.wait_for` to use `asyncio.wait` but that is still a long way off for many people: python/cpython#98518

Fallback to using async_timeout on older python asyncio.wait_for has some underlying problems that are only fixed in cpython 3.12. See python/cpython#98518

asyncio.wait_for has some underlying problems that are only fixed in cpython 3.12. Use async_timeout instead until the minimum supported version is 3.11+ and it can be replaced with asyncio.timeout See python/cpython#98518

async_timeout does not suffer from the same race problems as asyncio.wait_for see python/cpython#98518 for more details

wasted some time because asyncio.wait_for() was suppressing cancellations. [0][1][2] deja vu... [3] Looks like this is finally getting fixed in cpython 3.12 [4] So far away... In attempt to avoid encountering this again, let's try using asyncio.timeout in 3.11, which is how upstream reimplemented wait_for in 3.12 [4], and aiorpcx.timeout_after in 3.8-3.10. [0] python/cpython#86296 [1] https://bugs.python.org/issue42130 [2] https://bugs.python.org/issue45098 [3] kyuupichan/aiorpcX#44 [4] python/cpython#98518

`asyncio.wait_for` creates another tasks which leads to some race conditions in cancelation and a performance hit cpython 3.12 will change the underlying implementation of `asyncio.wait_for` to use `asyncio.wait` but that is still a long way off for many people: python/cpython#98518

* Switch usage of asyncio.wait_for to async_timeout `asyncio.wait_for` creates another tasks which leads to some race conditions in cancelation and a performance hit cpython 3.12 will change the underlying implementation of `asyncio.wait_for` to use `asyncio.wait` but that is still a long way off for many people: python/cpython#98518 * adjust ci

bedevere-bot added the awaiting review label Oct 21, 2022

kumaraditya303 force-pushed the waitfor branch 2 times, most recently from 45d0725 to 6c944d9 Compare October 21, 2022 13:09

gvanrossum reviewed Oct 21, 2022

View reviewed changes

Lib/asyncio/tasks.py Show resolved Hide resolved

Lib/test/test_asyncio/test_futures2.py Show resolved Hide resolved

Lib/test/test_asyncio/test_waitfor.py Show resolved Hide resolved

gvanrossum reviewed Oct 21, 2022

View reviewed changes

Lib/asyncio/tasks.py Show resolved Hide resolved

gvanrossum mentioned this pull request Oct 21, 2022

AsyncIO's wait_for can hide cancellation in a rare race condition #86296

Closed

gvanrossum reviewed Oct 22, 2022

View reviewed changes

Misc/NEWS.d/next/Library/2022-10-22-09-26-43.gh-issue-96764.Dh9Y5L.rst Outdated Show resolved Hide resolved

gvanrossum reviewed Oct 22, 2022

View reviewed changes

Lib/test/test_asyncio/test_waitfor.py Show resolved Hide resolved

gvanrossum approved these changes Oct 24, 2022

View reviewed changes

Lib/test/test_asyncio/test_waitfor.py Outdated Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting review labels Oct 24, 2022

kumaraditya303 marked this pull request as ready for review October 24, 2022 06:03

kumaraditya303 requested review from 1st1 and asvetlov as code owners October 24, 2022 06:03

kumaraditya303 added type-feature A feature request or enhancement topic-asyncio labels Oct 25, 2022

kumaraditya303 requested a review from gvanrossum October 25, 2022 17:40

akulakov reviewed Nov 3, 2022

View reviewed changes

Lib/test/test_asyncio/test_futures2.py Outdated Show resolved Hide resolved

bedevere-bot mentioned this pull request Nov 30, 2022

Rewrite asyncio.wait_for using asyncio.timeout #96764

Closed

robsdedude mentioned this pull request Dec 14, 2022

GH-86296: Fix for asyncio.wait_for() swallowing cancellation, and add tests #98607

Closed

python deleted a comment from netlify bot Dec 14, 2022

kumaraditya303 added 6 commits January 5, 2023 15:24

rewrite wait_for to use timeouts.timeout

487eac1

skip one test

794cb3a

fix test as wait_for no longer creates

13ab78f

a new task so no sleep is required

remove outdated tests

1eb0053

fix future test

c704996

more tests

bdaed9d

bdraco mentioned this pull request Mar 8, 2023

Switch to using asyncio.timeout instead of asyncio.wait_for bdraco/websockets#1

Closed

puddly mentioned this pull request Apr 4, 2023

Use asyncio.timeout instead of asyncio.wait_for zigpy/zigpy#1187

Merged

This was referenced Apr 9, 2023

Make LIFX update handle transient communication failures home-assistant/core#90891

Closed

Lifx integration with many devices frequently goes unavailable home-assistant/core#78876

Closed

This was referenced Apr 10, 2023

Switch to using async_timeout/asyncio.timeout frawau/aiolifx#71

Merged

Blacken code to fix lint CI frawau/aiolifx#73

Merged

Suppress CancelledError in zwave-js unload home-assistant/core#91222

Merged

farmio mentioned this pull request Apr 13, 2023

Use asyncio.timeout instead of asyncio.wait_for XKNX/xknx#1246

Merged

7 tasks

willingc mentioned this pull request Apr 26, 2023

GH-98388: Add tests for happy eyeballs and its internal workings #98389

Open

bdraco mentioned this pull request Jul 21, 2023

Replace asyncio.wait_for with async-timeout python-kasa/python-kasa#480

Merged

bdraco added a commit to bdraco/ha-HAP-python that referenced this pull request Jul 23, 2023

Switch timeouts to use async_timeout

b1e0a11

async_timeout does not suffer from the same race problems as asyncio.wait_for see python/cpython#98518 for more details

bdraco mentioned this pull request Jul 23, 2023

Switch timeouts to use async_timeout ikalchev/HAP-python#447

Merged

bdraco mentioned this pull request Aug 31, 2023

Switch usage of asyncio.wait_for to async_timeout JeffLIrion/python-androidtv#337

Merged

cdce8p mentioned this pull request Sep 18, 2023

TEST: Add 3.12-dev [ci] cdce8p/ha-core#53

Closed

32 tasks

cdce8p mentioned this pull request Sep 27, 2023

Fix additional test cases for Python 3.12 home-assistant/core#101006

Merged

20 tasks

bdraco mentioned this pull request Nov 8, 2023

Make check_port an async function home-assistant/supervisor#4677

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout` #98518

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout` #98518

kumaraditya303 commented Oct 21, 2022 •

edited

kumaraditya303 commented Oct 21, 2022

gvanrossum left a comment

gvanrossum left a comment

gvanrossum commented Oct 25, 2022

gvanrossum commented Oct 25, 2022

GH-96764: rewrite asyncio.wait_for to use asyncio.timeout #98518

GH-96764: rewrite asyncio.wait_for to use asyncio.timeout #98518

Conversation

kumaraditya303 commented Oct 21, 2022 • edited

kumaraditya303 commented Oct 21, 2022

gvanrossum left a comment

Choose a reason for hiding this comment

gvanrossum left a comment

Choose a reason for hiding this comment

gvanrossum commented Oct 25, 2022

gvanrossum commented Oct 25, 2022

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout` #98518

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout` #98518

kumaraditya303 commented Oct 21, 2022 •

edited