Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-32751: Wait for task cancellation in asyncio.wait_for() #7216

Merged
merged 3 commits into from May 29, 2018

Conversation

Projects
None yet
6 participants
@elprans
Copy link
Contributor

commented May 29, 2018

Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately. This is problematic when
fut is a Task, because it will be left running for an arbitrary
amount of time. This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:

condition = asyncio.Condition()
async with condition:
    await asyncio.wait_for(condition.wait(), timeout=0.5)

Currently, instead of raising a TimeoutError, the above code will fail
with RuntimeError: cannot wait on un-acquired lock, because
__aexit__ is reached before condition.wait() finishes its
cancellation and re-acquires the condition lock.

To resolve this, make wait_for await for the task cancellation.
The tradeoff here is that the timeout promise may be broken if the
task decides to handle its cancellation in a slow way. This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.

https://bugs.python.org/issue32751

bpo-32751: Wait for task cancellation in asyncio.wait_for()
Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately.  This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time.  This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:

    condition = asyncio.Condition()
    async with condition:
        await asyncio.wait_for(condition.wait(), timeout=0.5)

Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.

To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way.  This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
# We cannot wait on *fut* directly to make
# sure _cancel_and_wait itself is reliably cancellable.
await waiter
except futures.CancelledError:

This comment has been minimized.

Copy link
@1st1

1st1 May 29, 2018

Member

Do we need this except at all?

This comment has been minimized.

Copy link
@elprans

elprans May 29, 2018

Author Contributor

No, it's an oversight from an earlier code iteration.

waiter = loop.create_future()
cb = functools.partial(_release_waiter, waiter)
fut.add_done_callback(cb)
fut.cancel()

This comment has been minimized.

Copy link
@1st1

1st1 May 29, 2018

Member

I'd move 'fut.cancel()' into the 'try' block

@1st1

1st1 approved these changes May 29, 2018

@asvetlov
Copy link
Contributor

left a comment

I suspect if a task has a slow cancellation routine -- many timeout related assumptions will be broken.

LGTM


try:
fut.cancel()
# We cannot wait on *fut* directly to make

This comment has been minimized.

Copy link
@asvetlov

asvetlov May 29, 2018

Contributor

Nice trick!

@1st1

This comment has been minimized.

Copy link
Member

commented May 29, 2018

I suspect if a task has a slow cancellation routine -- many timeout related assumptions will be broken.

Yeah. Broken timeouts is a small price to have the rest of asyncio code working correctly. A related bug where wait_for caused an incorrect asyncio.Condition behaviour scared me.

@1st1 1st1 merged commit e2b340a into python:master May 29, 2018

7 of 9 checks passed

VSTS: Linux-PR Linux-PR_20180529.44 failed
Details
VSTS: Windows-PR Windows-PR_20180529.44 failed
Details
VSTS: Linux-PR-Coverage Linux-PR-Coverage_20180529.63 succeeded
Details
VSTS: docs docs_20180529.76 succeeded
Details
VSTS: macOS-PR macOS-PR_20180529.44 succeeded
Details
bedevere/issue-number Issue number 32751 found
Details
bedevere/news News entry found in Misc/NEWS.d
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@miss-islington

This comment has been minimized.

Copy link

commented May 29, 2018

Thanks @elprans for the PR, and @1st1 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒🤖 I'm not a witch! I'm not a witch!

miss-islington added a commit to miss-islington/cpython that referenced this pull request May 29, 2018

bpo-32751: Wait for task cancellation in asyncio.wait_for() (pythonGH…
…-7216)

Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately.  This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time.  This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:

    condition = asyncio.Condition()
    async with condition:
        await asyncio.wait_for(condition.wait(), timeout=0.5)

Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.

To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way.  This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
(cherry picked from commit e2b340a)

Co-authored-by: Elvis Pranskevichus <elvis@magic.io>
@bedevere-bot

This comment has been minimized.

Copy link

commented May 29, 2018

GH-7223 is a backport of this pull request to the 3.7 branch.

miss-islington added a commit that referenced this pull request May 29, 2018

bpo-32751: Wait for task cancellation in asyncio.wait_for() (GH-7216)
Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately.  This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time.  This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:

    condition = asyncio.Condition()
    async with condition:
        await asyncio.wait_for(condition.wait(), timeout=0.5)

Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.

To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way.  This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
(cherry picked from commit e2b340a)

Co-authored-by: Elvis Pranskevichus <elvis@magic.io>

lisroach added a commit to lisroach/cpython that referenced this pull request Sep 10, 2018

bpo-32751: Wait for task cancellation in asyncio.wait_for() (pythonGH…
…-7216)

Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately.  This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time.  This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:

    condition = asyncio.Condition()
    async with condition:
        await asyncio.wait_for(condition.wait(), timeout=0.5)

Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.

To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way.  This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.

yahya-abou-imran added a commit to yahya-abou-imran/cpython that referenced this pull request Nov 2, 2018

bpo-32751: Wait for task cancellation in asyncio.wait_for() (pythonGH…
…-7216)

Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately.  This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time.  This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:

    condition = asyncio.Condition()
    async with condition:
        await asyncio.wait_for(condition.wait(), timeout=0.5)

Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.

To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way.  This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.