asyncio.Task doesn't propagate CancelledError() exception correctly. #89553
Comments
|
I've spotted a little bug in how asyncio.CancelledError() exception is propagated inside an asyncio.Task. Since python 3.9 the asyncio.Task.cancel() method has a new 'msg' parameter, that will create an asyncio.CancelledError(msg) exception incorporating that message. The exception is successfully propagated to the coroutine the asyncio.Task is running, so the coroutine successfully gets raised an asyncio.CancelledError(msg) with the specified message in asyncio.Task.cancel(msg) method. But, once the asyncio.Task is cancelled, is impossible to retrieve that original asyncio.CancelledError(msg) exception with the message, because it seems that *a new* asyncio.CancelledError() [without the message] is raised when asyncio.Task.result() or asyncio.Task.exception() methods are called. I have the feeling that this is just wrong, and that the original message specified in asyncio.Task.cancel(msg) should be propagated even also asyncio.Task.result() is called. I'm including a little snippet of code that clearly shows this bug. I'm using python 3.9.6, in particular: |
pagliariccim
mannequin
added
3.9
|
This seems to be present in both the Python implementation as well as the accelerated C _asyncio module. It looks like that when a Task awaits a cancelled future, |
You say it's "impossible", but isn't the message accessible via the exception chain (and visible in the traceback)? One benefit of not duplicating the message on the internal call to cancel() is that it makes it easier to pinpoint which CancelledError object is associated with the user's call to cancel(), and which is associated with the call done by asyncio internals, which is a different cancellation. Another benefit is that it prevents info from being duplicated in the traceback. |
|
afaik this is intentional https://bugs.python.org/issue31033 |
|
OK, I see your point.
Is this really wanted? Sorry, but I still find this a lot confusing. Thank you for your time, M. On Thu, Oct 7, 2021 at 10:25 AM Thomas Grainger <report@bugs.python.org>
|
Re-raise asyncio.CancelledError where? (And what do you mean by "re-raise"?) Call asyncio.Task.exception() where? This isn't part of your example, so it's not clear what you mean exactly. |
|
Chris, My point is that, if we substitute that line 10, with the commented line AT LEAST should be very well specified in the docs. Regards, On Sat, Oct 9, 2021 at 2:51 PM Chris Jerdonek <report@bugs.python.org>
|
|
I still don't see you calling asyncio.Task.exception() in your new attachment... |
|
Chris, Thanks, On Sat, Oct 9, 2021 at 3:06 PM Chris Jerdonek <report@bugs.python.org>
|
|
Here's a simplification of Marco's snippet to focus the discussion. import asyncio
async def job():
# raise RuntimeError('error!')
await asyncio.sleep(5)
async def main():
task = asyncio.create_task(job())
await asyncio.sleep(1)
task.cancel('cancel job')
await task
if __name__=="__main__":
asyncio.run(main())Running this pre-Python 3.9 gives something like this-- Traceback (most recent call last):
File "test.py", line 15, in <module>
asyncio.run(main())
File "/.../python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/.../python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
concurrent.futures._base.CancelledErrorRunning this with Python 3.9+ gives something like the following. The difference is that the traceback now starts at the sleep() call: Traceback (most recent call last):
File "/.../test.py", line 6, in job
await asyncio.sleep(5)
File "/.../python3.9/asyncio/tasks.py", line 654, in sleep
return await future
asyncio.exceptions.CancelledError: cancel job
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/.../test.py", line 12, in main
await task
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/.../test.py", line 15, in <module>
asyncio.run(main())
File "/.../python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/.../python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
asyncio.exceptions.CancelledErrorUncommenting the RuntimeError turns it into this-- Traceback (most recent call last):
File "/.../test.py", line 15, in <module>
asyncio.run(main())
File "/.../python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/.../python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/.../test.py", line 12, in main
await task
File "/.../test.py", line 5, in job
raise RuntimeError('error!')
RuntimeError: error!I agree it would be a lot nicer if the original CancelledError('cancel job') could bubble up just like the RuntimeError does, instead of creating a new CancelledError at each await and chaining it to the previous CancelledError. asyncio's creation of a new CancelledError at each stage predates the PR that added the chaining, so this could be viewed as an evolution of the change that added the chaining. I haven't checked to be sure, but the difference in behavior between CancelledError and other exceptions might be explained by the following lines: Lines 242 to 250 in 3d1ca86 You can see that for exceptions other than CancelledError, the exception is propagated by calling super().set_exception(exc), whereas with CancelledError, it is propagated by calling super().cancel() again. Maybe this would even be an easy change to make. Instead of asyncio creating a new CancelledError and chaining it to the previous, asyncio can just raise the existing one. For the pure Python implementation at least, it may be as simple as making a change here, inside _make_cancelled_error(): cpython/Lib/asyncio/futures.py Lines 135 to 142 in 3d1ca86 |
|
I have a pull request for the issue. The patch is not backported because it changes existing behavior a little. I'd like to avoid a situation when third-party code works with Python 3.11+, 3.10.3+, and 3.9.11+ only. |
|
Andrew, My Best Regards, Marco On Thu, Feb 17, 2022 at 10:29 AM Andrew Svetlov <report@bugs.python.org>
|
|
Andrew, the approach I described would I feel be much better. It would result in more concise, less verbose tracebacks, as opposed to more verbose -- not just because the message won't be repeated, but also because it eliminates the unneeded creation of intermediate exceptions. It would also cause is checks to work, which is what we want since behaviorally it should be the same exception. |
|
Seems a CancelledError message can be lost also in Condition.wait(). |
|
For future reference, with Andrew's change merged above, the traceback for the example snippet in my message above: Traceback (most recent call last):
File "/home/andrew/projects/cpython/exc_traceback.py", line 14, in <module>
asyncio.run(main())
^^^^^^^^^^^^^^^^^^^
File "/home/andrew/projects/cpython/Lib/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/andrew/projects/cpython/Lib/asyncio/base_events.py", line 640, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/andrew/projects/cpython/exc_traceback.py", line 11, in main
await task
^^^^^^^^^^
File "/home/andrew/projects/cpython/exc_traceback.py", line 5, in job
await asyncio.sleep(5)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/andrew/projects/cpython/Lib/asyncio/tasks.py", line 619, in sleep
return await future
^^^^^^^^^^^^
asyncio.exceptions.CancelledError: cancel job(This is copied from Andrew's comment in the PR here: Serhiy, can you provide a sample snippet for your case with output, like I did in my message linked above? |
|
Serhiy is right, Condition.wait() has the following code: if cancelled:
raise exceptions.CancelledErrorIt swallows CancelledError exceptions from waiters and re-raises CancelledError without the cancellation message. |
|
Also Future.result() and Future.exception() can raise a CancelledError. So a CancelledError raised in a task may not contain a message passed to Task.cancel(). import asyncio
import random
async def main():
fut = asyncio.Future()
fut.cancel()
async def job():
if random.random() < 0.5:
await asyncio.sleep(2)
fut.result()
await asyncio.sleep(5)
task = asyncio.create_task(job())
await asyncio.sleep(1)
task.cancel("cancel task")
await task
asyncio.run(main())You need to catch a CancelledError raised in a coroutine and re-raise a new CancelledError with the specified cancel message if the task was cancelled. |
|
there could be multiple messages here perhaps it could be: |
|
We should really stop appending to a closed issue. Anyway, raising ExceptionGroup is backwards incompatible, since "except CancelledError" wouldn't cancel the group. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: