bpo-31033: Make traceback for cancelled asyncio tasks more useful

[cjerdonek]

When an asyncio.Task is cancelled, the exception traceback now starts with where the task was first interrupted.

Previously, the traceback only had "depth one," which wasn't too useful.

https://bugs.python.org/issue31033

[cjerdonek]

@1st1 Yury, would you also be able to review / take a look at this one? I feel this is more important than PR #19979 for 3.9 from an asyncio perspective and is also simpler. (The PR's are compatible with one another.) Thank you in advance.

[cjerdonek]

@1st1 Here's a new version of the PR after rebasing with PR #19979 and ensuring that the cancelled message isn't duplicated multiple times in the traceback. I'll show you what the output looks like in a comment after this.

I wasn't able to get the tests passing using a weakref or setting self._cancelled_exc to None like you suggested. It might not be possible in general. When a weakref is used in Task with CFuture, the exception gets deallocated here before the CancelledError ever has a chance to be constructed:

cpython/Lib/asyncio/tasks.py

Lines 360 to 363 in aa92a7c

	except BaseException as exc:
	# This may also be a cancellation.
	self.__step(exc)
	else:

Similarly, setting self._cancelled_exc to None wasn't working in certain tests for different reasons.

Can this be an optimization that's considered later?

[cjerdonek]

Here's an example to show you what the output look like. For this code:

import asyncio, sys

async def nested():
    fut = asyncio.sleep(1)
    await fut  # AWAIT #1

async def run():
    task = asyncio.create_task(nested())
    await asyncio.sleep(0)
    task.cancel('POSSIBLY LONG CANCEL MESSAGE')

    await task  # AWAIT #2

loop = asyncio.new_event_loop()
try:
    loop.run_until_complete(run())
finally:
    loop.close()

Here is before the PR:

Traceback (most recent call last):
  File "/.../cpython/test-cancel.py", line 17, in <module>
    loop.run_until_complete(run())
  File "/.../cpython/Lib/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
asyncio.exceptions.CancelledError: POSSIBLY LONG CANCEL MESSAGE

And here is after the PR:

Traceback (most recent call last):
  File "/.../cpython/test-cancel.py", line 6, in nested
    await fut  # AWAIT #1
  File "/.../cpython/Lib/asyncio/tasks.py", line 669, in sleep
    return await future
asyncio.exceptions.CancelledError: POSSIBLY LONG CANCEL MESSAGE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../cpython/test-cancel.py", line 13, in run
    await task  # AWAIT #2
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../cpython/test-cancel.py", line 17, in <module>
    loop.run_until_complete(run())
  File "/.../cpython/Lib/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
asyncio.exceptions.CancelledError

[cjerdonek]

Okay, I was able to get it working with setting self._cancelled_exc to None. This is done in _make_cancelled_error() in the Python code and in _asyncio_Future__make_cancelled_error_impl() in the C code.

[cjerdonek]

@1st1 Is there any way this can be merged by tomorrow? I addressed your first review comment (setting self._cancelled_exc to None when it's no longer needed). The other one (weakref) I don't think is compatible with the feature (which I can explain more fully).

[1st1]

Chris, this looks good and the usability improvement is huge. I've left a nit comment, feel free to address it in this PR or later, up to you. Thanks!

[1st1]

@ambv I think this needs to be included in 3.9. I'm going to merge it myself tomorrow morning if Chris doesn't. Or you can merge it as is!