awaiting a wrapped asyncio.Task multiple times gives long, repeative tracebacks

[lilydjwg]

BPO	42682
Nosy	@asvetlov, @lilydjwg, @1st1, @iritkatriel
Superseder	bpo-45924: Incorrect traceback when future's exception is raised multiple times

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-18.09:37:02.741>
created_at = <Date 2020-12-19.08:09:24.657>
labels = ['type-feature', '3.9', 'expert-asyncio']
title = 'awaiting a wrapped asyncio.Task multiple times gives long, repeative tracebacks'
updated_at = <Date 2022-03-18.09:37:02.740>
user = 'https://github.com/lilydjwg'

bugs.python.org fields:

activity = <Date 2022-03-18.09:37:02.740>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-18.09:37:02.741>
closer = 'iritkatriel'
components = ['asyncio']
creation = <Date 2020-12-19.08:09:24.657>
creator = 'lilydjwg'
dependencies = []
files = []
hgrepos = []
issue_num = 42682
keywords = []
message_count = 4.0
messages = ['383364', '383637', '415476', '415477']
nosy_count = 4.0
nosy_names = ['asvetlov', 'lilydjwg', 'yselivanov', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '45924'
type = 'enhancement'
url = 'https://bugs.python.org/issue42682'
versions = ['Python 3.9']

[lilydjwg]

import asyncio

async def crash(key):
  raise Exception('crash!')

async def wait(fu):
  await fu

async def main():
  crasher = asyncio.create_task(crash(()))
  fs = [wait(crasher) for _ in range(10)]
  for fu in asyncio.as_completed(fs):
    try:
      await fu
    except Exception:
      import traceback
      traceback.print_exc()

if __name__ == '__main__':
  asyncio.run(main())

This code will give a very long traceback 10 times. I expect it to be short.

I'm caching the result of an async function like this:

  async def get(
    self,
    key: Hashable,
    func: Callable[[Hashable], Coroutine[Any, Any, Any]],
  ) -> Any:
    async with self.lock:
      cached = self.cache.get(key)
      if cached is None:
        coro = func(key)
        fu = asyncio.create_task(coro)
        self.cache[key] = fu

    if asyncio.isfuture(cached): # pending
      return await cached # type: ignore
    elif cached is not None: # cached
      return cached
    else: # not cached
      r = await fu
      self.cache[key] = r
      return r

It works fine, except that when there is an exception the traceback is very long.

[asvetlov]

The traceback contains frames from asyncio internal machinery, that's why the traceback is long.

IFIAK Python standard library never filters such calls, asyncio is not an exception.

On the other hand, well-known pytest library supports __tracebackhide__ = True for skipping pytest internals in test failure reports.
I'm not sure if we want to do such black magic though.

Plus, sometimes an information about internals is helpful to detect the actual source of the problem,

I have no clear vision of how to skip internal frames easily.

Yuri, maybe you have an idea?

[iritkatriel]

I don't think the OP is complaining about the internal asyncio machinery frames, but rather about the repetition of the "return f.result() # May raise f.exception()." frame.

I will paste the output of the script in the next comment.

[iritkatriel]

>>> asyncio.run(main())
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
Traceback (most recent call last):
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 6, in main
  File "C:\Users\User\src\cpython\Lib\asyncio\tasks.py", line 597, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  File "<stdin>", line 2, in wait
  [Previous line repeated 7 more times]
  File "<stdin>", line 2, in crash
Exception: crash!
>>>