exc_info can get lost when throwing into an await or yield from

[gschaffner]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

3.8, 3.9, 3.10, 3.11, 3.12, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.11.5 (main, Aug 28 2023, 16:29:45) [GCC 13.2.1 20230801]

A clear and concise description of the bug:

if an exception is active inside a coroutine/generator (coro1/agen1) and another exception is .thrown into its await coro2/yield from agen2, then coro2/agen2 loses the original exc_info.

• a generator example:

  import sys
  
  
  def outer():
      try:
          raise RuntimeError("boom")
      except RuntimeError:  # or `__exit__`/`finally`
          print(sys.exc_info())
          yield from inner()
          # note: if you inline `inner()` here, things work fine.
          print(sys.exc_info())
  
  
  def inner():
      print(sys.exc_info())
      try:
          yield
      except ValueError:
          pass
      print(sys.exc_info())
  
  
  gen = outer()
  gen.send(None)
  try:
      gen.throw(ValueError)
  except StopIteration:
      pass

  output:

  (<class 'RuntimeError'>, RuntimeError('boom'), <traceback object at 0x7fdb6b313840>)
  (<class 'RuntimeError'>, RuntimeError('boom'), <traceback object at 0x7fdb6b313840>)
  (None, None, None)
  (<class 'RuntimeError'>, RuntimeError('boom'), <traceback object at 0x7fdb6b313840>)
  

  expected output:

  same exc_info all four times.

• two coroutine examples (using asyncio in particular, with asyncio calling throw):

  import asyncio
  import sys
  from asyncio import CancelledError
  from math import inf
  
  
  async def outer():
      try:
          raise RuntimeError("boom")
      except RuntimeError:  # or `__aexit__`/`finally`
          print(sys.exc_info())
          await clean_up_and_log()
          # note: if you inline `clean_up_and_log()` here, things work fine.
          print(sys.exc_info())
  
  
  async def clean_up_and_log():
      print(sys.exc_info())
  
      # attempt to do some clean up that we are okay skipping part of if we get cancelled
      # during it or if we already have a cancellation request.
      #
      # (to reproduce, suppose that we get a cancellation request before the optional
      # clean up step.)
      asyncio.current_task().cancel()
      try:
          await asyncio.sleep(0)
      except CancelledError:
          if sys.version_info >= (3, 11):
              asyncio.current_task().uncancel()
  
      # after the optional async clean up, log the exception.
      print(sys.exc_info())
  
  
  async def clean_up_and_log():  # this example is >= 3.11 only.
      print(sys.exc_info())
  
      # attempt to do some clean up that we want to skip if it takes too long (e.g. due to
      # a non-responsive peer) or if we get cancelled during it or if we already have a
      # cancellation request.
      try:
          async with asyncio.timeout(0.1):
              await asyncio.sleep(inf)
      except TimeoutError:
          pass
      except CancelledError:
          asyncio.current_task().uncancel()
  
      # after the optional async clean up, log the exception.
      print(sys.exc_info())
  
  
  asyncio.run(outer())

  output and expected output: same as the generator example.

this problem is a special case of GH-73773 that was not discussed or fixed as part of that issue¹.

for comparison, PyPy behaves the way I expect here.

Footnotes

1. I am not sure whether it would be preferred that I necrobump the old closed issue or file a new report here. ↩

[sobolevn]

cc @iritkatriel

[gschaffner]

GH-84871 may be related. in GH-84871, exc_info appears to be getting reset at the end of a yield from statement when it should not (#84871 (comment)). if that is what's causing GH-84871¹, then a curious (positive?) side effect of it is that it limits the scope of this issue: when exc_info gets lost in inner, it's only lost temporarily because once yield from inner() ends, exc_info will get reset (to what happens to be the correct exception for outer in the example above, but is not the correct exception for outer in GH-84871).

Footnotes

1. namely exc_info getting mistakenly reset at the end of a yield from (and in particular the full exc_info getting reset, not just exc_info[1].__context__) ↩

[gvanrossum]

What's the behavior if inner is a function that catches an exception?

Did you confirm the bug occurs on all the versions you listed (3.8 and up)?

[iritkatriel]

Thank you @gschaffner. You say it's impacting "3.8, 3.9, 3.10, 3.11, 3.12, CPython main branch" but later you mention 3.11.5 as the version. Is it impacting all the way back to 3.8?

I will look into this (but not treating as urgent for now).

[gvanrossum]

What's the behavior if inner is a function that catches an exception?

All four print() statements report RuntimeError.

Did you confirm the bug occurs on all the versions you listed (3.8 and up)?

I confirmed it.

The conclusion is that this is a long-standing bug in generators.

[gschaffner]

Did you confirm the bug occurs on all the versions you listed (3.8 and up)?

I confirmed it.

You say it's impacting "3.8, 3.9, 3.10, 3.11, 3.12, CPython main branch" but later you mention 3.11.5 as the version. Is it impacting all the way back to 3.8?

yes. sorry for any confusion! I should have left the python -VV text field in the issue template blank.