Incorrect stack traces when re-entering a generator/coroutine stack via .throw()

[njsmith]

BPO	29590
Nosy	@njsmith, @cjerdonek, @markshannon, @miss-islington
PRs	bpo-29590: fix stack trace for gen.throw() with yield from #19896 [3.9] bpo-29590: fix stack trace for gen.throw() with yield from (GH-19896) #21416 [3.9] bpo-29590: fix stack trace for gen.throw() with yield from (GH-19896) #22106
Files	weird-throw-stack.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-08.16:21:05.701>
created_at = <Date 2017-02-17.14:23:48.590>
labels = ['interpreter-core', '3.7', '3.8', '3.9']
title = 'Incorrect stack traces when re-entering a generator/coroutine stack via .throw()'
updated_at = <Date 2021-12-08.16:21:05.700>
user = 'https://github.com/njsmith'

bugs.python.org fields:

activity = <Date 2021-12-08.16:21:05.700>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-08.16:21:05.701>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2017-02-17.14:23:48.590>
creator = 'njs'
dependencies = []
files = ['46644']
hgrepos = []
issue_num = 29590
keywords = ['patch']
message_count = 5.0
messages = ['288010', '368012', '369417', '373404', '376414']
nosy_count = 4.0
nosy_names = ['njs', 'chris.jerdonek', 'Mark.Shannon', 'miss-islington']
pr_nums = ['19896', '21416', '22106']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue29590'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[njsmith]

The attached script sets up a stack of generators calling each other via 'yield from': f yields from g yield from h. Then the generator at the bottom of the stack yields.

Before they yield, sys._getframe shows the expected stack (or if you put in traceback.print_stack() you get the same thing).

After they yield, it depends: if the generator is resumed via 'gen.send(None)', then the stack looks sensible. But if the generator is resumed via 'gen.throw(...)', then the stack is weird: Objects/genobject.c:_gen_throw implements 'yield from' in an weird manual way, where it first directly resumes the innermost generator frame, and then propagates any result to the next generator frame, etc. So the output I get from the sample script is:

~$ python3.6 weird-throw-stack.py
-- f before yielding --
f
<module>
-- g before yielding --
g
f
<module>
-- h before yielding --
h
g
f
<module>
-- h after yielding --
h
<module>
-- g after yielding --
g
<module>
-- f after yielding --
f
<module>

This causes problems for stack-based profiling (vmprof/vmprof-python#117), debuggers, and other tools that need to introspect the stack.

[cjerdonek]

I proposed a PR to fix this: #19896

[cjerdonek]

I just filed a related issue to this that's also similar to bpo-29587:
https://bugs.python.org/issue40694

[markshannon]

New changeset 8b33961 by Chris Jerdonek in branch 'master':
bpo-29590: fix stack trace for gen.throw() with yield from (bpo-19896)
8b33961

[miss-islington]

New changeset e92219d by Miss Islington (bot) in branch '3.9':
bpo-29590: fix stack trace for gen.throw() with yield from (GH-19896)
e92219d