"yield from" kills generator on re-entry

[scoder]

BPO	14220
Nosy	@ncoghlan, @scoder, @benjaminp, @markshannon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-03-08.20:32:45.731>
created_at = <Date 2012-03-07.11:09:02.466>
labels = ['interpreter-core', 'type-bug']
title = '"yield from" kills generator on re-entry'
updated_at = <Date 2012-03-08.21:13:37.264>
user = 'https://github.com/scoder'

bugs.python.org fields:

activity = <Date 2012-03-08.21:13:37.264>
actor = 'Mark.Shannon'
assignee = 'none'
closed = True
closed_date = <Date 2012-03-08.20:32:45.731>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2012-03-07.11:09:02.466>
creator = 'scoder'
dependencies = []
files = []
hgrepos = []
issue_num = 14220
keywords = []
message_count = 7.0
messages = ['155072', '155112', '155132', '155133', '155134', '155151', '155183']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'scoder', 'benjamin.peterson', 'Mark.Shannon', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14220'
versions = ['Python 3.3']

[scoder]

Based on the existing "test_attempted_yield_from_loop" in Lib/test/test_pep380.py, I wrote this test and I wonder why it does not work:

"""
def test_attempted_reentry():
    """
    >>> for line in test_attempted_reentry(): print(line)
    g1: starting
    Yielded: y1
    g1: about to yield from g2
    g2: starting
    Yielded: y2
    g2: about to yield from g1
    g2: caught ValueError
    Yielded: y3
    g1: after delegating to g2
    Yielded: y4
    """
    trace = []
    def g1():
        trace.append("g1: starting")
        yield "y1"
        trace.append("g1: about to yield from g2")
        yield from g2()
        trace.append("g1: after delegating to g2")
        yield "y4"

    def g2():
        trace.append("g2: starting")
        yield "y2"
        trace.append("g2: about to yield from g1")
        try:
            yield from gi
        except ValueError:
            trace.append("g2: caught ValueError")
        else:
            trace.append("g1 did not raise ValueError on reentry")
        yield "y3"
    gi = g1()
    for y in gi:
        trace.append("Yielded: %s" % (y,))
    return trace
"""

In current CPython, I get this:

"""
Failed example:
for line in test_attempted_reentry(): print(line)
Expected:
g1: starting
Yielded: y1
g1: about to yield from g2
g2: starting
Yielded: y2
g2: about to yield from g1
g2: caught ValueError
Yielded: y3
g1: after delegating to g2
Yielded: y4
Got:
g1: starting
Yielded: y1
g1: about to yield from g2
g2: starting
Yielded: y2
g2: about to yield from g1
g2: caught ValueError
Yielded: y3
"""

Even though I catch the ValueError (raised on generator reentry) at the position where I run the "yield from", the outer generator (g1) does not continue to run after the termination of g2. It shouldn't normally have an impact on the running g1 that someone attempts to jump back into it, but it clearly does here.

I noticed this while trying to adapt the implementation for Cython, because the original test was one of the few failing cases and it made the code jump through the generator support code quite wildly.

[scoder]

Here is an analysis of this (less verbose) code:

    def g1():
        yield "y1"
        yield from g2()
        yield "y4"

    def g2():
        yield "y2"
        try:
            yield from gi
        except ValueError:
            pass  # catch "already running" error
        yield "y3"

    gi = g1()
    for y in gi:
        print("Yielded: %s" % (y,))

This is what it currently does:

1. g1() delegates to a new g2()

2. g2 delegates back to the g1 instance and asks for its next value

3. Python sees the active delegation in g1 and asks g2 for its next value

4. g2 sees that it's already running and throws an exception

Ok so far. Now:

5. the exception is propagated into g1 at call level 3), not at level 1)!

6. g1 undelegates and terminates by the exception

7. g2 catches the exception, yields "y3" and then terminates normally

8. g1 gets control back but has already terminated and does nothing

Effect: "y4" is not yielded anymore.

The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2.

[ncoghlan]

Added Mark Shannon to the nosy list - he's been tinkering with this area of the interpreter lately.

This definitely needs to be fixed though (even if that does mean major surgery on the implementation, up to and including the introduction of an __iter_from__ method)

[benjaminp]

Don't worry! I'll be fixing it in a moment...

[python-dev]

New changeset 3357eac1ba62 by Benjamin Peterson in branch 'default':
make delegating generators say they running (closes bpo-14220)
http://hg.python.org/cpython/rev/3357eac1ba62

[scoder]

Ah, yes, that should work in both implementations. I'll give it a try. Thanks!

[markshannon]

I've just added bpo-14230 which overlaps with this issue somewhat.