yield in list comprehensions possibly broken in 3.0

[erickt]

BPO	3267
Nosy	@brettcannon, @benjaminp
Superseder	bpo-10544: yield expression inside generator expression does nothing

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-07-03.17:30:59.207>
created_at = <Date 2008-07-03.06:28:18.781>
labels = ['interpreter-core', 'type-bug']
title = 'yield in list comprehensions possibly broken in 3.0'
updated_at = <Date 2010-12-08.20:12:02.175>
user = 'https://bugs.python.org/erickt'

bugs.python.org fields:

activity = <Date 2010-12-08.20:12:02.175>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2008-07-03.17:30:59.207>
closer = 'brett.cannon'
components = ['Interpreter Core']
creation = <Date 2008-07-03.06:28:18.781>
creator = 'erickt'
dependencies = []
files = []
hgrepos = []
issue_num = 3267
keywords = []
message_count = 3.0
messages = ['69168', '69195', '69210']
nosy_count = 3.0
nosy_names = ['brett.cannon', 'benjamin.peterson', 'erickt']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = '10544'
type = 'behavior'
url = 'https://bugs.python.org/issue3267'
versions = ['Python 3.0']

[erickt]

This may be a known consequence of python 3.0, but I couldn't find any
reference to it, nor a test case that covers it. Here's a valid use of yield
in 2.5.1:

>>> def foo():
...   x=[(yield x) for x in 1,2,3]
...   yield 5
...   yield x
>>> x=foo()
>>> x.next()
1
>>> x.send(6)
2
>>> x.send(7)
3
>>> x.send(8)
5
>>> x.send(9)
[6, 7, 8]
>>> x.send(10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

But in python 3.0, this code results in:

>>> def foo():
...   x=[(yield x) for x in (1,2,3)]
...   yield 5
...   yield x
>>> x=foo()
>>> next(x)
5
>>> x.send(6)
<generator object <listcomp> at 0x3678f0>
>>> x.send(7)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Looking further, it seems that this is a comprehension:

>>> def foo(): [(yield 5)]
>>> type(foo())
<class 'generator'>

Whereas this is not:

>>> def foo(): [(yield 5) for x in range(3)]
>>> type(foo())
<class 'NoneType'>

Is this expected behavior?

[benjaminp]

This is because list comprehensions are implemented as functions in 3.0
so their scope is not leaked out to the rest of the function.

Here is the bytecode for 2.6:
>>> dis.dis(f)
  2           0 BUILD_LIST               0
              3 DUP_TOP             
              4 STORE_FAST               0 (_[1])
              7 LOAD_CONST               5 ((1, 2, 3))
             10 GET_ITER            
        >>   11 FOR_ITER                14 (to 28)
             14 STORE_FAST               1 (x)
             17 LOAD_FAST                0 (_[1])
             20 LOAD_FAST                1 (x)
             23 YIELD_VALUE         
             24 LIST_APPEND         
             25 JUMP_ABSOLUTE           11
        >>   28 DELETE_FAST              0 (_[1])
             31 STORE_FAST               1 (x)

3 34 LOAD_CONST 4 (5)
37 YIELD_VALUE
38 POP_TOP

4 39 LOAD_FAST 1 (x)
42 YIELD_VALUE
43 POP_TOP
44 LOAD_CONST 0 (None)
47 RETURN_VALUE

and here it is for 3.0:
>>> dis.dis(f)
  2           0 LOAD_CONST               1 (<code object <listcomp> at
0x740770, file "<stdin>", line 2>) 
              3 MAKE_FUNCTION            0 
              6 LOAD_CONST               6 ((1, 2, 3)) 
              9 GET_ITER             
             10 CALL_FUNCTION            1 
             13 STORE_FAST               0 (x) 

3 16 LOAD_CONST 5 (5)
19 YIELD_VALUE
20 POP_TOP

4 21 LOAD_FAST 0 (x)
24 YIELD_VALUE
25 POP_TOP
26 LOAD_CONST 0 (None)
29 RETURN_VALUE

[brettcannon]

Yes, this change in semantics is expected. Closing as "won't fix".