Comprehensions in a class definition mostly cannot access class variable

[mjs0]

BPO	11796
Nosy	@rhettinger, @terryjreedy, @merwok, @voidspace, @Trundle, @florentx, @durban
Superseder	bpo-13557: exec of list comprehension fails on NameError

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-07-07.15:20:51.321>
created_at = <Date 2011-04-07.12:55:20.540>
labels = ['type-bug', 'docs']
title = 'Comprehensions in a class definition mostly cannot access class variable'
updated_at = <Date 2015-10-29.23:21:46.699>
user = 'https://bugs.python.org/mjs0'

bugs.python.org fields:

activity = <Date 2015-10-29.23:21:46.699>
actor = 'Anne Rosa Wangnick'
assignee = 'docs@python'
closed = True
closed_date = <Date 2012-07-07.15:20:51.321>
closer = 'flox'
components = ['Documentation']
creation = <Date 2011-04-07.12:55:20.540>
creator = 'mjs0'
dependencies = []
files = []
hgrepos = []
issue_num = 11796
keywords = []
message_count = 9.0
messages = ['133213', '133341', '133359', '133361', '133362', '133371', '133378', '164871', '253700']
nosy_count = 13.0
nosy_names = ['rhettinger', 'terry.reedy', 'eric.araujo', 'michael.foord', 'Trundle', 'jonathan.hartley', 'flox', 'daniel.urban', 'docs@python', 'westley.martinez', 'mjs0', 'josmiley', 'Anne Rosa Wangnick']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'needs patch'
status = 'closed'
superseder = '13557'
type = 'behavior'
url = 'https://bugs.python.org/issue11796'
versions = ['Python 3.1', 'Python 3.2', 'Python 3.3']

[mjs0]

A list comprehension or generator expression in a class definition fails with NameError if it has a condition that refers to another class variable. This doesn't occur if the class variable is used the "in" part of the expression.

The following works:

class Foo:
    x = range(3)
    y = [z for z in x]

but this doesn't:

class Foo:
    x = 3
    y = [z for z in range(5) if z < x]

The error reported is: NameError: global name 'x' is not defined

Both of these examples work in Python 2.

bpo-3692 suggests that class variables can't be referred to inside list comprehensions and gen expressions inside class definitions and that this won't be fixed, but these examples show that it is possible to refer to an outside class variable depending on usage. This is inconsistent and confusing.

The Python 2 behaviour makes much more sense. I understand that we don't want list comprehensions to leak internal variables but they should still be able to pull names from outside scopes in a consistent way.

[jonathanhartley]

Is also exhibited by other class variable being used in the 'output' clause of the list comprehension:

>>> class C:
...     x = 3
...     z = [z*x for z in range(4)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in C
  File "<stdin>", line 3, in <listcomp>

[terryjreedy]

Devs are aware that there is an exception to the general rule for the 'for' clause. There is a technical reason why the exception is possible, though I have forgotten it.

Since you already know that changing the general behavior has been rejected, are you asking that the exception be removed (for consistency) , so that your first example would fail? If so, that will be rejected also.

I am changing this to a doc issue in case you or someone else wishes to suggest a doc improvement.

The solution to the limitation on generator expressions, of course, is to write out the generator one is trying to abbreviate.

def clipper(max):
    for i in range(5):
        if i < max:
            yield i

class Foo:
    x = 3
    y = list(clipper(x))

print(Foo.y)
# [0, 1, 2]

[terryjreedy]

Title changed.
Generator expressions had the same limitation in 2.x.
All comprehensions have the same limitation in 3.x.

[rhettinger]

Devs are aware that there is an exception to the general rule
for the 'for' clause. There is a technical reason why the
exception is possible, though I have forgotten it.

It is best understood when thinking about a gexexp that
gets run long after is created:

   ge = (result_exp(loop_var) for loop_var in iter_exp)
 
The idea was to have the body of the iterator expression, iter_exp, fail early, before the generator is run and while the local context is still set=up and available:

   ge = (1/0 for i in pow('a', 'b'))

We want the TypeError for pow to be raised immediately. And if a valid expression were evaluated, we would want the body's ZeroDivisionError to be raised only when the generator is invoked using next(ge()).

In the former case, the local context is still available. In the latter case, it could be long gone.

[terryjreedy]

Thanks. I remember now: the initial iter_exp is evaluated immediately because it always *can* be, because it is only evaluated once and can only involve 'outside' names, whereas the result expression and any conditional expressions and further iteration expressions cannot be (evaluated immediately) because in general they usually involve the loop_var and are evaluated many times, each time with a different value of the loop_var (which are not available until the code is run).

The current 3.2 doc more or less says this already, though in fewer words.

To put it another way, the iter_exp either becomes or is treated like a parameter default value expression, so that

ge = (result_exp(loop_var) for loop_var in iter_exp)

is like

def __gf__(_it=iter_exp):
    for loop_var in _it:
        yield result_exp(loop_var)
ge = __gf__()
del __gf__

I wonder if something like this should be added to 5.2.8. Generator expressions, with that section moved up before the set/list/dict displays sections, and with the comprehension grammar included therein.

If one does *not* want the immediately evaluation of the iter_exp (which does not normally happen in generator functions) then, again, one should write a generator function.

I guess the real lesson is that in 3.x, while comprehensions (including g.e.'s are very similar to nested loops and conditions or generator functions with the same, they are not identical in scoping behavior and need to be thought of as their own thing and not only as syntactic sugar. This is probably easier for people who start with 3.x ;-).

Along that line, here is another replacement for the not-working example 2:

class Foo:
    x = 3
    y = []
    for z in range(5):
        if z < x:
            y.append(z)

print(Foo.y)
# [0, 1, 2]

[mjs0]

Thanks to everyone for the explanations.

I was hoping for behaviour along the lines of Python 2 (certainly not artificially blocking more cases in the name of consistency) but it doesn't look like that's going to happen. I think this is one slight and rare area where Python 3 has taken a step backwards when compared to Python 2 but I can live with it. As noted there's other ways to achieve the same thing.

The new Python 3 behaviour seems odd to me but perhaps that's just a bias due to many years of development with Python 2. A documentation improvement would help.

Thanks.

[florentx]

It looks as a duplicate of bpo-13557.
I close this one, because a doc patch is attached on the other.

[AnneRosaWangnick]

With https://bugs.python.org/issue13557 closed, this one should be reopened to amend 6.2.8 Generator expressions: "[However, the leftmost for clause is immediately evaluated] (in the local context)[, so that an error produced by it can be seen ...]"