[doc] Unintuitive error when using generator expression in class property

[frewsxcv]

BPO	26951
Nosy	@mdickinson, @stevendaprano, @serhiy-storchaka, @frewsxcv, @zhangyangyu, @Vgr255

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-05-04.14:34:32.938>
labels = ['3.11', 'type-bug', '3.9', '3.10', 'docs']
title = '[doc] Unintuitive error when using generator expression in class property'
updated_at = <Date 2021-12-12.00:51:42.440>
user = 'https://github.com/frewsxcv'

bugs.python.org fields:

activity = <Date 2021-12-12.00:51:42.440>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2016-05-04.14:34:32.938>
creator = 'corey'
dependencies = []
files = []
hgrepos = []
issue_num = 26951
keywords = []
message_count = 5.0
messages = ['264819', '264821', '264830', '264845', '348191']
nosy_count = 7.0
nosy_names = ['mark.dickinson', 'steven.daprano', 'docs@python', 'serhiy.storchaka', 'corey', 'xiang.zhang', 'abarry']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26951'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[frewsxcv]

class A:
    B = range(10)
    C = frozenset([4, 5, 6])
    D = list(i for i in B)
    E = list(i for i in B if i in C)

coreyf@frewbook-pro /tmp [1]> python3 a.py
Traceback (most recent call last):
  File "a.py", line 1, in <module>
    class A:
  File "a.py", line 5, in A
    E = list(i for i in B if i in C)
  File "a.py", line 5, in <genexpr>
    E = list(i for i in B if i in C)
NameError: name 'C' is not defined

Why should I be able to access B but not C?

[Vgr255]

Using a simple metaclass shows that definition order is not at fault here:

>>> class PrintOrder(dict):
...   def __setitem__(self, item, value):
...     print(item, value)
...     super().__setitem__(item, value)
...
>>> class Show(type):
...   def __prepare__(name, bases): return PrintOrder()
...
>>> class A(metaclass=Show):
...   B = range(10)
...   C = frozenset([4, 5, 6])
...   D = list(i for i in B)
...   E = list(i for i in B if i in C)
...
__module__ __main__
__qualname__ A
B range(0, 10)
C frozenset({4, 5, 6})
D [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in A
  File "<stdin>", line 5, in <genexpr>
NameError: name 'C' is not defined

However, the following works:

>>> def local_definition():
...   F = frozenset([4, 5, 6])
...   class A(metaclass=Show):
...     B = range(10)
...     C = frozenset([4, 5, 6])
...     D = list(i for i in B)
...     E = list(i for i in B if i in F)
...   return A
...
>>> local_definition()
__module__ __main__
__qualname__ local_definition.<locals>.A
B range(0, 10)
C frozenset({4, 5, 6})
D [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
E [4, 5, 6]
<class '__main__.local_definition.<locals>.A'>

Sounds like either an inconsistency between 'for' and 'if' parts of generator expressions, or something weird about the way generator expressions work that I'm unaware of.

Furthermore, this isn't documented in the tutorial, the functional programming How-To, the 2.4 release notes or the PEP-289:

https://docs.python.org/3/tutorial/classes.html#generator-expressions
https://docs.python.org/3/howto/functional.html#generator-expressions-and-list-comprehensions
https://docs.python.org/3/whatsnew/2.4.html#pep-289-generator-expressions
https://www.python.org/dev/peps/pep-0289/

[stevendaprano]

On snap! Nicely found! This seems to have something to do with the way generator expressions are run inside their own scope.

In Python 2.7, a list comprehension in a class sees the locals correctly:

py> class K:
... a = 2; x = [a+i for i in range(a)]
...
py> K.x
[2, 3]

But change the list comp to a generator expression, and the situation is different:

py> class K:
...     a = 2; x = list(i for i in range(a))  # Okay
...     b = 2; y = list(b+i for i in range(a))  # Raises
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in K
  File "<stdin>", line 3, in <genexpr>
NameError: global name 'b' is not defined

In Python 3, list comps use the same sort of temporary scope as genexprs, and sure enough, the list comp fails with the same error as the generator expression.

[mdickinson]

The outer for loop in a generator expression is evaluated immediately; everything after that is evaluated lazily. This was a deliberate design choice: see https://www.python.org/dev/peps/pep-0289/#early-binding-versus-late-binding for some rationale. This explains why you're able to access "B". The inability to access "C" is due to the usual rule for Python scope resolution: class scopes are skipped when examining nested scopes.

The early evaluation of the outermost for is covered in the docs, here: https://docs.python.org/2/reference/expressions.html#generator-expressions

"""
However, the leftmost for clause is immediately evaluated, so that an error produced by it can be seen before any other possible error in the code that handles the generator expression.
"""

So this behaviour is by design, though there may be a doc issue here. Changing components and versions accordingly.

[serhiy-storchaka]

See also bpo-3692.

[LLyaudet]

Hello,
I think I have a similar issue :

laurent@laurent-GL73-8SD:~$ python3
Python 3.10.7 (main, Nov  2 2022, 18:49:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Test:
...     def __init__(self):
...         pass
... 
>>> class Magic:
...     TEST = Test()
...     TEST.ID_TO_LABEL = {1: "foo", 2: "bar"}
...     TEST.CODE_TO_ID = {"coo": 1, "car": 2}
...     TEST.CODE_TO_ID2 = {code: some_id for code, some_id in TEST.CODE_TO_ID.items()}
...     TEST.CODE_TO_LABEL = {code: TEST.ID_TO_LABEL[some_id] for code, some_id in TEST.CODE_TO_ID.items()}
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 6, in Magic
  File "<stdin>", line 6, in <dictcomp>
NameError: name 'TEST' is not defined. Did you mean: 'Test'?
>>> 
laurent@laurent-GL73-8SD:~$ python3
Python 3.10.7 (main, Nov  2 2022, 18:49:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Test:
...     def __init__(self):
...         pass
... 
>>> TEST = Test()
>>> TEST.ID_TO_LABEL = {1: "foo", 2: "bar"}
>>> TEST.CODE_TO_ID = {"coo": 1, "car": 2}
>>> TEST.CODE_TO_LABEL = {code: TEST.ID_TO_LABEL[some_id] for code, some_id in TEST.CODE_TO_ID.items()}
>>> 
>>> 
laurent@laurent-GL73-8SD:~$ python3
Python 3.10.7 (main, Nov  2 2022, 18:49:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Test:
...     def __init__(self):
...         pass
... 
>>> class Magic:
...     TEST = Test()
...     TEST.ID_TO_LABEL = {1: "foo", 2: "bar"}
...     TEST.CODE_TO_ID = {"coo": 1, "car": 2}
...     TEST.CODE_TO_ID2 = {code: some_id for code, some_id in TEST.CODE_TO_ID.items()}
...     TEST.CODE_TO_LABEL = {code: Magic.TEST.ID_TO_LABEL[some_id] for code, some_id in TEST.CODE_TO_ID.items()}
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 6, in Magic
  File "<stdin>", line 6, in <dictcomp>
NameError: name 'Magic' is not defined
>>> 

I don't think it is a doc bug. It belongs to interpreter-core.
There is no use case reason that explains that some code at main level works, and the same code in class definition crash.
I see you will probably not change that, and the work around is simply to create TEST object on main level then "inject" it as constant in the defined class.
But it creates an useless difficulty to the programmer and a WTF effect.
Even if it is "by design".
Please consider improving this.

Best regards,
Laurent Lyaudet

[LLyaudet]

The following equivalent code works :

laurent@laurent-GL73-8SD:~$ python3
Python 3.10.7 (main, Nov  2 2022, 18:49:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Test:
...     def __init__(self):
...         pass
... 
>>> class Magic:
...     TEST = Test()
...     TEST.ID_TO_LABEL = {1: "foo", 2: "bar"}
...     TEST.CODE_TO_ID = {"coo": 1, "car": 2}
...     TEST.CODE_TO_ID2 = {code: some_id for code, some_id in TEST.CODE_TO_ID.items()}
...     TEST.CODE_TO_LABEL = {}
...     for code, some_id in TEST.CODE_TO_ID.items():
...        TEST.CODE_TO_LABEL[code] = TEST.ID_TO_LABEL[some_id]
... 

It clearly shows that the implicit promise that dict comprehension is an equivalent way to replace a for loop is broken.