Comprehension doc doesn't mention buggy class scope behavior

[bzip2]

BPO	35625
Nosy	@rhettinger, @serhiy-storchaka, @aaronchall, @bzip2
Superseder	bpo-26951: [doc] Unintuitive error when using generator expression in class property

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-07-19.19:00:22.255>
created_at = <Date 2018-12-31.10:48:00.465>
labels = ['3.8', 'type-bug', '3.7', 'docs']
title = "Comprehension doc doesn't mention buggy class scope behavior"
updated_at = <Date 2019-07-19.19:00:22.253>
user = 'https://github.com/bzip2'

bugs.python.org fields:

activity = <Date 2019-07-19.19:00:22.253>
actor = 'serhiy.storchaka'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-07-19.19:00:22.255>
closer = 'serhiy.storchaka'
components = ['Documentation']
creation = <Date 2018-12-31.10:48:00.465>
creator = 'bzip2'
dependencies = []
files = []
hgrepos = []
issue_num = 35625
keywords = []
message_count = 4.0
messages = ['332808', '333037', '333061', '348193']
nosy_count = 5.0
nosy_names = ['rhettinger', 'docs@python', 'serhiy.storchaka', 'Aaron Hall', 'bzip2']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = '26951'
type = 'behavior'
url = 'https://bugs.python.org/issue35625'
versions = ['Python 3.7', 'Python 3.8']

[bzip2]

The sections on list, set and dict comprehensions in the tutorial on data structures (ref. 1) state repeatedly that they are equivalent to for loops, but do not mention that this is not true in classes. In fact, the example used for nested list comprehensions (section 5.1.4) will work in a function, but not in a class. Similarly, there seems to be no mention of this scope "limitation" in the tutorial on classes (ref. 2), despite a section on scopes and namespaces (section 9.2) and another that mentions list comprehensions (section 9.10). The scope "limitation" is mentioned at the end of a section on resolution of names on a page about the execution model in the reference guide (ref. 3), and of course in various forums, where people may perhaps eventually find them after wasting time trying to figure out what they've done wrong.

If comprehensions are "equivalent" to for loops only under certain conditions (in a class, but only in a class, only one variable from outside the comprehension is accessible in the comprehension, and it must be the outermost iterable), they are not equivalent and should not be described as such. This "limitation" should be mentioned prominently wherever comprehensions are described, since both classes and comprehensions are presumably common constructs. When people read "is equivalent to" without a qualifier, they assume "is always equivalent to".

Returning to section 9.10 in ref. 2, the unique_words example is misleading because it strongly implies that nested for loops in a comprehension should work in a class. Since that's only true in some cases, the example should be qualified. More broadly, because that tutorial is about classes, the relevance of the last three sections should be revisited.

As an aside, I agree with the developers who consider this scope "limitation" a bug and not (paraphrasing) "just how the language works", since the exact same two lines of code, which depend on no other variables or functions, work in a function or module but not in a class.

1. https://docs.python.org/3/tutorial/datastructures.html
2. https://docs.python.org/3/tutorial/classes.html
3. https://docs.python.org/3/reference/executionmodel.html

[rhettinger]

We should remove the "equivalent to for-loops" wording for list comprehensions. That is a hold-over from 2.7 where it used to be true. That said, the list comprehension section is too early in the tutorial to go into scopes and generators, so a full explanation will need to be deferred.

As an aside, I agree with the developers who consider
this scope "limitation" a bug and not (paraphrasing)
"just how the language works", since the exact same two
lines of code, which depend on no other variables or
functions, work in a function or module but not in a class.

If you view classes as just another nested scope, I can see why you might think the behavior is buggy, limited, or undesirable. That however is not how the language is designed. Think about why methods have to reference class variables using "self.classvar" rather than just "classvar". When the methods run, they have access to their own locals() and to the module level globals(). To access the locals() for the class, they need use "self.classvar" or "classname.classvar". This is central to how python works . There are two separate lookup chains, one for variables (locals -> nested scopes -> globals -> __builtins__ -> NameError) and another for attributes (inst_dict -> class_dict -> parent_class_dict -> AttributeError).

Guido intentionally shifted list comprehensions to work like generator expressions. In a class scope, they behave like methods in that they have access to the outer globals but no direct access to the locals() in the class.

[serhiy-storchaka]

See also bpo-3692.

[serhiy-storchaka]

As a documentation issue this is a duplicate of bpo-26951.