Missing nested scope vars in class scope (bis)

[arigo]

BPO	19979
Nosy	@arigo, @terryjreedy, @ncoghlan, @taleinat, @serhiy-storchaka
Superseder	bpo-9226: erroneous behavior when creating classes inside a closure

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-07-25.18:38:41.694>
created_at = <Date 2013-12-13.21:31:17.377>
labels = ['interpreter-core', 'type-feature']
title = 'Missing nested scope vars in class scope (bis)'
updated_at = <Date 2018-07-25.18:38:41.691>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2018-07-25.18:38:41.691>
actor = 'taleinat'
assignee = 'none'
closed = True
closed_date = <Date 2018-07-25.18:38:41.694>
closer = 'taleinat'
components = ['Interpreter Core']
creation = <Date 2013-12-13.21:31:17.377>
creator = 'arigo'
dependencies = []
files = []
hgrepos = []
issue_num = 19979
keywords = []
message_count = 8.0
messages = ['206149', '206875', '218778', '218779', '218783', '245558', '314262', '322382']
nosy_count = 5.0
nosy_names = ['arigo', 'terry.reedy', 'ncoghlan', 'taleinat', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '9226'
type = 'enhancement'
url = 'https://bugs.python.org/issue19979'
versions = ['Python 3.5']

[arigo]

This is a repeat of the old bpo-532860: "NameError assigning to class in a func". It is about class statements' variable lookups, which has different behavior at module level or in a nested scope:

def f(n):
    class A:
        n = n     # doesn't work, tries to look up 'n' as a global

The point of repeating this very old issue is the much more recent bpo-17853: "Conflict between lexical scoping and name injection in __prepare__". This was a slightly different problem, but resolved by adding the exact opcode that would be needed to fix the old issue too: LOAD_CLASSDEREF. It's an opcode which looks in the locals dict for a name, and if not found, falls back to freevars instead of to globals.

The present bug report is here to argue that this new opcode should be used a bit more systematically by the compiler. Contrary to the conclusions reached in the very old bug report, I argue that nowadays it would seem reasonable to expect that the previous example should work. By no means is it an essential issue in my opinion, but this would probably be a slight simplification and prevent corner-case surprizes. Moreover it probably leads to a simplification in the compiler: no need to track which variables are local or not in class bodies --- instead just compile the loading of 'n' above as a LOAD_CLASSDEREF without worrying about the fact that the same 'n' is also assigned to in the same scope.

[terryjreedy]

As near as I can tell, "class A: n = n" currently works the same at module and nested scope, the latter with or without nonlocal n.

>>> class A: n=n
[...]
NameError: name 'n' is not defined

>>> def f():
	class A: n=n
	
>>> f()
[...]
NameError: name 'n' is not defined

>>> def f(n):
	class A: n=n
	
>>> f(2)
[...]
NameError: name 'n' is not defined

Repeat after 'n=1' at module scope and the NameErrors disappear. It appears that you are asking that the class statement be made to act differently when nested instead of the same. This would break code that depends on the current behavior. This would need discussion on python-ideas and pydev lists.

[terryjreedy]

bpo-17853 was in the context of metaclasses. Even so, I am puzzled by the opening statement there that

from enum import Enum  # I added this as necessary
class Season(Enum):
    SPRING = Season()

"works beautifully" at top level as it indeed raises
NameError: name 'Season' is not defined

My point here is that changing the output of

n=1
def f(n=2):
    class A: n=n
    return A
print(f().n)

from 1, as it has been from the beginning (except for the print change) to 2, which I believe is implied in the request, is not the sort of syntax change we do. Hence I think this should be closed.

[ethanfurman]

Terry remarked:
---------------

I am puzzled by the opening statement there that

from enum import Enum # I added this as necessary
class Season(Enum):
SPRING = Season()

"works beautifully" at top level as it indeed raises
NameError: name 'Season' is not defined

Pay close attention to the line just before that example:

I tried having the metaclass insert an object into the custom dict
(aka namespace) returned by __prepare__; this object has the same
name as the to-be-created class.

It does not raise a NameError because the name was injected via the metaclass __prepare__ method. (Or did at that time -- I don't think that bit of cleverness made the final cut.)

As far as this enhancement request goes, I think the compatibility break is likely too large to have it in the 3.x series.

[arigo]

Terry: I meant exactly what I wrote, and not some unrelated examples:

    def f():
        n = 1
        class A: n = n

doesn't work, but the same two lines ("n = 1"; "class A: n = n") work if written at module level instead of in a function.

[ncoghlan]

With the fact that the existence of "Python without closures" predates Python 2.2, this now reads like a straight up compiler bug to me.

Compare the behaviour with no local assignment in the class body:

>>> def f():
...     n = 1
...     class C:
...         print(n)
... 
>>> f()
1

To the behaviour once the local is assigned:

>>> def f():
...     n = 1
...     class C:
...         n = n
...         print(n)
... 
>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in f
  File "<stdin>", line 4, in C
NameError: name 'n' is not defined

The issue is that the latter will still emit a LOAD_NAME/STORE_NAME pair, while the former emits LOAD_CLASSDEREF (and has emitted LOAD_DEREF for as long as I can recall hacking on the compiler).

Off the top of my head, I'm not sure our current symbol table analysis pass can actually cope with this idea though - it would require separating "just a class body local, which may or may not first be retrieved as a global or builtin" from "a class body local which is first retrieved from a closure reference".

[serhiy-storchaka]

This looks like a duplicate of bpo-9226.

[taleinat]

This does indeed seem to be a duplicate of bpo-9226.