erroneous behavior when creating classes inside a closure

[monsanto]

BPO	9226
Nosy	@gvanrossum, @terryjreedy, @mdickinson, @ncoghlan, @taleinat, @ericvsmith, @benjaminp, @merwok, @bitdancer, @iritkatriel
Files	test.py: Test case obscure_corner_cases.patch obscure_corner_cases2.patch test3.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-07-11.19:56:34.789>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11']
title = 'erroneous behavior when creating classes inside a closure'
updated_at = <Date 2021-12-01.16:29:28.072>
user = 'https://bugs.python.org/monsanto'

bugs.python.org fields:

activity = <Date 2021-12-01.16:29:28.072>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2010-07-11.19:56:34.789>
creator = 'monsanto'
dependencies = []
files = ['17950', '17953', '17954', '18164']
hgrepos = []
issue_num = 9226
keywords = ['patch']
message_count = 15.0
messages = ['110037', '110038', '110039', '110040', '110041', '110045', '110046', '110048', '111386', '111488', '111489', '111509', '111512', '322383', '407468']
nosy_count = 11.0
nosy_names = ['gvanrossum', 'terry.reedy', 'mark.dickinson', 'ncoghlan', 'taleinat', 'eric.smith', 'benjamin.peterson', 'eric.araujo', 'r.david.murray', 'monsanto', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9226'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[monsanto]

I have a function whose closure contains a local variable that shadows a global variable (lets call it x). If I create a class as follows:

class Test(object): x = x

Test.x will contain the value of the global x, not the local x. This ONLY happens when the names are the same, and it only happens in the class body; i.e., "class Test(object): y = x" and class "Test(object): pass; Test.x = x" work fine.

However, if there is an assignment x = x AND you make other assignments, such as y = x, in the body, the other variables will have the wrong value too.

Test case attached. Problem noticed on Python 2.6.2 on Windows and 2.6.5 on Linux.

[monsanto]

A friend confirmed that this was the case on 3.1.2 as well.

[benjaminp]

I'm not sure what I correct behavior is in this case. Consider the function equivalent:

x = 3
def f(x):
    def m():
        x = x
        print x
    m()
f(4)

which gives:

Traceback (most recent call last):
  File "x.py", line 7, in <module>
    f(4)
  File "x.py", line 6, in f
    m()
  File "x.py", line 4, in m
    x = x
UnboundLocalError: local variable 'x' referenced before assignment

The class example works because name namespaces are unoptimized, so failing to find a binding in the local (class) namepsace, Python looks at the globals and finds the global definition.

[mdickinson]

I don't see anything in

http://docs.python.org/reference/executionmodel.html#naming-and-binding

to suggest that the class should behave differently from a nested function here; that is, I'd expect UnboundLocalError.

[mdickinson]

Jython 2.5.1 gives the same results as Python:

newton:~ dickinsm$ cat test.py
x = "error"

def test(x):
    class Test(object):
        x = x
    print("x: ", x)
    print("Test.x: ", Test.x)

test("success")
newton:~ dickinsm$ jython2.5.1/jython test.py
('x: ', 'success')
('Test.x: ', 'error')

[bitdancer]

I agree with Mark, I'd expect an UnboundLocalError. I remembered this thread on Python-dev that may or may not be relevant, but is certainly analogous:

http://www.mail-archive.com/python-dev@python.org/msg37576.html

[benjaminp]

Here's a patch. It raises a NameError in that case.

[mdickinson]

I think it would be worth bringing this up on python-dev, especially since it affects alternative Python implementations.

It would also be good to have a documentation fix to the reference manual that clearly explains whatever behaviour is decided on; it's not at all clear (to me, anyway), how to extract this information from the docs.

[terryjreedy]

Chris, when posting something like this, *please* include the output. I had to insert ()s to run this with 3.1. I will upload the py3 version as test3.py. Is your output the same as mine?

x: success
Test.x: error
Test2.y: success
Test3.x: error
Test3.y: error
Test4.x: success

There is an obvious inconsistency between Test2 and Test/Test3. This shows up also in the dis.dis(test) output. So there is definitely a bug.

To me, the Test2 result is the error. I base this on 7.7 Class Definitions: "The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace." I interpret this to mean that intermediate namespaces are not used (as was the case before 2.2). Indeed, this sentence is unchanged from the 2.1 doc (and before).
http://docs.python.org/release/2.1/ref/class.html

Of course, the intent could have changed without changing the wording, by reference to the Naming and Binding section, but then this sentence really should be changed too.

The current Naming and Binding section includes:

"A scope defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods."

So class blocks are an exception in propagating down, and I thought they were also an exception for propagating into, for the reason stated above.

It is possible that this is an undefined corner of the language. Certainly, the compiler is confused as it treats one nested class (Test2) as a closure and the other three nested classes as not.

Since the name and binding design is Guido's and central to Python's operation, I personally would not touch it without his input. Hence I have added him as nosy and second the idea of pydev discussion.

[gvanrossum]

Hm. This seems an old bug, probably introduced when closures where first introduced (2.1 ISTR, by Jeremy Hylton).

Class scopes *do* behave differently from function scopes; outside a nested function, this should work:

x = 1
class C(object):
  x = x
assert X.x == 1

And I think it should work that way inside a function too.

So IMO the bug is that in classes Test and Test3, the x defined in the function scope is not used. Test2 shows that normally, the x defined in the inner scope is accessed.

So, while for *function scopes* the rules are "if it is assigned anywhere in the function, every reference to it references the local version", for *class scopes* (outsided methods) the lookup rules are meant to be dynamic, meaning "if it isn't defined locally yet at the point of reference, use the next outer definition".

I haven't reviewed the patches.

[gvanrossum]

I meant, of course,

assert C.x == 1

[terryjreedy]

Guido clarified:

Class scopes *do* behave differently from function scopes;
outside a nested function, this should work:
x = 1
class C(object):
x = x
assert C.x == 1
And I think it should work that way inside a function too.

I take that to mean that

x = 0
def f()
  x = 1
  class C(object):
    x = x
  assert C.x == 1
f()

should work, meaning that C.x==0 and UnboundLocalError are both wrong.

That would mean to me that in "The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace." the phrase "the original global namespace" should be changed to "the surrounding namespaces".

I also think this from Guido

"So, while for *function scopes* the rules are "if it is assigned anywhere in the function, every reference to it references the local version", for *class scopes* (outsided methods) the lookup rules are meant to be dynamic, meaning "if it isn't defined locally yet at the point of reference, use the next outer definition"."

should somehow also be clearer, probably also in the class page, so that people will neither expect an UnboundLocalError.

[gvanrossum]

On Sat, Jul 24, 2010 at 3:21 PM, Terry J. Reedy <report@bugs.python.org> wrote:

Terry J. Reedy <tjreedy@udel.edu> added the comment:

Guido clarified:
> Class scopes *do* behave differently from function scopes;
> outside a nested function, this should work:
> x = 1
> class C(object):
>   x = x
> assert C.x == 1
> And I think it should work that way inside a function too.

I take that to mean that

x = 0
def f()
 x = 1
 class C(object):
   x = x
 assert C.x == 1
f()

should work, meaning that C.x==0 and UnboundLocalError are both wrong.

Indeed.

That would mean to me that in "The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace." the phrase "the original global namespace" should be changed to "the surrounding namespaces".

Those words sound like they were never revised since I wrote them for
Python 0.9.8 or so...

I also think this from Guido

"So, while for *function scopes* the rules are "if it is assigned anywhere in the function, every reference to it references the local version", for *class scopes* (outsided methods) the lookup rules are meant to be dynamic, meaning "if it isn't defined locally yet at the point of reference, use the next outer definition"."

should somehow also be clearer, probably also in the class page, so that people will neither expect an UnboundLocalError.

FWIW, unless something drastically changed recently, the language
reference is likely out of date in many areas. I would love it if a
team of anal retentive freaks started going through it with a fine
comb so as to make it describe the state of the implementation(s) more
completely.