An inconsistency with nested scopes

[nickjacobson]

BPO	991196
Nosy	@arigo, @birkenfeld, @josiahcarlson, @mdickinson, @devdanzin
Files	python1.py: Nested scope of y fails, and it shouldn't.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-03-31.15:56:24.677>
created_at = <Date 2004-07-14.21:30:23.000>
labels = ['interpreter-core', 'type-bug']
title = 'An inconsistency with nested scopes'
updated_at = <Date 2010-05-26.11:40:09.116>
user = 'https://bugs.python.org/nickjacobson'

bugs.python.org fields:

activity = <Date 2010-05-26.11:40:09.116>
actor = 'mark.dickinson'
assignee = 'jhylton'
closed = True
closed_date = <Date 2009-03-31.15:56:24.677>
closer = 'jhylton'
components = ['Interpreter Core']
creation = <Date 2004-07-14.21:30:23.000>
creator = 'nickjacobson'
dependencies = []
files = ['1342']
hgrepos = []
issue_num = 991196
keywords = []
message_count = 8.0
messages = ['21611', '21612', '21613', '21614', '82038', '84813', '106513', '106523']
nosy_count = 8.0
nosy_names = ['jhylton', 'arigo', 'georg.brandl', 'josiahcarlson', 'mark.dickinson', 'nickjacobson', 'ajaksu2', 'hawkett']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue991196'
versions = ['Python 2.6']

[nickjacobson]

There's an inconsistency with nested scopes.

From the Python Ref. Manual:

"If [a local variable] definition occurs in a function block,
the scope extends to any blocks contained within the
defining one..."

i.e. So as long as code is not on the module level,
scopes are extended. Therefore this works:

def main():
	y = 3
	def execfunc():
		print y
	execfunc()

if __name__ == '__main__':
	main()

In addition, if code IS on the module level, its variables
go in globals(). So this works too:

y = 3
def execfunc():
	print y
execfunc()

However, (here's the inconsistency) the following code
fails, saying that y is undefined:

def main():
	s = \
"""
y = 3
def execfunc():
	print y
execfunc()
"""
	d = {}
	e = {}
	exec s in d, e

if __name__ == '__main__':
	main()

In this case, the code in s is treated like it's on the
module level, and the nested scope treatment of y
doesn't occur. BUT, unlike normal code on the module
level, y doesn't go in globals(). I think globals() is
locked or something?

Conclusion:

The latter piece of code should work; that is, y should
be nested and therefore recognized by execfunc().

[josiahcarlson]

Logged In: YES
user_id=341410

>>> def f():
...     y = 5
...     print 'y' in globals(), 'y' in locals()
...     def i():
...         print 'y' in globals(), 'y' in locals()
...     i()
...
>>> f()
False True
False False
>>>
>>> def g():
...     gl = {};lo={}
...     exec '''y = 5
... print 'y' in globals(), 'y' in locals()
... def i():
...     print 'y' in globals(), 'y' in locals()
... i()
... ''' in gl, lo
...
>>> g()
False True
False False

That looks constant...but what if we print out 'y'?

>>> def s():
...     y = 5
...     print 'y' in globals(), 'y' in locals(), y
...     def i():
...         print 'y' in globals(), 'y' in locals(), y
...     i()
...
>>> s()
False True 5
False True 5
>>>
>>> def t():
...     gl = {};lo = {}
...     exec '''y = 5
... print 'y' in globals(), 'y' in locals(), y
... def i():
...     print 'y' in globals(), 'y' in locals(), y
... i()
... ''' in gl, lo
...
>>> t()
False True 5
False False
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 3, in t
  File "<string>", line 5, in ?
  File "<string>", line 4, in i
NameError: global name 'y' is not defined

Now why did 'y' stick itself into the locals() of i() in
s()? Is this another bug?

What if we remove the namespaces gl and lo?

>>> def u():
...     exec '''y = 5
... print 'y' in globals(), 'y' in locals(), y
... def i():
...     print 'y' in globals(), 'y' in locals(), y
... i()
... '''
...
>>> u()
False True 5
False False
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in u
  File "<string>", line 5, in ?
  File "<string>", line 4, in i
NameError: global name 'y' is not defined

Nope, still dies. It does seem to be a bug in exec. I'm
still curious why 'y' was placed into i()'s local namespace
in s().

[arigo]

Logged In: YES
user_id=4771

The behavior you get can be explained quite easy, but it seems nevertheless inconsistent with the documentation: in my opinion it is a serious bug.

The reason the "exec"ed code doesn't work like the same code put at the global module level is that code that runs at the module level always runs with the same dictionary for its globals and locals, whereas in your example you use two different dictionaries.

Assignments always go to the locals; that's why 'y' goes into the dictionary 'e'. Now a function can only see its own locals and the surrounding globals; that's why execfunc() misses the value of 'y'. This is the old way Python worked. In recent versions, a special trick was added so that functions defined inside another function find variable bindings from the enclosing function. I think you found a case where this trick fails to apply.

[birkenfeld]

Logged In: YES
user_id=849994

Closed bpo-1167300 as a duplicate.

[devdanzin]

Confirmed in trunk,

[jhylton]

This code behaves as intended. The module-level execution environment
is different than other environments. The global scope and local scope
are the same dictionary. Assignments at the top-level become globals
because of this behavior of the execution environment. If you want exec
to mimic the top-level environment, you need to pass it a single dictionary.

[hawkett]

bpo-8819 was closed as duplicate. That issue linked a description of the problem on stack overflow http://stackoverflow.com/questions/2904274/globals-and-locals-in-python-exec. I would like to argue that this is a bug, and should be fixed in 2.6+. The definition of bug here is that python does not behave as documented - that variable name resolution does *not* check the locals() of the enclosing scope. The fact that the code mistakenly assumes locals and globals would be the same thing in this situation does not mean it is not a bug.

The statement in the previous comment - 'if you want exec to mimc the top level environment, you need to pass it a single dictionary' is true, but it hides that fact that this is the *only* thing you can do - if you *don't* want exec to mimic the top level environment, what's the approach? Doing anything else just creates a unique, undocumented, oddly behaving scope that doesn't apply closures correctly.

What are the use cases for passing in locals? Doing so means your code behaves abnormally, unless you think carefully about how you write it, and that's not good - 'Write python code like this, except for this situation where it doesn't work, and you have to write your python like this, avoiding certain closure scenarios that would otherwise work.' What's the exec() API with locals for?

If you don't pass in locals, or make globals and locals the same dictionary, then its an absolute pain to retrieve the definitions created in the exec'd code - they're mixed in with all the globals python adds, and if you don't know in advance what is being defined in the code block, it's close to impossible. To me, this is the primary use case for locals being passed in, and was surely the intention when the API was constructed. This bug prevents this use case.

I'm guessing that somewhere in the python source there is some code that goes (pseudo)

if scope == module: check_globals()
else:
  check_locals()
  check_globals()

and that this is done for performance reasons, but surely the check could be different without giving up much, and fix the problem?

if locals() is globals(): check_globals()
else:
  check_locals()
  check_globals()

[mdickinson]

I'm guessing that somewhere in the python source there is some code that goes [...]

Unfortunately it's not nearly that simple. As I mentioned in my message on python-dev, the problem is that 'y' gets bound with a 'STORE_NAME' opcode, which puts 'y' into the locals dict, and then retrieved from within the function with a 'LOAD_GLOBAL' opcode, which looks in the globals dict; hence the NameError.

So should the compiler be generating a 'LOAD_NAME' instead of a 'LOAD_GLOBAL' for this code? I'm not really familiar with the compilation process, so I've no idea whether this makes sense, or what impact it might have on existing code.