UnboundLocalError when trying to raise exceptions inside execfile

[jerryseutter]

BPO	2378
Nosy	@amauryfa
Files	core_wierd_bug_minimal.zip: Files listed in comments above, in a zipfile. "figleaf" and "test_broken.py" test_broken3.py debughelper.tgz localstofast.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/amauryfa'
closed_at = <Date 2008-07-21.22:01:19.467>
created_at = <Date 2008-03-18.02:55:44.479>
labels = ['interpreter-core', 'type-bug']
title = 'UnboundLocalError when trying to raise exceptions inside execfile'
updated_at = <Date 2008-07-21.22:01:19.465>
user = 'https://bugs.python.org/jerryseutter'

bugs.python.org fields:

activity = <Date 2008-07-21.22:01:19.465>
actor = 'amaury.forgeotdarc'
assignee = 'amaury.forgeotdarc'
closed = True
closed_date = <Date 2008-07-21.22:01:19.467>
closer = 'amaury.forgeotdarc'
components = ['Interpreter Core']
creation = <Date 2008-03-18.02:55:44.479>
creator = 'jerry.seutter'
dependencies = []
files = ['9716', '10698', '10699', '10953']
hgrepos = []
issue_num = 2378
keywords = ['patch']
message_count = 10.0
messages = ['63856', '66575', '66662', '66665', '66674', '68554', '68555', '68613', '68617', '70122']
nosy_count = 6.0
nosy_names = ['jhylton', 'amaury.forgeotdarc', 'jerry.seutter', 'werneck', 'nikolasco', 'tenuki']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2378'
versions = ['Python 2.6']

[jerryseutter]

Found a bug when trying to integrate figleaf coverage into trunk. I
have ripped the code down to the smallest subset that still causes the
behaviour. The code works on the latest release of Python 2.5 but is
broken on trunk. It comes in two files. The first is the caller (figleaf):

import os
import sys

def foo(f, e, o):
    pass
    
sys.settrace(foo)

import __main__
execfile('test_broken.py', __main__.__dict__)

The second file is the test (test_broken.py):

# This code breaks on trunk

def test_foo():
    class CustomException(Exception):
        pass

    class SomeClass:
        def foo(self):
            raise CustomException

    # The error only appears with enough nested blocks.
    if (True == True):
        try:
            raise IOError
        except CustomException:
            pass

It should raise IOError. When run, it gives the following output:

jerry-seutters-computer:~/code/python/core_wierd_bug_minimal jseutter$
../core/python.exe figleaf                
Traceback (most recent call last):
  File "figleaf", line 10, in <module>
    execfile('test_broken.py', __main__.__dict__)
  File "test_broken.py", line 18, in <module>
    test_foo()
  File "test_broken.py", line 15, in test_foo
    except CustomException:
UnboundLocalError: local variable 'CustomException' referenced before
assignment
[10019 refs]

[werneck]

Just note the error happens even without the try/except block inside the
'if' statement.

[nikolasco]

I can't reproduce this with r63075...

[werneck]

I get it with r63075, r63085, on Linux.

[nikolasco]

Apologies, I didn't run the test case correctly; I do get the error as
reported

[tenuki]

Shorter trigger code..

[tenuki]

Some debugging helper code and my conclutions of one work day:
debughelper.tgZ:
-test_broken1/2.py
one does triggers the bug, the other doesn't)
-rtest.sh
executes boths and compares its outputs
-frameobject.c.diff
applied to Objects/frameobject.c, adds some debug info.

What I found:
1. The CustomException is disappearing from locals()
2. PyFrame_FastToLocals() (from that .c file) is updating the locals, 
and removing that exception from there.
3. In the failing case this code:
                if (deref) {
                        assert(PyCell_Check(value));
                        value = PyCell_GET(value);
                }
 is returning value==NULL.

Don't know why that happens.
But you could inspect out1.txt/out2.txt made with rtest.sh, and could
discover something..

[amauryfa]

The problem seems to have been introduced by r53954.

[amauryfa]

The problem is in PyFrame_LocalsToFast.

(As I understand it: "Locals" refers to the locals() dictionary of the
frame; "Fast" refers to an optimization where local variables are stored
in an array.
Each call to locals() or the trace function normalizes data by copying
the Fast array into the Locals dictionary; with sys.settrace, the user
may modify the locals, so the inverse transformation is done after each
call to the trace function)

When defining a class, PyFrame_FastToLocals does not copy the free
variables from enclosing scopes. This is the goal of r53954 "Do not copy
free variables to locals in class namespaces".
When executing code in the class body, the sys.settrace function is
called with this reduced set of locals().
The problem is that PyFrame_LocalsToFastdoes does not have the same
test, and uses the locals() to update (and clear) the free variables as
well.

I attach a patch that makes PyFrame_LocalsToFast symmetric to
PyFrame_FastToLocals: in the case of a class statement, do not update
the free variables.

[amauryfa]

Committed as r65177.