IPythonShellEmbed fails to recognize local variables #136

Open
prsteel opened this Issue Jul 19, 2010 · 14 comments

Projects

None yet

10 participants

@prsteel
prsteel commented Jul 19, 2010

Issue

Embedded IPython shells can lose track of local variables.

Test Case

Minimal test case:

class Foo(object):
    """ Container-like object """
    def __setattr__(self, obj, val):
        self.__dict__[obj] = val

    def __getattr__(self, obj, val):
        return self.__dict__[obj]

f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
    f.values[x] = x

def bar(foo):
    import IPython
    IPython.Shell.IPShellEmbed()()
    return sum(foo.values[x] for x in foo.indices)

print bar(f)

To see the error, first run the code in Python (or IPython) and exit from the spawned shell; the final print statement correctly displays '15'. Run the code again, but this time type sum(foo.values[x] for x in foo.indices) in the spawned shell, and we receive the error

" NameError: global name 'foo' is not defined".

@tomspur
Contributor
tomspur commented Aug 1, 2010

Hmm, where did you define 'foo'?

If you just run call foo outside of the function bar, foo should not exist (and obvously it doesn't).

When you instead run "sum(f.values[x] for x in f.indices)" you get 15 again...

@prsteel
prsteel commented Aug 3, 2010

Quite correct. However, I'm refering to the use of foo inside the spawned IPython shell, which is in turn spawned inside the function definition where foo is a local variable.

@takluyver
Member

Is this likely to be the same issue as #62?

@fperez
Member
fperez commented Nov 28, 2011

No, @takluyver: this is a separate issue and indeed a real bug in our embedding code. I was hoping your recent work with namespaces would have fixed it, but it didn't. For reference, here's the example code to run with the current embedding api:

class Foo(object):
    """ Container-like object """
    def __setattr__(self, obj, val):
        self.__dict__[obj] = val

    def __getattr__(self, obj, val):
        return self.__dict__[obj]

f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
    f.values[x] = x

def bar(foo):
    import IPython
    IPython.embed()
    return sum(foo.values[x] for x in foo.indices)

print bar(f)

Then, in the spawned, embedded IPython, this fails:

In [1]: sum(foo.values[x] for x in foo.indices)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/home/fperez/tmp/junk/ipython/foo.py in <module>()
----> 1 sum(foo.values[x] for x in foo.indices)

/home/fperez/tmp/junk/ipython/foo.py in <genexpr>((x,))
----> 1 sum(foo.values[x] for x in foo.indices)

NameError: global name 'foo' is not defined

And it doesn't work even if we pass to the embed call user_ns=locals() explicitly, but in that case we get in addition a crash on exit:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2702, in atexit_operations
    self.reset(new_session=False)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1100, in reset
    self.displayhook.flush()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/displayhook.py", line 319, in flush
    self.shell.user_ns['_oh'].clear()
KeyError: '_oh'

It looks like our embedding machinery is in pretty bad shape...

@takluyver takluyver was assigned Nov 28, 2011
@takluyver
Member

Assigned this to myself to look at.

@fperez
Member
fperez commented Nov 29, 2011

Excellent, thanks.

@takluyver
Member

Unfortunately, I think this is a limitation of Python itself. It appears code compiled dynamically cannot define a closure, which is essentially what we're doing here with a generator expression. Here is a minimal test case:

def f():
   x = 1
   exec "def g(): print x\ng()"

f()

Which gives:

Traceback (most recent call last):
  File "scopetest.py", line 5, in <module>
    f()
  File "scopetest.py", line 3, in f
    exec "def g(): print x\ng()"
  File "<string>", line 2, in <module>
  File "<string>", line 1, in g
NameError: global name 'x' is not defined

Note that you can still see local variables in IPython - in the example given, plain print foo works. But you can't close a new scope over them.

@takluyver
Member

I think it may be possible to make this work using collections.ChainMap from Python 3.3 so that IPython sees both local and global variables where it is embedded as globals. However, from the lack of noise about this over the last two years, I don't think it is high priority, so I'm retagging accordingly, and hopefully getting to this some time after 1.0.

@clebio
clebio commented Nov 13, 2014

Is it acceptable to up-vote, 👍 ? This affects me as well. I can add my use-case if requested.

@minrk
Member
minrk commented Nov 13, 2014

I would be 100% okay if the fix for this only worked on Python 3.

@amelio-dx

I am also having the same problem under Python 3. Thanks for re-openning.

@amelio-vazquez-reina

Same problem here 👍 both in Python 2 and 3.

@minrk minrk removed the prio-medium label Jan 14, 2015
@Erotemic

I have this issue in both Python 2 and 3. It affects me on a daily basis.

@nikitakit
nikitakit commented Jun 27, 2016 edited

I looked into this issue a bit, and it's definitely fixable (though maintaining fixes for both Python 2 and 3 may be messy).

The ChainMap solution would be easiest to include into IPython proper. However, there's a slight catch that eval/exec require globals to be a dict. Creating a class MyChainMap(ChainMap, dict): pass can work around this.

I also wrote a Python 3.5+ fix based on a different strategy of simulating closure cells and forcing the python compiler to emit the correct bytecode to work with them. The relevant file is here, part of my xdbg demo. It works by replacing get_ipython().run_ast_nodes.

As far as I can tell, the two approaches differ only in their handling of closures. When xdbg is embedded at a scope that has closed over some variables, it can correctly access those variables by reference and mutate them. Additionally, if any functions are created in the interactive interpreter, they will close over any local variables they need while allowing the rest of the local scope to be garbage collected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment