Skip to content
This repository

Global variables undefined in interactive use of embedded ipython shell #62

Closed
ipython opened this Issue May 10, 2010 · 26 comments

12 participants

IPython: interactive computing in Python Fernando Perez Thomas Kluyver Dave Foster Dan Fletcher Tomalty JoeZ99 Andrew Badr Virendra Rajput Min RK Matthias Bussonnier
IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

Original Launchpad bug 399627: https://bugs.launchpad.net/ipython/+bug/399627
Reported by: h-fangohr (Hans Fangohr).

The error can be reproduced as follows:

  1. Start Python, and start embedded ipython session. Following ipython's manual, we do
fangohr@eta:~$ python
Python 2.4.3 (#1, Jun  8 2009, 14:09:06) 
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython.Shell import IPShellEmbed
>>> ipshell=IPShellEmbed()
>>> ipshell()

Within the just started ipython session, global variables are sometimes not visible. Two examples are:

Example 1:

In [1]: a=1

In [2]: def f(x):
   ...:     return a*x
   ...: 

In [3]: f(2)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/home/fangohr/<ipython console> 

/home/fangohr/<ipython console> in f(x)

NameError: global name 'a' is not defined

Example 2:

In [4]: b=1

In [5]: (lambda :b)()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/home/fangohr/<ipython console> 

/home/fangohr/<ipython console> in <lambda>()

NameError: global name 'b' is not defined

There is no error if "b=1; (lambda :b)()" is put in a script, and this script is executed using ipython's "run" command.

There is no error if "b=1; (lambda :b)()" is executed interactively after having started ipython.

The only way the error seems comes up is if an embedded IPython shell is started from a Python prompt AND the commands are executed interactively at the prompt.

I find the same error when trying ipython-0.9.1 (with python2.4).

The bug was reported by Olivier Klein to the nmag team (http://nmag.soton.ac.uk).

IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

[ LP comment 1 by: Fernando Perez, on 2010-04-25 23:36:38.673176+00:00 ]

OK, I can confirm the problem (even on trunk), but it's hard. I am just for now making sure this bug is confirmed so we keep tracking it, but I'm not sure how to fix it.

The issue is that in embedded shells, we try to update the global namespace to use the surrounding scope (that's the point of an embedded shell, to be able to see what's around you). But this then causes python to fail to resolve the ipython interactive namespace when nested things (like local functions) are defined. See the mainloop() method of the embedded shell for details.

I need to think a lot more about how to fix this one, any ideas very much welcome.

Fernando Perez
Owner

Over at launchpad...

chairmanK wrote 20 hours ago:

Me Too. In addition to functions, this bug also appears in generator expressions.

Thomas Kluyver
Collaborator

The equivalent component in trunk seems to be IPython.frontend.terminal.InteractiveShellEmbed. But that's broken in other ways, and evidently hasn't been tested much. Does anyone know what its future is?

Thomas Kluyver
Collaborator

Could this be the same issue as #136?

Fernando Perez
Owner

This has been now fixed:

amirbar[ipython]> python
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython import embed
>>> embed()
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: a = 1

In [2]: def f(x):
   ...:     return a*x
   ...: 

In [3]: f(3)
Out[3]: 3

In [4]: b = 1

In [5]: (lambda : b)()
Out[5]: 1

In [6]: 
Fernando Perez fperez closed this November 27, 2011
Dave Foster

Can someone explain a little more what went into the fix of this? I still encounter the issue, but only when I'm one layer removed via a method call, and only if I run from a script, not interactively.

python 2.7.2 on OSX 10.6.8, ipython 0.11 and 0.12 both exhibit similar behavior (using 0.12 for this comment)

This is a problem with our project which features (prominently) an embedded IPython shell.

testembed.py

from IPython import embed

def hi():
    embed()

if __name__ == '__main__':
    #embed()
    hi()

Run this on the command line with python testembed.py and see this session:

Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import time

In [2]: def tim():
   ...:     print time.time()
   ...:     

In [3]: tim()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
[snip] in <module>()
----> 1 tim()

[snip] in tim()
      1 def tim():
----> 2     print time.time()
      3 

NameError: global name 'time' is not defined

In [4]: 

However, comment out the call to hi() and replace it with the embed() call:

Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import time

In [2]: def tim():
    print time.time()
   ...:     

In [3]: tim()
1328639444.29

In [4]: 

After poking around, I think it has to do with the stack_depth parameter used here and this block: https://github.com/ipython/ipython/blob/master/IPython/frontend/terminal/embed.py#L185

Thoughts?

Thomas Kluyver
Collaborator

This is somewhat complicated, but I believe that's a restriction in Python itself.

In the failing case you show, time is not actually put into the global namespace: because you called embed inside the hi function, new variables you create interactively are local to that function. Ideally, tim() should work as a closure, closing over the reference to the time module. However, closures only appear to work when the containing function is compiled in one go. As far as I can tell, there is no way to define a closure dynamically. This simple example fails:

def outer():
    import time
    exec("def inner(): return time.time()")
    return inner

outer()()

This is probably because nested scopes were added to Python relatively late (they were a future import in 2.1, and always on in 2.2).

Dave Foster

Ok, I think I understand. Looks like we can't really do anything about this then, other than dumping what I would write interactively to a file and then reading that file back in. Probably too complex for what we want to be able to do interactively.

Thanks, btw.

Dave Foster

Sorry to keep harping on here, but it just makes interactive sessions feel very clunky:

Regular python:

>>> d={'one':1, 'two':2}
>>> getkeys=lambda: d.keys()
>>> getkeys()
['two', 'one']

Regular IPython:

In [1]: d={'one':1, 'two':2}

In [2]: getkeys=lambda: d.keys()

In [3]: getkeys()
Out[3]: ['two', 'one']

Embedded IPython:

>>> from IPython import embed
>>> embed()
Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: d={'one':1, 'two':2}

In [2]: getkeys=lambda: d.keys()

In [3]: getkeys()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <module>()
----> 1 getkeys()

/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <lambda>()
----> 1 getkeys=lambda: d.keys()

NameError: global name 'd' is not defined

I guess there isn't much that can be done, but I don't get why we can create a dynamic closure in regular python/ipython but not the embedded version.

Thomas Kluyver
Collaborator

By the oddities of Python scoping, in standard Python/IPython, we're not actually creating a closure. d is a global variable, and the rules for accessing those work differently from closures. Each function keeps a reference to the global namespace where it was defined (getkeys.func_globals), so it can access any variables defined there.

In contrast, when you make a closure, Python attaches references to each variable over which it has closed, as determined by the compiler - but that only works when the inner and outer functions are compiled at the same time. It looks like this:

In [8]: def outer():
    a = 1
    def inner():
        return a
    return inner
   ...: 

In [9]: f = outer()

In [10]: f
Out[10]: <function __main__.inner>

In [11]: f.func_closure
Out[11]: (<cell at 0x9f5e344: int object at 0x9a830b0>,)

This is possibly done to save memory - if the closure carried a reference to the local scope where it was defined, none of the variables from that scope could be freed while the closure was live.

liokm commented April 24, 2012

I'm using Python 2.7.2 and IPython 0.12.1 on OSX 10.7.3 and having this issue still. When I run ./manage.py shell of Django which invokes IPython.embed(), the issue occurs. However, manually invoking the embed() in Python shell or from simple script file, there is no problem.

Thomas Kluyver
Collaborator

There's not a lot we can directly do about that, but we should probably encourage third parties away from embed for non-trivial uses.

liokm commented April 24, 2012

@takluyver Do you mean that better to use ipython directly in this case?

Thomas Kluyver
Collaborator

It's possible for Django to start IPython in a way that won't cause this problem, but that's not the way we currently make easy. The problem occurs when IPython starts with separate local and global namespaces. There's no reason that Django requires separate local and global namespaces, but that's what calling embed() inside a function implies.

For reference, here's the code in Django:
https://code.djangoproject.com/browser/django/trunk/django/core/management/commands/shell.py

liokm commented April 24, 2012

@takluyver That makes sense, thank! I'm opening a Django ticket for this.

Fernando Perez
Owner

@takluyver, now that we've merged embed_kernel so all the main pieces are in place, do you want to tackle a bit a cleanup of this to make slightly more fine-tuned uses easier?

Thomas Kluyver
Collaborator
Dan

I am still having trouble with this issue. I've tried narrowing down exactly what's causing it, and the best I can determine is that it only happens on Ubuntu 12.04. I've tried all of the latest release versions on other servers, and it works fine. But any time I define a function in ipython or use %cpaste to paste in a function from another file, the inside of that function has no access to the global scope. It makes it basically impossible to do anything useful in terms of writing functions on the fly.

Fletcher Tomalty

I am still having this problem with IPython 0.13 when it is called from other tools (e.g. Django's debugsqlshell command). It's very frustrating that something as basic as defining a function is totally broken.

Thomas Kluyver
Collaborator
JoeZ99

Not only ubuntu. debian wheezy also shows that.

Andrew Badr

FYI, the Django ticket @liokm created above is https://code.djangoproject.com/ticket/18204, which now points to https://code.djangoproject.com/ticket/17078, which looks to have been fixed in trunk. It should land in 1.5.

Virendra Rajput

I m having the same issue on Ubuntu with Ipython 0.13.2
screenshot from 2013-08-07 18 13 33

Min RK
Owner

@bkvirendra that is fixed in django 1.6

Virendra Rajput

But 1.6 is not even stable yet!

Matthias Bussonnier
Collaborator

But 1.6 is not even stable yet!

Software are not always stable, and between releases there might still be bugs. But there is nothing that should be fixed in IPython. Even if we do something here, the fix would not be in IPython stable before it is released either...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.