How to preserve user variables from import clashing?

[klonuo]

Yesterday I opened issue in MPL project: pylab overwrites user variable(s) which I wanted to open first here, but then thought that perhaps MPL is more appropriate.

Issue is closed, and from the comments it may be reasonable. I don't know if this is highly subjective, but the issue I had seemed as a bad design to me - as mentioned, I had f hdf5 file object that was connected to couple of numpy arrays which just all disappeared after I entered %pylab mode, as file handle was reassigned.
Suggested approaches as entering %pylab before doing else, or don't name your variable as f or any other name present in %pylab space, aren't good enough, obviously.

In [1]: %pylab

In [2]: [x for x in globals().keys() if len(x) < 2]
Out[2]: ['f', 'x', '_', 'e']

There are no other lower case, one letter variables that can be overwritten except f and e in such scenario. And also maybe some other potential handy named variables that user may assign and are exposed in %pylab mode.

Does IPython have instrument to avoid similar clashing with variable names in current user namespace?

[jenshnielsen]

Another option is to configure pylab to not import anything into the global namespace
using: c.InteractiveShellApp.pylab_import_all = False and others.

What pylab imports does is essentially:

import numpy as np
import matplotlib.pyplot as plt
from pylab import * 

It is the last line that gives the problem and can be removed by the configure variable.
I don't think there is any way in python to prevent a from x import * to overwrite variables
with the same name already defined and most likely it is not a good idea either since it would
give inconsistent behaviour.

If you just want an eventloop integration you can also use the %gui magic instead of %pylab
but this will not set matplotlib into interactive mode so you need to do that with matplotlib.interactive(True)
if you want to use mpl interactively and I don't think it works with the inline backend.

pylab_import_all can also be configured from inside IPython using the %config magic
%config InteractiveShellApp.pylab_import_all = False

[bfroehle]

Thanks @jenshnielsen for the nice answer. There isn't anything else we can do here.

[bfroehle]

I guess one thing we could do is extend the %pylab syntax to add a --no-import-all flag, or similar. Would that suffice?

[klonuo]

@jenshnielsen, @bfroehle, thanks for your suggestions but I think that would defeat %pylab purpose

Checking can be done, IMHO.

Pylab magic imports matplotlib and numpy, so we have their namespaces.
pylab.py matplotlib module does couple of imports from above namespaces, so IPython pylab magic can check list of user variables against list of variables/functions that would be exposed when importing pylab, in no time. And then refuse to import if there is a match and inform user.
Only question is if you guys find that worth.

[bfroehle]

The relevant bit of code is at https://github.com/ipython/ipython/blob/master/IPython/core/pylabtools.py#L238

I take it you would advocate for something like:

        s = ("from matplotlib.pylab import *\n"
             "from numpy import *\n")
        pylab_ns = {}
        exec s in pylab_ns

        for key in user_ns:
            pylab_ns.pop(key, None)
        user_ns.update(pylab_ns)

Personally, I think variable the clobbering is to be expected.

[klonuo]

Not exactly like that.
I would suggest taking care of all import * from pylab.py.
Snippet like this:

import matplotlib.pyplot as plt
import numpy as np

[x for x in dir(plt) + dir(np) + dir(np.fft) + dir(np.random) + dir(np.linalg) if x.islower() and not x.startswith('_')]

that is executed before importing pylab anyway, will give us a list of names that could potentially crash user variables. Then just do the matching before importing pylab

[jenshnielsen]

There are multiple purposes of %pylab. To me the most important is that it enables a gui eventloop for interactive plotting and sets matplotlib in interactive mode. The imports only save a few lines of code.

To me the from pylab import * is only there to make an easier transition from languages that do not use
name spaces such as Matlab.

I don't think filtering on import is a good idea.
What if you have a long interactive session and forget that you have a defined a now useless variable in the beginning
then you would end up in the opposite situation i.e. why is f not numpy.f ?

But adding a warning to pylab is not a bad idea, something like "warning pylab imports has overwritten the local variable f with the function f". It could also be a good idea if %pylab would print what imports is does like isympy
does to make it more obvious what goes on.

[takluyver]

I'd agree with the others in this thread - code that magically detects what variables you already have and modifies its own behaviour sounds like a recipe for surprise and confusion. I agree that it would be good for %pylab to show what it has imported, and adding a --no-import-all flag isn't a bad idea.

[klonuo]

I left a space that this may be too subjective, but I'll reply as I believe it's right way to do it

I'd agree with the others in this thread - code that magically detects what variables you already have and modifies its own behaviour sounds like a recipe for surprise and confusion.

Suggested behavior is about special pylab construction, it's not about arbitrary known workflows. Enhanced interactive environment as IPython is, I believe should take care about this special case as IPython is one great example to support the pylab idea.
Price is cheap - just do list comparison and inform user. Result is sensitive - I would like to see IPython inform me that there is name clash and it wont overwrite my variables. Then perhaps provide additional flag to do pylab import regardless user variable namespace --force-import instead suggested vice versa flag --no-import-all

I agree that it would be good for %pylab to show what it has imported

I can't make the meaning of this

[takluyver]

No, I still think, even for %pylab, we shouldn't make it any more magical than it already is. "Special cases are not special enough".

By 'show what it has imported', I mean that the message it displays should say something like:

Welcome to Pylab. This has done the following imports:
import numpy as np
import matplotlib.pyplot as plt
from pylab import *

[klonuo]

But pylab IS special case.
Python zen is great but truth (interpretation) is in the eye of beholder.

[takluyver]

I agree that it's a special case, I just don't think it's special enough to
need that kind of magic.

On 7 December 2012 12:11, klonuo notifications@github.com wrote:

But pylab IS special case.
Python zen is great but truth (interpretation) is in the eye of beholder.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2664#issuecomment-11127791.

[Carreau]

I agree that having %pylab not import things because they are already in the namespace is the wrong idea.
People already don't know pylab is importing * and often filled duplicate false bug report for un expected behavior.
This also have the huge disadvantage of not beeing backward compatible.

This will break notebooks that might require (even implicitely or because of a temporary variable) on the fact that %pylab does overwrite variable.

That it warns of overwrite in user_ns why not.

[bfroehle]

As @Carreau described, I've certainly used the clobbering purposefully a few times. For example, if I accidentally define a variable named plot and want to get it back to plt.plot I'd just run %pylab.

To reiterate a few previous ideas:

• The %pylab docs should mention its import * characteristics.
• I'm in favor of adding --import-all and --no-import-all flags.
• I'm neutral about warning users that variables have been clobbered... maybe a good idea?

Quick question:

Would there be any side effects to switching

exec pylab_import_string in user_ns

to

pylab_ns = {}
exec pylab_import_string in pylab_ns
user_ns.update(pylab_ns)

This would allow us to build a list of clobbered variables as

clobbered = set(pylab_ns).intersection(user_ns)

Note that this would get very annoying if one ran %pylab twice and it printed the list of all symbols which it had imported...