ipython dev retains directory information after directory change

[jniedziela]

I started ipython in one working directory (A), and then changed to a different directory (B), each directory had the same filenames. The ipython shell maintained the use of the files from directory A even after switching to directory B in the ipython shell. Quitting ipython, changing in the terminal to directory B, then restarting ipython in directory B, results in the correct files being imported. Is this a bug?

[minrk]

what do you mean 'maintained the use of files'? Please provide specific code that produces the behavior you are concerned about. I expect this is simply the behavior of Python imports / path (when in doubt about this sort of thing, always try to compare IPython's behavior with Python's - if they do not differ, it is almost certainly not a bug.

[ahmadia]

@minrk - I think the idea here is that because IPython explicitly maintains a concept of directory state (the Python interpreter does not), the user should see files in the current working directory as being on their Python-import-ready sys.path.

Addressing this would involve modifying the directory-navigation IPython magic %cd so that it automatically pushes and pops the current working directory from sys.path during traversal. It doesn't seem unreasonable to me to consider this as a feature request, but yes, it is almost certainly not a bug.

[minrk]

But it does - the first entry on sys.path is '', which evaluates to the CWD at import time, so if you do:

mkdir bar
echo "which = 'just foo'" > foo.py
echo "which = 'in bar'" > bar/foo.py
ipython

Then foo will be whichever one is in the CWD at the time of import:

In [1]: import foo

In [2]: foo.which
Out[2]: 'just foo'

In [3]: cd bar
/Users/minrk/temp/bar

In [4]: reload(foo)
Out[4]: <module 'foo' from 'foo.py'>

In [5]: foo.which
Out[5]: 'in bar'

Plain Python behaves the same way:

>>> import foo
>>> foo.which
'just foo'
>>> import os
>>> os.chdir('bar')
>>> reload(foo)
<module 'foo' from 'foo.pyc'>
>>> foo.which
'in bar'

[ahmadia]

@minrk Consider me taken to school :)

@jennajella - were you using a !cd command instead of IPython's native cd
or %cd syntax? The two do different things (the first launches a new shell
that does a cd command and exits, the second form actually changes the
IPython interpreter state).

On Mon, May 6, 2013 at 8:55 PM, Min RK notifications@github.com wrote:

But it does - the first entry on sys.path is '', which evaluates to the
CWD at import time, so if you do:

mkdir barecho "which = 'just foo'" > foo.pyecho "which = 'in bar'" > bar/foo.py
ipython

Then foo will be whichever one is in the CWD at the time of import:

In [1]: import foo
In [2]: foo.whichOut[2]: 'just foo'
In [3]: cd bar/Users/minrk/temp/bar
In [4]: reload(foo)Out[4]: <module 'foo' from 'foo.py'>
In [5]: foo.whichOut[5]: 'in bar'

Plain Python behaves the same way:

import foo>>> foo.which'just foo'>>> import os>>> os.chdir('bar')>>> reload(foo)<module 'foo' from 'foo.pyc'>>>> foo.which'in bar'

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3283#issuecomment-17504017
.

[minrk]

My guess is that @jennajella is importing foo (or the equivalent), and then changing directory, but foo stays the same because it has already been imported the first time (the reason I had to reload(foo) to get the new local foo after cd.

[jniedziela]

The first item on my path is the ' ', and I was using cd (ipython native). I will read the docs about the differences.

I've confirmed that the example from @minrk works as posted, and when I do the reset on the workspace, foo goes away.

But this still doesn't quite add up, since I've imported the module, run it, changed the directory, cleared the workspace, and then imported again when working in the new directory, but the import is defaulting to the old location. The ipython output for the exercise is below.

jj:sandbox  $ cd DW/

list directory contents

jj:DW  $ ls
RMSD.txt  
FORCE_CONSTANTS         
Nb_CF_Script.py     
Nb_input.py             
kernelGenerator
utils/

start ipython

jj:DW  $ ipython
Python 2.7.2 (default, Oct 11 2012, 20:14:37) 
Type "copyright", "credits" or "license" for more information.

IPython 0.14.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

#run the script.  The location of the NbSetupFiles module is not on the path at this time, so the import fails.

In [1]: run Nb_CF_Script.py
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/Library/Python/2.7/site-packages/ipython-0.14.dev-py2.7.egg/IPython/utils/py3compat.pyc in execfile(fname, *where)
    181             else:
    182                 filename = fname
--> 183             __builtin__.execfile(filename, *where)

/Users/ /Documents/sandbox/DW/Nb_CF_Script.py in <module>()
      1 
      2 
----> 3 from NbSetupFiles import *
      4 
      5 

ImportError: No module named NbSetupFiles

In [2]: quit()

Change to CF directory (predecessor of DW)

jj:DW  $ cd ../CF

list directory contents

NbSetupFiles is in the top level here, and is imported when the script is run in ipython line [1]. Import fails because the remainder of the required packages is not on the path.

jj:CF  $ ls
NbSetupFiles.py     
Nb_m1m1m3_narrow.txt        
kernelGenerator
CalcSQE_Nb_5K.png   
NbSetupFiles.pyc        
Nb_input.py         
Nb_narrow_simulation.txt
utils/
Nb_CF_Script.py 
Nb_m1m1m3_full.txt      
RMSD.txt

jj:CF  $ ipython
Python 2.7.2 (default, Oct 11 2012, 20:14:37) 
Type "copyright", "credits" or "license" for more information.

IPython 0.14.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.


In [1]: run Nb_CF_Script.py
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/Library/Python/2.7/site-packages/ipython-0.14.dev-py2.7.egg/IPython/utils/py3compat.pyc in execfile(fname, *where)
    181             else:
    182                 filename = fname
--> 183             __builtin__.execfile(filename, *where)

/Users/jj/Documents/sandbox/CF/Nb_CF_Script.py in <module>()
      1 
      2 
----> 3 from NbSetupFiles import *
      4 
      5 

/Users/jj/Documents/sandbox/CF/NbSetupFiles.py in <module>()
      2 import matplotlib.pyplot as plt
      3 import matplotlib.pylab
----> 4 from SystemDefinition import *
      5 from BinnedSQE import *
      6 from kernelGenerator.sqeForExistedPhononsForGivenQ import SQECalculatorForExistingPhononsForGivenQ

ImportError: No module named SystemDefinition

I reset the namespace to clear out the old imports

In [2]: reset
Once deleted, variables cannot be recovered. Proceed (y/[n])? y

Change to the desired directory

In [3]: cd ../DW/
/Users/jj/Documents/sandbox/DW

show directory contents - no NbSetupFiles as expected

In [4]: ls
RMSD.txt  
FORCE_CONSTANTS         
Nb_CF_Script.py     
Nb_input.py             
kernelGenerator
utils/

Check that the namespace is empty

In [5]: whos
Interactive namespace is empty.

Run the script, and the old NbSetupFiles is used.

In [6]: run Nb_CF_Script.py
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/Library/Python/2.7/site-packages/ipython-0.14.dev-py2.7.egg/IPython/utils/py3compat.pyc in execfile(fname, *where)
    181             else:
    182                 filename = fname
--> 183             __builtin__.execfile(filename, *where)

/Users/jj/Documents/sandbox/DW/Nb_CF_Script.py in <module>()
      1 
      2 
----> 3 from NbSetupFiles import *
      4 
      5 

/Users/jj/Documents/sandbox/CF/NbSetupFiles.py in <module>()
      2 import matplotlib.pyplot as plt
      3 import matplotlib.pylab
----> 4 from SystemDefinition import *
      5 from BinnedSQE import *
      6 from kernelGenerator.sqeForExistedPhononsForGivenQ import SQECalculatorForExistingPhononsForGivenQ

ImportError: No module named SystemDefinition

[minrk]

I reset the namespace to clear out the old imports

Here is where you are getting tripped up - reset does not 'undo' imports, it only clears out the user namespace. You must use reload (via reload, dreload, or autoreload) if you want to reimport a module from a new file.

[ahmadia]

Python's import caches previously loaded modules, even when you clear your workspace.

You have to explicitly reload(modulename) if you want to reload a new one. To play off Min's example:

In [1]: import foo

In [2]: %reset
Once deleted, variables cannot be recovered. Proceed (y/[n])? y

In [3]: cd bar
/Users/aron/sandbox/playground/bar

In [4]: foo
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-d3b07384d113> in <module>()
----> 1 foo

NameError: name 'foo' is not defined

In [5]: import foo

In [6]: foo.which
Out[6]: 'just foo'

In [7]: reload(foo)
Out[7]: <module 'foo' from 'foo.pyc'>

In [8]: foo.which
Out[8]: 'in bar'

[jniedziela]

OK - I believe this is cleared up - Thanks for your help!