Add directory for startup files #950

minrk · 2011-10-30T20:59:39Z

profile/startup dir is checked for files, and they are run prior to explicit exec_code and exec_files.

a short README is added to the dir explaining how they work, and a short paragraph is added to the docs, in both the interactive tutorial and config overview.

closes #914

fperez · 2011-10-30T21:22:57Z

Looks great, thanks! I'd like to have a test in the suite for this though: a simple temp-generated file with zzz = 123456 and checking afterwards that the initialized instance has zzz with that value should suffice. But at least it will ensure the codepaths for loading startup files do get exercised in the test suite.

Once we get a test, good to go. Great work, this will be really liked by users.

minrk · 2011-10-30T21:38:27Z

I don't quite know how to write that test, but I'll figure it out. I think I need to copy a lot of the env code from test_path.

fperez · 2011-10-30T21:56:01Z

Great, thanks. Let me know if you get stuck and I can pitch in.

minrk · 2011-10-30T22:22:42Z

test added (and fixed a small bug in ipexec_validate, which was needed)

fperez · 2011-10-31T00:09:22Z

Mmh, the check for the startup files/directory must be optional. Right now, IPython won't even start if those files aren't there:

p
Error in sys.excepthook:
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 144, in __call__
    traceback = TBhandler.text(etype,evalue,etb,context=31)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 402, in text
    tb_offset, context)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 964, in structured_traceback
    ipinst = ipapi.get()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ipapi.py", line 28, in get
    return InteractiveShell.instance()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 318, in instance
    inst = cls(*args, **kwargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 384, in __init__
    self.init_profile_dir(profile_dir)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 503, in init_profile_dir
    ProfileDir.create_profile_dir_by_name(self.ipython_dir, 'default')
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 181, in create_profile_dir_by_name
    return cls(location=profile_dir, config=config)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 87, in __init__
    super(Configurable, self).__init__(**kwargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 421, in __init__
    setattr(self, key, value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
    obj._notify_trait(self.name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
    c(name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 86, in _location_changed
    self.startup_dir = os.path.join(new, self.startup_dir_name)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
    obj._notify_trait(self.name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
    c(name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 98, in _startup_dir_changed
    self.check_startup_dir()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 106, in check_startup_dir
    shutil.copy(src, readme)
  File "/usr/lib/python2.7/shutil.py", line 116, in copy
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 81, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/home/fperez/usr/lib/python2.7/site-packages/IPython/config/profile/README_STARTUP'

Having those files and directories should be completely optional... Furthermore, it looks like you forgot to commit the README file itself, I think.

minrk · 2011-10-31T00:27:02Z

The bug is the new readme missing from git. IPython creates any missing dirs, so it doesn't matter if you make them or remove them, it will always run just fine.

The readme is entirely pointless if IPython doesn't create the dir.

On Oct 30, 2011, at 17:09, Fernando Perezreply@reply.github.com wrote:

Mmh, the check for the startup files/directory must be optional. Right now, IPython won't even start if those files aren't there:

p
Error in sys.excepthook:
Traceback (most recent call last):
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 144, in __call__
   traceback = TBhandler.text(etype,evalue,etb,context=31)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 402, in text
   tb_offset, context)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 964, in structured_traceback
   ipinst = ipapi.get()
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ipapi.py", line 28, in get
   return InteractiveShell.instance()
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 318, in instance
   inst = cls(*args, **kwargs)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 384, in __init__
   self.init_profile_dir(profile_dir)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 503, in init_profile_dir
   ProfileDir.create_profile_dir_by_name(self.ipython_dir, 'default')
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 181, in create_profile_dir_by_name
   return cls(location=profile_dir, config=config)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 87, in __init__
   super(Configurable, self).__init__(**kwargs)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 421, in __init__
   setattr(self, key, value)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
   obj._notify_trait(self.name, old_value, new_value)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
   c(name, old_value, new_value)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 86, in _location_changed
   self.startup_dir = os.path.join(new, self.startup_dir_name)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
   obj._notify_trait(self.name, old_value, new_value)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
   c(name, old_value, new_value)
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 98, in _startup_dir_changed
   self.check_startup_dir()
 File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 106, in check_startup_dir
   shutil.copy(src, readme)
 File "/usr/lib/python2.7/shutil.py", line 116, in copy
   copyfile(src, dst)
 File "/usr/lib/python2.7/shutil.py", line 81, in copyfile
   with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/home/fperez/usr/lib/python2.7/site-packages/IPython/config/profile/README_STARTUP'

Having those files and directories should be completely optional... Furthermore, it looks like you forgot to commit the README file itself, I think.

Reply to this email directly or view it on GitHub:
#950 (comment)

Any .py or .ipy file in this directory will be run at startup. Files are run in in lexicographical order, as described in ipython#914

more profile tests should be added to this file

minrk · 2011-10-31T00:30:33Z

Missing readme added to git.

You absolutely don't have to create the dir yourself, IPython will do it for you (just like log, security, pid), and stage the README into it.

fperez · 2011-10-31T00:33:34Z

Looks great now, thanks! Merging...

Add directory for startup files, enabling users to add local configuration more simply without having to know how the config system works much at all. Now, `profile/startup` dir is checked for files, and they are run prior to explicit exec_code and exec_files. A short README is added to the dir explaining how they work, and a short paragraph is added to the docs, in both the interactive tutorial and config overview.

minrk · 2011-10-31T00:38:41Z

awesome, thanks!

pierre-haessig · 2012-04-06T09:45:40Z

Hi,
One question about startup files in terms of namespace. Looking at http://python.6.n6.nabble.com/Run-code-at-startup-td1652498.html it feels like startup files would work in a similar fasion as exec_files and exec_lines.

However, I can use exec_lines to import a module like

c.InteractiveShellApp.exec_lines = ['from future import division']

and then the module is available in the shell
but if I drop the line from __future__ import division in a simple Python startup file, it is not. I guess it is not executed in the shell namespace ?

Was this behavioral difference intentional ?

takluyver · 2012-04-06T14:36:26Z

I don't think that was intentional. Does it affect all import, or just
future statements? future works in a very different way to
normal imports.

pierre-haessig · 2012-04-06T14:56:26Z

Good point ! Indeed I've just checked that for instance import math works well. Thanks for the feedback.

I don't know the internal machinery behind future imports though. I just noticed that there is a complain if the future import statement is not the first one in the file. But I guess it's the same for any Python script. Is it ?

takluyver · 2012-04-06T15:09:28Z

Yes, I think that's the same for any Python file.

OK, the problem is that startup files are run by ip.safe_execfile()
(https://github.com/ipython/ipython/blob/master/IPython/core/interactiveshell.py#L2306),
which doesn't put them through the compiler that remembers future
directives (ip,compile()). We'll need to work out exactly what should
and shouldn't affect the future state of the shell. For now, if
you give the files a .ipy extension, they should go through a
different route where future statements will have an effect.

pierre-haessig · 2012-04-11T13:17:32Z

I've checked that indeed using the .ipy extension makes future imports work. Thanks for the tip.

I feel like this issue is too small to deserve a bug report but I think it should be fixed at some point. In the meantime, my suggestion is to add a footnote in the README in the startup directory that would explain: 1) the problem 2) the solution you mentioned (using .ipy instead of .py).

I propose something like this :

About importing modules : using import statements in startup .py files will make the imported modules available in the running ipython session. This is one of the useful uses of startup scripts.
However, there is an exception for imports from the future module (like division, print_function, ...) which won't be available due to the way the startup scripts are currently executed. This limitation may be lifted in a next release but in the meantime, the simplest solution is to use the .ipy extension which changes the execution path of the script and fixes the problem.

What do you think ?

takluyver · 2012-04-11T14:35:37Z

It may require a note in the documentation, depending on what we decide to do. On reflection, though, using .ipy is a rather ugly hack, and we might decide that it shouldn't work from .ipy files either. A better workaround is to call get_ipython().run_cell('from __future__ import division') in your startup script.

The tricky thing is working out what semantics people expect. If your startup.py includes from __future__ import division, and you do from startup import * at a plain Python prompt, you don't get new style division. These __future__ imports were always intended to work at the module level, without affecting other code in the process. You may want new style division only within that startup file, or you may want it to affect interactive use.

So, where can IPython run interactive code that we need to consider:

At startup: exec_lines, exec_files, startup files. There are .py and .ipy files, and they should definitely behave the same way with future statements.
%run (and %run -i), %rerun, %edit, macros, %loadpy
Extensions
Code loaded by the autoreload extension

My gut feeling is that of those, exec_lines, %loadpy, macros and maybe %run -i should 'leak' future statements to the interactive environment. Everything else should keep future statements to itself (except by using run_cell()). I'll take this to the mailing list.

Add directory for startup files, enabling users to add local configuration more simply without having to know how the config system works much at all. Now, `profile/startup` dir is checked for files, and they are run prior to explicit exec_code and exec_files. A short README is added to the dir explaining how they work, and a short paragraph is added to the docs, in both the interactive tutorial and config overview.

minrk added 5 commits October 30, 2011 17:28

add startup_dir to profiles

7c21f7c

Any .py or .ipy file in this directory will be run at startup. Files are run in in lexicographical order, as described in ipython#914

add README to startup dir

8a68c1b

add startup files to docs

3d36e7e

fix missing options arg to ipexec in ipexec_validate

a4d4260

add test_profile with startup tests

5a904a2

more profile tests should be added to this file

fperez merged commit b131150 into ipython:master Oct 31, 2011

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add directory for startup files #950

Add directory for startup files #950

minrk commented Oct 30, 2011

fperez commented Oct 30, 2011

minrk commented Oct 30, 2011

fperez commented Oct 30, 2011

minrk commented Oct 30, 2011

fperez commented Oct 31, 2011

minrk commented Oct 31, 2011

minrk commented Oct 31, 2011

fperez commented Oct 31, 2011

minrk commented Oct 31, 2011

pierre-haessig commented Apr 6, 2012

takluyver commented Apr 6, 2012

pierre-haessig commented Apr 6, 2012

takluyver commented Apr 6, 2012

pierre-haessig commented Apr 11, 2012

takluyver commented Apr 11, 2012

Add directory for startup files #950

Add directory for startup files #950

Conversation

minrk commented Oct 30, 2011

fperez commented Oct 30, 2011

minrk commented Oct 30, 2011

fperez commented Oct 30, 2011

minrk commented Oct 30, 2011

fperez commented Oct 31, 2011

minrk commented Oct 31, 2011

minrk commented Oct 31, 2011

fperez commented Oct 31, 2011

minrk commented Oct 31, 2011

pierre-haessig commented Apr 6, 2012

takluyver commented Apr 6, 2012

pierre-haessig commented Apr 6, 2012

takluyver commented Apr 6, 2012

pierre-haessig commented Apr 11, 2012

takluyver commented Apr 11, 2012