Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Add directory for startup files #950

Merged
merged 5 commits into from

4 participants

@minrk
Owner

profile/startup dir is checked for files, and they are run prior to explicit exec_code and exec_files.

a short README is added to the dir explaining how they work, and a short paragraph is added to the docs, in both the interactive tutorial and config overview.

closes #914

@fperez
Owner

Looks great, thanks! I'd like to have a test in the suite for this though: a simple temp-generated file with zzz = 123456 and checking afterwards that the initialized instance has zzz with that value should suffice. But at least it will ensure the codepaths for loading startup files do get exercised in the test suite.

Once we get a test, good to go. Great work, this will be really liked by users.

@minrk
Owner

I don't quite know how to write that test, but I'll figure it out. I think I need to copy a lot of the env code from test_path.

@fperez
Owner

Great, thanks. Let me know if you get stuck and I can pitch in.

@minrk
Owner

test added (and fixed a small bug in ipexec_validate, which was needed)

@fperez
Owner

Mmh, the check for the startup files/directory must be optional. Right now, IPython won't even start if those files aren't there:

p
Error in sys.excepthook:
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 144, in __call__
    traceback = TBhandler.text(etype,evalue,etb,context=31)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 402, in text
    tb_offset, context)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 964, in structured_traceback
    ipinst = ipapi.get()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ipapi.py", line 28, in get
    return InteractiveShell.instance()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 318, in instance
    inst = cls(*args, **kwargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 384, in __init__
    self.init_profile_dir(profile_dir)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 503, in init_profile_dir
    ProfileDir.create_profile_dir_by_name(self.ipython_dir, 'default')
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 181, in create_profile_dir_by_name
    return cls(location=profile_dir, config=config)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 87, in __init__
    super(Configurable, self).__init__(**kwargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 421, in __init__
    setattr(self, key, value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
    obj._notify_trait(self.name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
    c(name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 86, in _location_changed
    self.startup_dir = os.path.join(new, self.startup_dir_name)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
    obj._notify_trait(self.name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
    c(name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 98, in _startup_dir_changed
    self.check_startup_dir()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 106, in check_startup_dir
    shutil.copy(src, readme)
  File "/usr/lib/python2.7/shutil.py", line 116, in copy
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 81, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/home/fperez/usr/lib/python2.7/site-packages/IPython/config/profile/README_STARTUP'

Having those files and directories should be completely optional... Furthermore, it looks like you forgot to commit the README file itself, I think.

@minrk
Owner
minrk added some commits
@minrk minrk add startup_dir to profiles
Any .py or .ipy file in this directory will be run at startup.
Files are run in in lexicographical order, as described in #914
7c21f7c
@minrk minrk add README to startup dir 8a68c1b
@minrk minrk add startup files to docs 3d36e7e
@minrk minrk fix missing options arg to ipexec in ipexec_validate a4d4260
@minrk minrk add test_profile with startup tests
more profile tests should be added to this file
5a904a2
@minrk
Owner

Missing readme added to git.

You absolutely don't have to create the dir yourself, IPython will do it for you (just like log, security, pid), and stage the README into it.

@fperez
Owner

Looks great now, thanks! Merging...

@fperez fperez merged commit b131150 into ipython:master
@minrk
Owner

awesome, thanks!

@fperez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
@pierre-haessig

Hi,
One question about startup files in terms of namespace. Looking at http://python.6.n6.nabble.com/Run-code-at-startup-td1652498.html it feels like startup files would work in a similar fasion as exec_files and exec_lines.

However, I can use exec_lines to import a module like

c.InteractiveShellApp.exec_lines = ['from future import division']

and then the module is available in the shell
but if I drop the line from __future__ import division in a simple Python startup file, it is not. I guess it is not executed in the shell namespace ?

Was this behavioral difference intentional ?

@takluyver
Owner
@pierre-haessig

Good point ! Indeed I've just checked that for instance import math works well. Thanks for the feedback.

I don't know the internal machinery behind future imports though. I just noticed that there is a complain if the future import statement is not the first one in the file. But I guess it's the same for any Python script. Is it ?

@takluyver
Owner
@pierre-haessig

I've checked that indeed using the .ipy extension makes future imports work. Thanks for the tip.

I feel like this issue is too small to deserve a bug report but I think it should be fixed at some point. In the meantime, my suggestion is to add a footnote in the README in the startup directory that would explain: 1) the problem 2) the solution you mentioned (using .ipy instead of .py).

I propose something like this :

About importing modules : using import statements in startup .py files will make the imported modules available in the running ipython session. This is one of the useful uses of startup scripts.
However, there is an exception for imports from the future module (like division, print_function, ...) which won't be available due to the way the startup scripts are currently executed. This limitation may be lifted in a next release but in the meantime, the simplest solution is to use the .ipy extension which changes the execution path of the script and fixes the problem.

What do you think ?

@takluyver
Owner

It may require a note in the documentation, depending on what we decide to do. On reflection, though, using .ipy is a rather ugly hack, and we might decide that it shouldn't work from .ipy files either. A better workaround is to call get_ipython().run_cell('from __future__ import division') in your startup script.

The tricky thing is working out what semantics people expect. If your startup.py includes from __future__ import division, and you do from startup import * at a plain Python prompt, you don't get new style division. These __future__ imports were always intended to work at the module level, without affecting other code in the process. You may want new style division only within that startup file, or you may want it to affect interactive use.

So, where can IPython run interactive code that we need to consider:

  • At startup: exec_lines, exec_files, startup files. There are .py and .ipy files, and they should definitely behave the same way with future statements.
  • %run (and %run -i), %rerun, %edit, macros, %loadpy
  • Extensions
  • Code loaded by the autoreload extension

My gut feeling is that of those, exec_lines, %loadpy, macros and maybe %run -i should 'leak' future statements to the interactive environment. Everything else should keep future statements to itself (except by using run_cell()). I'll take this to the mailing list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 31, 2011
  1. @minrk

    add startup_dir to profiles

    minrk authored
    Any .py or .ipy file in this directory will be run at startup.
    Files are run in in lexicographical order, as described in #914
  2. @minrk

    add README to startup dir

    minrk authored
  3. @minrk

    add startup files to docs

    minrk authored
  4. @minrk
  5. @minrk

    add test_profile with startup tests

    minrk authored
    more profile tests should be added to this file
This page is out of date. Refresh to see the latest.
View
11 IPython/config/profile/README_STARTUP
@@ -0,0 +1,11 @@
+This is the IPython startup directory
+
+.py and .ipy files in this directory will be run *prior* to any code or files specified
+via the exec_lines or exec_files configurables whenever you load this profile.
+
+Files will be run in lexicographical order, so you can control the execution order of files
+with a prefix, e.g.::
+
+ 00-first.py
+ 50-middle.py
+ 99-last.ipy
View
15 IPython/core/profiledir.py
@@ -59,9 +59,11 @@ class ProfileDir(LoggingConfigurable):
security_dir_name = Unicode('security')
log_dir_name = Unicode('log')
+ startup_dir_name = Unicode('startup')
pid_dir_name = Unicode('pid')
security_dir = Unicode(u'')
log_dir = Unicode(u'')
+ startup_dir = Unicode(u'')
pid_dir = Unicode(u'')
location = Unicode(u'', config=True,
@@ -81,6 +83,7 @@ def _location_changed(self, name, old, new):
# ensure config files exist:
self.security_dir = os.path.join(new, self.security_dir_name)
self.log_dir = os.path.join(new, self.log_dir_name)
+ self.startup_dir = os.path.join(new, self.startup_dir_name)
self.pid_dir = os.path.join(new, self.pid_dir_name)
self.check_dirs()
@@ -91,6 +94,17 @@ def check_log_dir(self):
if not os.path.isdir(self.log_dir):
os.mkdir(self.log_dir)
+ def _startup_dir_changed(self, name, old, new):
+ self.check_startup_dir()
+
+ def check_startup_dir(self):
+ if not os.path.isdir(self.startup_dir):
+ os.mkdir(self.startup_dir)
+ readme = os.path.join(self.startup_dir, 'README')
+ src = os.path.join(get_ipython_package_dir(), u'config', u'profile', u'README_STARTUP')
+ if not os.path.exists(readme):
+ shutil.copy(src, readme)
+
def _security_dir_changed(self, name, old, new):
self.check_security_dir()
@@ -119,6 +133,7 @@ def check_dirs(self):
self.check_security_dir()
self.check_log_dir()
self.check_pid_dir()
+ self.check_startup_dir()
def copy_config_file(self, config_file, path=None, overwrite=False):
"""Copy a default config file into the active profile directory.
View
18 IPython/core/shellapp.py
@@ -22,6 +22,7 @@
from __future__ import absolute_import
+import glob
import os
import sys
@@ -175,6 +176,7 @@ def init_extensions(self):
def init_code(self):
"""run the pre-flight code, specified via exec_lines"""
+ self._run_startup_files()
self._run_exec_lines()
self._run_exec_files()
self._run_cmd_line_code()
@@ -230,6 +232,22 @@ def _exec_file(self, fname):
finally:
sys.argv = save_argv
+ def _run_startup_files(self):
+ """Run files from profile startup directory"""
+ startup_dir = self.profile_dir.startup_dir
+ startup_files = glob.glob(os.path.join(startup_dir, '*.py'))
+ startup_files += glob.glob(os.path.join(startup_dir, '*.ipy'))
+ if not startup_files:
+ return
+
+ self.log.debug("Running startup files from %s...", startup_dir)
+ try:
+ for fname in sorted(startup_files):
+ self._exec_file(fname)
+ except:
+ self.log.warn("Unknown error in handling startup files:")
+ self.shell.showtraceback()
+
def _run_exec_files(self):
"""Run files from IPythonApp.exec_files"""
if not self.exec_files:
View
103 IPython/core/tests/test_profile.py
@@ -0,0 +1,103 @@
+"""Tests for profile-related functions.
+
+Currently only the startup-dir functionality is tested, but more tests should
+be added for:
+
+ * ipython profile create
+ * ipython profile list
+ * ipython profile create --parallel
+ * security dir permissions
+
+Authors
+-------
+
+* MinRK
+
+"""
+from __future__ import absolute_import
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+import os
+import shutil
+import sys
+import tempfile
+
+import nose.tools as nt
+from nose import SkipTest
+
+from IPython.core.profiledir import ProfileDir
+
+from IPython.testing import decorators as dec
+from IPython.testing import tools as tt
+from IPython.utils import py3compat
+
+
+#-----------------------------------------------------------------------------
+# Globals
+#-----------------------------------------------------------------------------
+TMP_TEST_DIR = tempfile.mkdtemp()
+HOME_TEST_DIR = os.path.join(TMP_TEST_DIR, "home_test_dir")
+IP_TEST_DIR = os.path.join(HOME_TEST_DIR,'.ipython')
+
+#
+# Setup/teardown functions/decorators
+#
+
+def setup():
+ """Setup test environment for the module:
+
+ - Adds dummy home dir tree
+ """
+ # Do not mask exceptions here. In particular, catching WindowsError is a
+ # problem because that exception is only defined on Windows...
+ os.makedirs(IP_TEST_DIR)
+
+
+def teardown():
+ """Teardown test environment for the module:
+
+ - Remove dummy home dir tree
+ """
+ # Note: we remove the parent test dir, which is the root of all test
+ # subdirs we may have created. Use shutil instead of os.removedirs, so
+ # that non-empty directories are all recursively removed.
+ shutil.rmtree(TMP_TEST_DIR)
+
+
+#-----------------------------------------------------------------------------
+# Test functions
+#-----------------------------------------------------------------------------
+
+def test_startup_py():
+ # create profile dir
+ pd = ProfileDir.create_profile_dir_by_name(IP_TEST_DIR, 'test')
+ # write startup python file
+ with open(os.path.join(pd.startup_dir, '00-start.py'), 'w') as f:
+ f.write('zzz=123\n')
+ # write simple test file, to check that the startup file was run
+ fname = os.path.join(TMP_TEST_DIR, 'test.py')
+ with open(fname, 'w') as f:
+ f.write('print zzz\n')
+ # validate output
+ tt.ipexec_validate(fname, '123', '',
+ options=['--ipython-dir', IP_TEST_DIR, '--profile', 'test'])
+
+def test_startup_ipy():
+ # create profile dir
+ pd = ProfileDir.create_profile_dir_by_name(IP_TEST_DIR, 'test')
+ # write startup ipython file
+ with open(os.path.join(pd.startup_dir, '00-start.ipy'), 'w') as f:
+ f.write('%profile\n')
+ # write empty script, because we don't need anything to happen
+ # after the startup file is run
+ fname = os.path.join(TMP_TEST_DIR, 'test.py')
+ with open(fname, 'w') as f:
+ f.write('')
+ # validate output
+ tt.ipexec_validate(fname, 'test', '',
+ options=['--ipython-dir', IP_TEST_DIR, '--profile', 'test'])
+
+
View
2  IPython/testing/tools.py
@@ -259,7 +259,7 @@ def ipexec_validate(fname, expected_out, expected_err='',
import nose.tools as nt
- out, err = ipexec(fname)
+ out, err = ipexec(fname, options)
#print 'OUT', out # dbg
#print 'ERR', err # dbg
# If there are any errors, we must check those befor stdout, as they may be
View
29 docs/source/config/overview.txt
@@ -358,6 +358,35 @@ you create profiles with the name of one of our shipped profiles, these config
files will be copied over instead of starting with the automatically generated
config files.
+Security Files
+--------------
+
+If you are using the notebook, qtconsole, or parallel code, IPython stores
+connection information in small JSON files in the active profile's security
+directory. This directory is made private, so only you can see the files inside. If
+you need to move connection files around to other computers, this is where they will
+be. If you want your code to be able to open security files by name, we have a
+convenience function :func:`IPython.utils.path.get_security_file`, which will return
+the absolute path to a security file from its filename and [optionally] profile
+name.
+
+Startup Files
+-------------
+
+If you want some code to be run at the beginning of every IPython session with a
+particular profile, the easiest way is to add Python (.py) or IPython (.ipy) scripts
+to your :file:`<profile>/startup` directory. Files in this directory will always be
+executed as soon as the IPython shell is constructed, and before any other code or
+scripts you have specified. If you have multiple files in the startup directory,
+they will be run in lexicographical order, so you can control the ordering by adding
+a '00-' prefix.
+
+.. note::
+
+ Automatic startup files are new in IPython 0.12. Use the
+ InteractiveShellApp.exec_files configurable for similar behavior in 0.11.
+
+
.. _commandline:
Command-line arguments
View
15 docs/source/interactive/tutorial.txt
@@ -142,4 +142,19 @@ Profiles allow you to use IPython for different tasks, keeping separate config
files and history for each one. More details in :ref:`the profiles section
<profiles>`.
+Startup Files
+-------------
+
+If you want some code to be run at the beginning of every IPython session, the
+easiest way is to add Python (.py) or IPython (.ipy) scripts to your
+:file:`<profile>/startup` directory. Files in this directory will always be executed
+as soon as the IPython shell is constructed, and before any other code or scripts
+you have specified. If you have multiple files in the startup directory, they will
+be run in lexicographical order, so you can control the ordering by adding a '00-'
+prefix.
+
+.. note::
+
+ Automatic startup files are new in IPython 0.12. Use the
+ InteractiveShellApp.exec_files configurable for similar behavior in 0.11.
View
2  setupbase.py
@@ -132,7 +132,7 @@ def find_package_data():
static_data.append(os.path.join(parent, f))
package_data = {
- 'IPython.config.profile' : ['README', '*/*.py'],
+ 'IPython.config.profile' : ['README*', '*/*.py'],
'IPython.testing' : ['*.txt'],
'IPython.frontend.html.notebook' : ['templates/*'] + static_data,
'IPython.frontend.qt.console' : ['resources/icon/*.svg'],
Something went wrong with that request. Please try again.