Skip to content
This repository

Add directory for startup files #950

Merged
merged 5 commits into from over 2 years ago

4 participants

Min RK Fernando Perez Pierre Haessig Thomas Kluyver
Min RK
Owner

profile/startup dir is checked for files, and they are run prior to explicit exec_code and exec_files.

a short README is added to the dir explaining how they work, and a short paragraph is added to the docs, in both the interactive tutorial and config overview.

closes #914

Fernando Perez
Owner

Looks great, thanks! I'd like to have a test in the suite for this though: a simple temp-generated file with zzz = 123456 and checking afterwards that the initialized instance has zzz with that value should suffice. But at least it will ensure the codepaths for loading startup files do get exercised in the test suite.

Once we get a test, good to go. Great work, this will be really liked by users.

Min RK
Owner

I don't quite know how to write that test, but I'll figure it out. I think I need to copy a lot of the env code from test_path.

Fernando Perez
Owner

Great, thanks. Let me know if you get stuck and I can pitch in.

Min RK
Owner

test added (and fixed a small bug in ipexec_validate, which was needed)

Fernando Perez
Owner

Mmh, the check for the startup files/directory must be optional. Right now, IPython won't even start if those files aren't there:

p
Error in sys.excepthook:
Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 144, in __call__
    traceback = TBhandler.text(etype,evalue,etb,context=31)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 402, in text
    tb_offset, context)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ultratb.py", line 964, in structured_traceback
    ipinst = ipapi.get()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/ipapi.py", line 28, in get
    return InteractiveShell.instance()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 318, in instance
    inst = cls(*args, **kwargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 384, in __init__
    self.init_profile_dir(profile_dir)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 503, in init_profile_dir
    ProfileDir.create_profile_dir_by_name(self.ipython_dir, 'default')
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 181, in create_profile_dir_by_name
    return cls(location=profile_dir, config=config)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 87, in __init__
    super(Configurable, self).__init__(**kwargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 421, in __init__
    setattr(self, key, value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
    obj._notify_trait(self.name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
    c(name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 86, in _location_changed
    self.startup_dir = os.path.join(new, self.startup_dir_name)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 307, in __set__
    obj._notify_trait(self.name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/traitlets.py", line 458, in _notify_trait
    c(name, old_value, new_value)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 98, in _startup_dir_changed
    self.check_startup_dir()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/profiledir.py", line 106, in check_startup_dir
    shutil.copy(src, readme)
  File "/usr/lib/python2.7/shutil.py", line 116, in copy
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 81, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/home/fperez/usr/lib/python2.7/site-packages/IPython/config/profile/README_STARTUP'

Having those files and directories should be completely optional... Furthermore, it looks like you forgot to commit the README file itself, I think.

Min RK
Owner
added some commits October 21, 2011
Min RK add startup_dir to profiles
Any .py or .ipy file in this directory will be run at startup.
Files are run in in lexicographical order, as described in #914
7c21f7c
Min RK add README to startup dir 8a68c1b
Min RK add startup files to docs 3d36e7e
Min RK fix missing options arg to ipexec in ipexec_validate a4d4260
Min RK add test_profile with startup tests
more profile tests should be added to this file
5a904a2
Min RK
Owner

Missing readme added to git.

You absolutely don't have to create the dir yourself, IPython will do it for you (just like log, security, pid), and stage the README into it.

Fernando Perez
Owner

Looks great now, thanks! Merging...

Fernando Perez fperez merged commit b131150 into from October 30, 2011
Fernando Perez fperez closed this October 30, 2011
Min RK
Owner

awesome, thanks!

Fernando Perez fperez referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Pierre Haessig

Hi,
One question about startup files in terms of namespace. Looking at http://python.6.n6.nabble.com/Run-code-at-startup-td1652498.html it feels like startup files would work in a similar fasion as exec_files and exec_lines.

However, I can use exec_lines to import a module like

c.InteractiveShellApp.exec_lines = ['from future import division']

and then the module is available in the shell
but if I drop the line from __future__ import division in a simple Python startup file, it is not. I guess it is not executed in the shell namespace ?

Was this behavioral difference intentional ?

Thomas Kluyver
Collaborator
Pierre Haessig

Good point ! Indeed I've just checked that for instance import math works well. Thanks for the feedback.

I don't know the internal machinery behind future imports though. I just noticed that there is a complain if the future import statement is not the first one in the file. But I guess it's the same for any Python script. Is it ?

Thomas Kluyver
Collaborator
Pierre Haessig

I've checked that indeed using the .ipy extension makes future imports work. Thanks for the tip.

I feel like this issue is too small to deserve a bug report but I think it should be fixed at some point. In the meantime, my suggestion is to add a footnote in the README in the startup directory that would explain: 1) the problem 2) the solution you mentioned (using .ipy instead of .py).

I propose something like this :

About importing modules : using import statements in startup .py files will make the imported modules available in the running ipython session. This is one of the useful uses of startup scripts.
However, there is an exception for imports from the future module (like division, print_function, ...) which won't be available due to the way the startup scripts are currently executed. This limitation may be lifted in a next release but in the meantime, the simplest solution is to use the .ipy extension which changes the execution path of the script and fixes the problem.

What do you think ?

Thomas Kluyver
Collaborator

It may require a note in the documentation, depending on what we decide to do. On reflection, though, using .ipy is a rather ugly hack, and we might decide that it shouldn't work from .ipy files either. A better workaround is to call get_ipython().run_cell('from __future__ import division') in your startup script.

The tricky thing is working out what semantics people expect. If your startup.py includes from __future__ import division, and you do from startup import * at a plain Python prompt, you don't get new style division. These __future__ imports were always intended to work at the module level, without affecting other code in the process. You may want new style division only within that startup file, or you may want it to affect interactive use.

So, where can IPython run interactive code that we need to consider:

  • At startup: exec_lines, exec_files, startup files. There are .py and .ipy files, and they should definitely behave the same way with future statements.
  • %run (and %run -i), %rerun, %edit, macros, %loadpy
  • Extensions
  • Code loaded by the autoreload extension

My gut feeling is that of those, exec_lines, %loadpy, macros and maybe %run -i should 'leak' future statements to the interactive environment. Everything else should keep future statements to itself (except by using run_cell()). I'll take this to the mailing list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 5 unique commits by 1 author.

Oct 30, 2011
Min RK add startup_dir to profiles
Any .py or .ipy file in this directory will be run at startup.
Files are run in in lexicographical order, as described in #914
7c21f7c
Min RK add README to startup dir 8a68c1b
Min RK add startup files to docs 3d36e7e
Min RK fix missing options arg to ipexec in ipexec_validate a4d4260
Min RK add test_profile with startup tests
more profile tests should be added to this file
5a904a2
This page is out of date. Refresh to see the latest.
11  IPython/config/profile/README_STARTUP
... ...
@@ -0,0 +1,11 @@
  1
+This is the IPython startup directory
  2
+
  3
+.py and .ipy files in this directory will be run *prior* to any code or files specified
  4
+via the exec_lines or exec_files configurables whenever you load this profile.
  5
+
  6
+Files will be run in lexicographical order, so you can control the execution order of files
  7
+with a prefix, e.g.::
  8
+
  9
+    00-first.py
  10
+    50-middle.py
  11
+    99-last.ipy
15  IPython/core/profiledir.py
@@ -59,9 +59,11 @@ class ProfileDir(LoggingConfigurable):
59 59
 
60 60
     security_dir_name = Unicode('security')
61 61
     log_dir_name = Unicode('log')
  62
+    startup_dir_name = Unicode('startup')
62 63
     pid_dir_name = Unicode('pid')
63 64
     security_dir = Unicode(u'')
64 65
     log_dir = Unicode(u'')
  66
+    startup_dir = Unicode(u'')
65 67
     pid_dir = Unicode(u'')
66 68
 
67 69
     location = Unicode(u'', config=True,
@@ -81,6 +83,7 @@ def _location_changed(self, name, old, new):
81 83
         # ensure config files exist:
82 84
         self.security_dir = os.path.join(new, self.security_dir_name)
83 85
         self.log_dir = os.path.join(new, self.log_dir_name)
  86
+        self.startup_dir = os.path.join(new, self.startup_dir_name)
84 87
         self.pid_dir = os.path.join(new, self.pid_dir_name)
85 88
         self.check_dirs()
86 89
 
@@ -91,6 +94,17 @@ def check_log_dir(self):
91 94
         if not os.path.isdir(self.log_dir):
92 95
             os.mkdir(self.log_dir)
93 96
 
  97
+    def _startup_dir_changed(self, name, old, new):
  98
+        self.check_startup_dir()
  99
+
  100
+    def check_startup_dir(self):
  101
+        if not os.path.isdir(self.startup_dir):
  102
+            os.mkdir(self.startup_dir)
  103
+        readme = os.path.join(self.startup_dir, 'README')
  104
+        src = os.path.join(get_ipython_package_dir(), u'config', u'profile', u'README_STARTUP')
  105
+        if not os.path.exists(readme):
  106
+            shutil.copy(src, readme)
  107
+
94 108
     def _security_dir_changed(self, name, old, new):
95 109
         self.check_security_dir()
96 110
 
@@ -119,6 +133,7 @@ def check_dirs(self):
119 133
         self.check_security_dir()
120 134
         self.check_log_dir()
121 135
         self.check_pid_dir()
  136
+        self.check_startup_dir()
122 137
 
123 138
     def copy_config_file(self, config_file, path=None, overwrite=False):
124 139
         """Copy a default config file into the active profile directory.
18  IPython/core/shellapp.py
@@ -22,6 +22,7 @@
22 22
 
23 23
 from __future__ import absolute_import
24 24
 
  25
+import glob
25 26
 import os
26 27
 import sys
27 28
 
@@ -175,6 +176,7 @@ def init_extensions(self):
175 176
 
176 177
     def init_code(self):
177 178
         """run the pre-flight code, specified via exec_lines"""
  179
+        self._run_startup_files()
178 180
         self._run_exec_lines()
179 181
         self._run_exec_files()
180 182
         self._run_cmd_line_code()
@@ -230,6 +232,22 @@ def _exec_file(self, fname):
230 232
         finally:
231 233
             sys.argv = save_argv
232 234
 
  235
+    def _run_startup_files(self):
  236
+        """Run files from profile startup directory"""
  237
+        startup_dir = self.profile_dir.startup_dir
  238
+        startup_files = glob.glob(os.path.join(startup_dir, '*.py'))
  239
+        startup_files += glob.glob(os.path.join(startup_dir, '*.ipy'))
  240
+        if not startup_files:
  241
+            return
  242
+        
  243
+        self.log.debug("Running startup files from %s...", startup_dir)
  244
+        try:
  245
+            for fname in sorted(startup_files):
  246
+                self._exec_file(fname)
  247
+        except:
  248
+            self.log.warn("Unknown error in handling startup files:")
  249
+            self.shell.showtraceback()
  250
+
233 251
     def _run_exec_files(self):
234 252
         """Run files from IPythonApp.exec_files"""
235 253
         if not self.exec_files:
103  IPython/core/tests/test_profile.py
... ...
@@ -0,0 +1,103 @@
  1
+"""Tests for profile-related functions.
  2
+
  3
+Currently only the startup-dir functionality is tested, but more tests should
  4
+be added for:
  5
+
  6
+    * ipython profile create
  7
+    * ipython profile list
  8
+    * ipython profile create --parallel
  9
+    * security dir permissions
  10
+
  11
+Authors
  12
+-------
  13
+
  14
+* MinRK
  15
+
  16
+"""
  17
+from __future__ import absolute_import
  18
+
  19
+#-----------------------------------------------------------------------------
  20
+# Imports
  21
+#-----------------------------------------------------------------------------
  22
+
  23
+import os
  24
+import shutil
  25
+import sys
  26
+import tempfile
  27
+
  28
+import nose.tools as nt
  29
+from nose import SkipTest
  30
+
  31
+from IPython.core.profiledir import ProfileDir
  32
+
  33
+from IPython.testing import decorators as dec
  34
+from IPython.testing import tools as tt
  35
+from IPython.utils import py3compat
  36
+
  37
+
  38
+#-----------------------------------------------------------------------------
  39
+# Globals
  40
+#-----------------------------------------------------------------------------
  41
+TMP_TEST_DIR = tempfile.mkdtemp()
  42
+HOME_TEST_DIR = os.path.join(TMP_TEST_DIR, "home_test_dir")
  43
+IP_TEST_DIR = os.path.join(HOME_TEST_DIR,'.ipython')
  44
+
  45
+#
  46
+# Setup/teardown functions/decorators
  47
+#
  48
+
  49
+def setup():
  50
+    """Setup test environment for the module:
  51
+
  52
+            - Adds dummy home dir tree
  53
+    """
  54
+    # Do not mask exceptions here.  In particular, catching WindowsError is a
  55
+    # problem because that exception is only defined on Windows...
  56
+    os.makedirs(IP_TEST_DIR)
  57
+
  58
+
  59
+def teardown():
  60
+    """Teardown test environment for the module:
  61
+
  62
+            - Remove dummy home dir tree
  63
+    """
  64
+    # Note: we remove the parent test dir, which is the root of all test
  65
+    # subdirs we may have created.  Use shutil instead of os.removedirs, so
  66
+    # that non-empty directories are all recursively removed.
  67
+    shutil.rmtree(TMP_TEST_DIR)
  68
+
  69
+
  70
+#-----------------------------------------------------------------------------
  71
+# Test functions
  72
+#-----------------------------------------------------------------------------
  73
+
  74
+def test_startup_py():
  75
+    # create profile dir
  76
+    pd = ProfileDir.create_profile_dir_by_name(IP_TEST_DIR, 'test')
  77
+    # write startup python file
  78
+    with open(os.path.join(pd.startup_dir, '00-start.py'), 'w') as f:
  79
+        f.write('zzz=123\n')
  80
+    # write simple test file, to check that the startup file was run
  81
+    fname = os.path.join(TMP_TEST_DIR, 'test.py')
  82
+    with open(fname, 'w') as f:
  83
+        f.write('print zzz\n')
  84
+    # validate output
  85
+    tt.ipexec_validate(fname, '123', '', 
  86
+        options=['--ipython-dir', IP_TEST_DIR, '--profile', 'test'])
  87
+
  88
+def test_startup_ipy():
  89
+    # create profile dir
  90
+    pd = ProfileDir.create_profile_dir_by_name(IP_TEST_DIR, 'test')
  91
+    # write startup ipython file
  92
+    with open(os.path.join(pd.startup_dir, '00-start.ipy'), 'w') as f:
  93
+        f.write('%profile\n')
  94
+    # write empty script, because we don't need anything to happen
  95
+    # after the startup file is run
  96
+    fname = os.path.join(TMP_TEST_DIR, 'test.py')
  97
+    with open(fname, 'w') as f:
  98
+        f.write('')
  99
+    # validate output
  100
+    tt.ipexec_validate(fname, 'test', '', 
  101
+        options=['--ipython-dir', IP_TEST_DIR, '--profile', 'test'])
  102
+    
  103
+    
2  IPython/testing/tools.py
@@ -259,7 +259,7 @@ def ipexec_validate(fname, expected_out, expected_err='',
259 259
 
260 260
     import nose.tools as nt
261 261
 
262  
-    out, err = ipexec(fname)
  262
+    out, err = ipexec(fname, options)
263 263
     #print 'OUT', out  # dbg
264 264
     #print 'ERR', err  # dbg
265 265
     # If there are any errors, we must check those befor stdout, as they may be
29  docs/source/config/overview.txt
@@ -358,6 +358,35 @@ you create profiles with the name of one of our shipped profiles, these config
358 358
 files will be copied over instead of starting with the automatically generated
359 359
 config files.
360 360
 
  361
+Security Files
  362
+--------------
  363
+
  364
+If you are using the notebook, qtconsole, or parallel code, IPython stores
  365
+connection information in small JSON files in the active profile's security
  366
+directory. This directory is made private, so only you can see the files inside. If
  367
+you need to move connection files around to other computers, this is where they will
  368
+be. If you want your code to be able to open security files by name, we have a
  369
+convenience function :func:`IPython.utils.path.get_security_file`, which will return
  370
+the absolute path to a security file from its filename and [optionally] profile
  371
+name.
  372
+
  373
+Startup Files
  374
+-------------
  375
+
  376
+If you want some code to be run at the beginning of every IPython session with a
  377
+particular profile, the easiest way is to add Python (.py) or IPython (.ipy) scripts
  378
+to your :file:`<profile>/startup` directory. Files in this directory will always be
  379
+executed as soon as the IPython shell is constructed, and before any other code or
  380
+scripts you have specified. If you have multiple files in the startup directory,
  381
+they will be run in lexicographical order, so you can control the ordering by adding
  382
+a '00-' prefix.
  383
+
  384
+.. note::
  385
+
  386
+    Automatic startup files are new in IPython 0.12. Use the
  387
+    InteractiveShellApp.exec_files configurable for similar behavior in 0.11.
  388
+
  389
+
361 390
 .. _commandline:
362 391
 
363 392
 Command-line arguments
15  docs/source/interactive/tutorial.txt
@@ -142,4 +142,19 @@ Profiles allow you to use IPython for different tasks, keeping separate config
142 142
 files and history for each one. More details in :ref:`the profiles section
143 143
 <profiles>`.
144 144
 
  145
+Startup Files
  146
+-------------
  147
+
  148
+If you want some code to be run at the beginning of every IPython session, the
  149
+easiest way is to add Python (.py) or IPython (.ipy) scripts to your
  150
+:file:`<profile>/startup` directory. Files in this directory will always be executed
  151
+as soon as the IPython shell is constructed, and before any other code or scripts
  152
+you have specified. If you have multiple files in the startup directory, they will
  153
+be run in lexicographical order, so you can control the ordering by adding a '00-'
  154
+prefix.
  155
+
  156
+.. note::
  157
+
  158
+    Automatic startup files are new in IPython 0.12. Use the
  159
+    InteractiveShellApp.exec_files configurable for similar behavior in 0.11.
145 160
 
2  setupbase.py
@@ -132,7 +132,7 @@ def find_package_data():
132 132
             static_data.append(os.path.join(parent, f))
133 133
 
134 134
     package_data = {
135  
-        'IPython.config.profile' : ['README', '*/*.py'],
  135
+        'IPython.config.profile' : ['README*', '*/*.py'],
136 136
         'IPython.testing' : ['*.txt'],
137 137
         'IPython.frontend.html.notebook' : ['templates/*'] + static_data,
138 138
         'IPython.frontend.qt.console' : ['resources/icon/*.svg'],
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.