Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Adding -m option to %run, similar to -m for python interpreter. #819

Closed
wants to merge 5 commits into from

3 participants

Jörgen Stenarson Thomas Kluyver Fernando Perez
Jörgen Stenarson
Collaborator

I have long been missing an option to %run that is similar to -m for the regular python interpreter.

The -m option searches for a module on the python path and launches that module as a script.

This pull request is a first shot at this implementation. can you see anything that is missing?

I added some helper functions at top-level for locating modules, should they be somewhere else?

/Jörgen

IPython/core/magic.py
((17 lines not shown))
mode='list',list_all=1)
+ if opts.has_key("m"):
+ modulename = opts.get("m")[0]
+ arg_lst = [find_mod(modulename)]
Thomas Kluyver Owner

This is currently going to replace all the arguments with the path to the module. Does Python itself allow you to do this sort of thing:

python -m somepkg.somemodule several args here

Fernando Perez Owner
fperez added a note

@jstenar, one more thing for when you get a chance: let's use if "m" in opts instead of has_key, which is the more modern way to spell it out. We may have lingering uses of has_key but we shouldn't be using that for new code.

Fernando Perez Owner
fperez added a note

And further, since you know that opts has m, then you can use modulename = opts['m'][0]. You only need to use get in dict access when you don't know for sure the key is available.

Jörgen Stenarson Collaborator
jstenar added a note

Should I remove old has_key uses in magic_run as well?

Thomas Kluyver Owner

That's probably worth doing while you're working on it.

Jörgen Stenarson Collaborator
jstenar added a note

I just pushed some changes that address these issues. I also did some pep-8 cleaning of magic_run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/core/magic.py
((18 lines not shown))
+ """Get __init__ file path for module with directory dirname
+ """
+ fbase = os.path.join(dirname, "__init__")
+ for ext in [".py", ".pyw", ".pyc", ".pyo"]:
+ fname = fbase + ext
+ if os.path.isfile(fname):
+ return fname
+
+
+def find_mod(name):
+ """Find module *name* on sys.path
+ """
+ parts = name.split(".")
+ if len(parts) == 1:
+ basepath = find_module(parts[0])
+ else:
Thomas Kluyver Owner

Doesn't look like you actually need this if/else test. Just doing the bit in else should be enough.

Jörgen Stenarson Collaborator
jstenar added a note

Changed in c16a5a0a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Thomas Kluyver
Owner

Another quick question - if the user for some reason does %run -m (with no module name) or %run -m misspelt.modname, how gracefully will this fail?

Jörgen Stenarson
Collaborator
Thomas Kluyver
Owner

That's fine, there's no rush. I was just checking through the list of pull requests. Hope you're well!

Jörgen Stenarson
Collaborator

Running %run -m without argument will fail with an error during parse_options:

In [1]: %run -m
UsageError: option -m requires argument ( allowed: "nidtN:b:pD:l:rs:T:em:" )
Fernando Perez
Owner

@jstenar, that error mode seems OK, as it gives the user enough information on what the issue is, so I'm OK with that. Did you see a problem with it?

On the other hand, due to other recent merges, this PR doesn't currently merge, so you'll need to give it a rebase and do a push --force. Let us know if you need a hand with any of that...

J�rgen Stena... added some commits
J�rgen Stenarson Adding -m option to %run, similar to -m for python interpreter.
Added helper functions at top-level, should they be somewhere else?
55eff15
J�rgen Stenarson Improve error handling for %run -m 42da76c
J�rgen Stenarson Replace has_key with in in magic_run and some pep-8 fixes. 269e9c9
Jörgen Stenarson
Collaborator

I think I did the rebase correctly (my first one).

While I'm mucking about in magic_run I could take a stab at issue #710 as well or should I do that in a separate branch?

Thomas Kluyver
Owner

I think let's do that in a separate branch - magic_run is some of the more thorny code, so it's worth keeping a clear eye on what's going on with it.

Jörgen Stenarson
Collaborator
IPython/core/magic.py
@@ -91,6 +91,49 @@ def needs_local_scope(func):
func.needs_local_scope = True
return func
+import imp, os
+
+def find_module(name, path=None):
+ """imp.find_module variant that only return path of module.
+ Return None if module is missing or does not have .py or .pyw extension
+ """
Fernando Perez Owner
fperez added a note

Let's try to follow the dev guidelines for docstrings; here it is for reference:

http://ipython.org/ipython-doc/dev/development/doc_guide.html#docstring-format

This ensures that we'll have consistency throughout, and that auto-generated API docs have the right signatures.

We just use the numpy standard, so it's nothing particularly complicated or special-case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/core/magic.py
@@ -91,6 +91,49 @@ def needs_local_scope(func):
func.needs_local_scope = True
return func
+import imp, os
Fernando Perez Owner
fperez added a note

In general, we keep imports at the top of the file and in their own line, alphabetically sorted. Just see the top of the file.

That makes it easier to see at a glance what any given module needs. The only exception is for particularly expensive or potentially conflicting dependencies, that get imported inside the functions that use them (such as gui tools).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner

@jstenar, thanks for the updated work! I think we're almost there. But I see there are three new functions in the magic module now, so in addition to my in-line feedback, a couple of more general comments:

  • they are fairly generic utility functions related to module handling, so they should probably go elsewhere. I'd suggest a new module in IPython.utils, named something like modpaths.py. You can find a template with the necessary boilerplate at /docs/source/development/template.py.

  • Now that you have a nice new module with just three functions in it, the next thing is to ensure that you also add in utils/tests a file named test_modpaths.py that tests each one of these in isolation.

This way, the code will be easier to reuse elsewhere and will be validated. We want to move away from putting more and more tools into massive files like magic.py, that is already very unweildy.

So with a bit more iteration, we're close to having this solid for merge. Thanks for being patient, but that's how we ensure we build a more robust project moving forward!

Fernando Perez fperez closed this pull request from a commit
Jörgen Stenarson Add -m option to %run to launch a module as a script.
The new -m option searches for a module on the python path and launches
that module as a script.  This is similar to Python's -m command-line
flag in behavior, but while staying inside IPython.

Conflicts:
	IPython/core/magic.py

Closes gh-819.
a4cfe5c
Fernando Perez fperez closed this in a4cfe5c
Jörgen Stenarson
Collaborator
Fernando Perez
Owner

@jstenar: yes, I'd seen that, and the PR was merged 4 days ago :) Good work!

Jörgen Stenarson
Collaborator
Fernando Perez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
matthew von rocketstein mattvonrocketstein referenced this pull request from a commit in mattvonrocketstein/ipython
Jörgen Stenarson Add -m option to %run to launch a module as a script.
The new -m option searches for a module on the python path and launches
that module as a script.  This is similar to Python's -m command-line
flag in behavior, but while staying inside IPython.

Conflicts:
	IPython/core/magic.py

Closes gh-819.
a34330c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 9, 2011
  1. Adding -m option to %run, similar to -m for python interpreter.

    J�rgen Stenarson authored
    Added helper functions at top-level, should they be somewhere else?
  2. Improve error handling for %run -m

    J�rgen Stenarson authored
  3. Replace has_key with in in magic_run and some pep-8 fixes.

    J�rgen Stenarson authored
  4. Fixing import statements and improving docstrings.

    J�rgen Stenarson authored
Commits on Oct 10, 2011
  1. Moving helper functions to utils.module_paths, adding tests.

    J�rgen Stenarson authored
This page is out of date. Refresh to see the latest.
88 IPython/core/magic.py
View
@@ -19,6 +19,7 @@
import __future__
import bdb
import inspect
+import imp
import os
import sys
import shutil
@@ -54,6 +55,7 @@
from IPython.testing.skipdoctest import skip_doctest
from IPython.utils import py3compat
from IPython.utils.io import file_read, nlprint
+from IPython.utils.module_paths import find_mod
from IPython.utils.path import get_py_filename, unquote_filename
from IPython.utils.process import arg_split, abbrev_cwd
from IPython.utils.terminal import set_term_title
@@ -91,6 +93,7 @@ def needs_local_scope(func):
func.needs_local_scope = True
return func
+
# Used for exception handling in magic_edit
class MacroToEdit(ValueError): pass
@@ -1448,7 +1451,7 @@ def magic_prun(self, parameter_s ='',user_mode=1,
return None
@skip_doctest
- def magic_run(self, parameter_s ='',runner=None,
+ def magic_run(self, parameter_s ='', runner=None,
file_finder=get_py_filename):
"""Run the named file inside IPython as a program.
@@ -1557,17 +1560,34 @@ def magic_run(self, parameter_s ='',runner=None,
There is one special usage for which the text above doesn't apply:
if the filename ends with .ipy, the file is run as ipython script,
just as if the commands were written on IPython prompt.
+
+ -m: specify module name to load instead of script path. Similar to
+ the -m option for the python interpreter. Use this option last if you
+ want to combine with other %run options. Unlike the python interpreter
+ only source modules are allowed no .pyc or .pyo files.
+ For example:
+
+ %run -m example
+
+ will run the example module.
+
"""
# get arguments and set sys.argv for program to be run.
- opts,arg_lst = self.parse_options(parameter_s,'nidtN:b:pD:l:rs:T:e',
- mode='list',list_all=1)
-
+ opts, arg_lst = self.parse_options(parameter_s, 'nidtN:b:pD:l:rs:T:em:',
+ mode='list', list_all=1)
+ if "m" in opts:
+ modulename = opts["m"][0]
+ modpath = find_mod(modulename)
+ if modpath is None:
+ warn('%r is not a valid modulename on sys.path'%modulename)
+ return
+ arg_lst = [modpath] + arg_lst
try:
filename = file_finder(arg_lst[0])
except IndexError:
warn('you must provide at least a filename.')
- print '\n%run:\n',oinspect.getdoc(self.magic_run)
+ print '\n%run:\n', oinspect.getdoc(self.magic_run)
return
except IOError as e:
try:
@@ -1582,7 +1602,7 @@ def magic_run(self, parameter_s ='',runner=None,
return
# Control the response to exit() calls made by the script being run
- exit_ignore = opts.has_key('e')
+ exit_ignore = 'e' in opts
# Make sure that the running script gets a proper sys.argv as if it
# were run from a system shell.
@@ -1591,9 +1611,9 @@ def magic_run(self, parameter_s ='',runner=None,
# simulate shell expansion on arguments, at least tilde expansion
args = [ os.path.expanduser(a) for a in arg_lst[1:] ]
- sys.argv = [filename]+ args # put in the proper filename
+ sys.argv = [filename] + args # put in the proper filename
- if opts.has_key('i'):
+ if 'i' in opts:
# Run in user's interactive namespace
prog_ns = self.shell.user_ns
__name__save = self.shell.user_ns['__name__']
@@ -1601,7 +1621,7 @@ def magic_run(self, parameter_s ='',runner=None,
main_mod = self.shell.new_main_mod(prog_ns)
else:
# Run in a fresh, empty namespace
- if opts.has_key('n'):
+ if 'n' in opts:
name = os.path.splitext(os.path.basename(filename))[0]
else:
name = '__main__'
@@ -1630,10 +1650,10 @@ def magic_run(self, parameter_s ='',runner=None,
try:
stats = None
with self.readline_no_record:
- if opts.has_key('p'):
- stats = self.magic_prun('',0,opts,arg_lst,prog_ns)
+ if 'p' in opts:
+ stats = self.magic_prun('', 0, opts, arg_lst, prog_ns)
else:
- if opts.has_key('d'):
+ if 'd' in opts:
deb = debugger.Pdb(self.shell.colors)
# reset Breakpoint state, which is moronically kept
# in a class
@@ -1642,11 +1662,11 @@ def magic_run(self, parameter_s ='',runner=None,
bdb.Breakpoint.bpbynumber = [None]
# Set an initial breakpoint to stop execution
maxtries = 10
- bp = int(opts.get('b',[1])[0])
- checkline = deb.checkline(filename,bp)
+ bp = int(opts.get('b', [1])[0])
+ checkline = deb.checkline(filename, bp)
if not checkline:
- for bp in range(bp+1,bp+maxtries+1):
- if deb.checkline(filename,bp):
+ for bp in range(bp + 1, bp + maxtries + 1):
+ if deb.checkline(filename, bp):
break
else:
msg = ("\nI failed to find a valid line to set "
@@ -1657,23 +1677,23 @@ def magic_run(self, parameter_s ='',runner=None,
error(msg)
return
# if we find a good linenumber, set the breakpoint
- deb.do_break('%s:%s' % (filename,bp))
+ deb.do_break('%s:%s' % (filename, bp))
# Start file run
print "NOTE: Enter 'c' at the",
print "%s prompt to start your script." % deb.prompt
try:
- deb.run('execfile("%s")' % filename,prog_ns)
+ deb.run('execfile("%s")' % filename, prog_ns)
except:
etype, value, tb = sys.exc_info()
# Skip three frames in the traceback: the %run one,
# one inside bdb.py, and the command-line typed by the
# user (run by exec in pdb itself).
- self.shell.InteractiveTB(etype,value,tb,tb_offset=3)
+ self.shell.InteractiveTB(etype, value, tb, tb_offset=3)
else:
if runner is None:
runner = self.shell.safe_execfile
- if opts.has_key('t'):
+ if 't' in opts:
# timed execution
try:
nruns = int(opts['N'][0])
@@ -1685,11 +1705,11 @@ def magic_run(self, parameter_s ='',runner=None,
twall0 = time.time()
if nruns == 1:
t0 = clock2()
- runner(filename,prog_ns,prog_ns,
+ runner(filename, prog_ns, prog_ns,
exit_ignore=exit_ignore)
t1 = clock2()
- t_usr = t1[0]-t0[0]
- t_sys = t1[1]-t0[1]
+ t_usr = t1[0] - t0[0]
+ t_sys = t1[1] - t0[1]
print "\nIPython CPU timings (estimated):"
print " User : %10.2f s." % t_usr
print " System : %10.2f s." % t_sys
@@ -1697,30 +1717,30 @@ def magic_run(self, parameter_s ='',runner=None,
runs = range(nruns)
t0 = clock2()
for nr in runs:
- runner(filename,prog_ns,prog_ns,
+ runner(filename, prog_ns, prog_ns,
exit_ignore=exit_ignore)
t1 = clock2()
- t_usr = t1[0]-t0[0]
- t_sys = t1[1]-t0[1]
+ t_usr = t1[0] - t0[0]
+ t_sys = t1[1] - t0[1]
print "\nIPython CPU timings (estimated):"
- print "Total runs performed:",nruns
- print " Times : %10.2f %10.2f" % ('Total','Per run')
- print " User : %10.2f s, %10.2f s." % (t_usr,t_usr/nruns)
- print " System : %10.2f s, %10.2f s." % (t_sys,t_sys/nruns)
+ print "Total runs performed:", nruns
+ print " Times : %10.2f %10.2f" % ('Total', 'Per run')
+ print " User : %10.2f s, %10.2f s." % (t_usr, t_usr / nruns)
+ print " System : %10.2f s, %10.2f s." % (t_sys, t_sys / nruns)
twall1 = time.time()
- print "Wall time: %10.2f s." % (twall1-twall0)
+ print "Wall time: %10.2f s." % (twall1 - twall0)
else:
# regular execution
- runner(filename,prog_ns,prog_ns,exit_ignore=exit_ignore)
+ runner(filename, prog_ns, prog_ns, exit_ignore=exit_ignore)
- if opts.has_key('i'):
+ if 'i' in opts:
self.shell.user_ns['__name__'] = __name__save
else:
# The shell MUST hold a reference to prog_ns so after %run
# exits, the python deletion mechanism doesn't zero it out
# (leaving dangling references).
- self.shell.cache_main_mod(prog_ns,filename)
+ self.shell.cache_main_mod(prog_ns, filename)
# update IPython interactive namespace
# Some forms of read errors on the file may mean the
125 IPython/utils/module_paths.py
View
@@ -0,0 +1,125 @@
+"""Utility functions for finding modules
+
+Utility functions for finding modules on sys.path.
+
+`find_mod` finds named module on sys.path.
+
+`get_init` helper function that finds __init__ file in a directory.
+
+`find_module` variant of imp.find_module in std_lib that only returns
+path to module and not an open file object as well.
+
+
+
+"""
+#-----------------------------------------------------------------------------
+# Copyright (c) 2011, the IPython Development Team.
+#
+# Distributed under the terms of the Modified BSD License.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+from __future__ import print_function
+
+# Stdlib imports
+import imp
+import os
+
+# Third-party imports
+
+# Our own imports
+
+
+#-----------------------------------------------------------------------------
+# Globals and constants
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Local utilities
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Classes and functions
+#-----------------------------------------------------------------------------
+def find_module(name, path=None):
+ """imp.find_module variant that only return path of module.
+
+ The `imp.find_module` returns a filehandle that we are not interested in.
+ Also we ignore any bytecode files that `imp.find_module` finds.
+
+ Parameters
+ ----------
+ name : str
+ name of module to locate
+ path : list of str
+ list of paths to search for `name`. If path=None then search sys.path
+
+ Returns
+ -------
+ filename : str
+ Return full path of module or None if module is missing or does not have
+ .py or .pyw extension
+ """
+ if name is None:
+ return None
+ try:
+ file, filename, _ = imp.find_module(name, path)
+ except ImportError:
+ return None
+ if file is None:
+ return filename
+ else:
+ file.close()
+ if os.path.splitext(filename)[1] in [".py", "pyc"]:
+ return filename
+ else:
+ return None
+
+def get_init(dirname):
+ """Get __init__ file path for module directory
+
+ Parameters
+ ----------
+ dirname : str
+ Find the __init__ file in directory `dirname`
+
+ Returns
+ -------
+ init_path : str
+ Path to __init__ file
+ """
+ fbase = os.path.join(dirname, "__init__")
+ for ext in [".py", ".pyw"]:
+ fname = fbase + ext
+ if os.path.isfile(fname):
+ return fname
+
+
+def find_mod(module_name):
+ """Find module `module_name` on sys.path
+
+ Return the path to module `module_name`. If `module_name` refers to
+ a module directory then return path to __init__ file. Return full
+ path of module or None if module is missing or does not have .py or .pyw
+ extension. We are not interested in running bytecode.
+
+ Parameters
+ ----------
+ module_name : str
+
+ Returns
+ -------
+ modulepath : str
+ Path to module `module_name`.
+ """
+ parts = module_name.split(".")
+ basepath = find_module(parts[0])
+ for submodname in parts[1:]:
+ basepath = find_module(submodname, [basepath])
+ if basepath and os.path.isdir(basepath):
+ basepath = get_init(basepath)
+ return basepath
133 IPython/utils/tests/test_module_paths.py
View
@@ -0,0 +1,133 @@
+# encoding: utf-8
+"""Tests for IPython.utils.path.py"""
+
+#-----------------------------------------------------------------------------
+# Copyright (C) 2008 The IPython Development Team
+#
+# Distributed under the terms of the BSD License. The full license is in
+# the file COPYING, distributed as part of this software.
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+from __future__ import with_statement
+
+import os
+import shutil
+import sys
+import tempfile
+import StringIO
+
+from os.path import join, abspath, split
+
+import nose.tools as nt
+
+from nose import with_setup
+
+import IPython
+from IPython.testing import decorators as dec
+from IPython.testing.decorators import skip_if_not_win32, skip_win32
+from IPython.testing.tools import make_tempfile
+from IPython.utils import path, io
+from IPython.utils import py3compat
+
+import IPython.utils.module_paths as mp
+
+env = os.environ
+TEST_FILE_PATH = split(abspath(__file__))[0]
+TMP_TEST_DIR = tempfile.mkdtemp()
+#
+# Setup/teardown functions/decorators
+#
+
+old_syspath = sys.path
+sys.path = [TMP_TEST_DIR]
+
+def make_empty_file(fname):
+ f = open(fname, 'w')
+ f.close()
+
+
+def setup():
+ """Setup testenvironment for the module:
+
+ """
+ # Do not mask exceptions here. In particular, catching WindowsError is a
+ # problem because that exception is only defined on Windows...
+ os.makedirs(join(TMP_TEST_DIR, "xmod"))
+ os.makedirs(join(TMP_TEST_DIR, "nomod"))
+ make_empty_file(join(TMP_TEST_DIR, "xmod/__init__.py"))
+ make_empty_file(join(TMP_TEST_DIR, "xmod/sub.py"))
+ make_empty_file(join(TMP_TEST_DIR, "pack.py"))
+ make_empty_file(join(TMP_TEST_DIR, "packpyc.pyc"))
+
+def teardown():
+ """Teardown testenvironment for the module:
+
+ - Remove tempdir
+ """
+ # Note: we remove the parent test dir, which is the root of all test
+ # subdirs we may have created. Use shutil instead of os.removedirs, so
+ # that non-empty directories are all recursively removed.
+ shutil.rmtree(TMP_TEST_DIR)
+
+
+def test_get_init_1():
+ """See if get_init can find __init__.py in this testdir"""
+ with make_tempfile(join(TMP_TEST_DIR, "__init__.py")):
+ assert mp.get_init(TMP_TEST_DIR)
+
+def test_get_init_2():
+ """See if get_init can find __init__.pyw in this testdir"""
+ with make_tempfile(join(TMP_TEST_DIR, "__init__.pyw")):
+ assert mp.get_init(TMP_TEST_DIR)
+
+def test_get_init_3():
+ """get_init can't find __init__.pyc in this testdir"""
+ with make_tempfile(join(TMP_TEST_DIR, "__init__.pyc")):
+ assert mp.get_init(TMP_TEST_DIR) is None
+
+def test_get_init_3():
+ """get_init can't find __init__ in empty testdir"""
+ assert mp.get_init(TMP_TEST_DIR) is None
+
+
+def test_find_mod_1():
+ modpath = join(TMP_TEST_DIR, "xmod", "__init__.py")
+ assert mp.find_mod("xmod") == modpath
+
+def test_find_mod_2():
+ modpath = join(TMP_TEST_DIR, "xmod", "__init__.py")
+ assert mp.find_mod("xmod") == modpath
+
+def test_find_mod_3():
+ modpath = join(TMP_TEST_DIR, "xmod", "sub.py")
+ assert mp.find_mod("xmod.sub") == modpath
+
+def test_find_mod_4():
+ modpath = join(TMP_TEST_DIR, "pack.py")
+ assert mp.find_mod("pack") == modpath
+
+def test_find_mod_5():
+ assert mp.find_mod("packpyc") is None
+
+def test_find_module_1():
+ modpath = join(TMP_TEST_DIR, "xmod")
+ assert mp.find_module("xmod") == modpath
+
+def test_find_module_2():
+ """Testing sys.path that is empty"""
+ assert mp.find_module("xmod", []) is None
+
+def test_find_module_3():
+ """Testing sys.path that is empty"""
+ assert mp.find_module(None, None) is None
+
+def test_find_module_4():
+ """Testing sys.path that is empty"""
+ assert mp.find_module(None) is None
+
+def test_find_module_5():
+ assert mp.find_module("xmod.nopack") is None
Something went wrong with that request. Please try again.