Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Magic arguments #199

Merged
merged 3 commits into from

3 participants

@rkern

Here are the magic arguments decorators. I haven't modified any of the existing magics.

@fperez
Owner

Great, many thanks! Only one minor note: don't use shlex.split, it's borked with unicode input. Instead, use this:

https://github.com/ipython/ipython/blob/master/IPython/utils/process.py#L106

It's kind of ugly that we actually have two shlex.split replacements, the second one is:

https://github.com/ipython/ipython/blob/master/IPython/core/completerlib.py#L58

That one does extra stuff to handle incomplete lines. Use whichever you prefer, but definitely don't rely on the plain one.

Other than that, this looks fantastic. Many thanks!

@rkern

Really, neither of those should be using sys.stdin.encoding. They can probably encode using utf-8 and then decoding the tokens on the way out. That shouldn't break up any of the multi-byte sequences. Most robustly, we could just include our own version that uses a unicode-aware StringIO.

@fperez
Owner

If you can add a more robust one in this same request, by all means go for it. I know this has caused real problems in the past, and we're likely to use it even more in magics, so the more robust it is the better.

@ellisonbg
Owner

Is this pull request ready to be merged?

@rkern

The I don't like either of the shlex.split replacements, but I guess I might as well use one and be done with it. I'll finish it up this weekend.

rkern added some commits
@rkern rkern BUG: when given unicode inputs, arg_split should return unicode outpu…
…ts. Always use utf-8 to encode the string instead of relying on sys.stdin.encoding, which may not be able to accept the full range of Unicode characters. When given unicode strings, arg_split is probably not receiving input from a terminal.
6aff23d
@rkern rkern BUG: Use arg_split instead of shlex.split e0e8fbb
@rkern

I updated arg_split to be more robust when it comes to unicode inputs. I updated magic-arguments to use it. It should be ready to merge now.

@fperez fperez commented on the diff
IPython/core/magic_arguments.py
((107 lines not shown))
+ help_text = parser.format_help()
+ if help_text.startswith('usage: '):
+ help_text = help_text.replace('usage: ', '%', 1)
+ else:
+ help_text = '%' + help_text
+
+ # Replace the magic function's docstring with the full help text.
+ magic_func.__doc__ = help_text
+
+ return parser
+
+
+def parse_argstring(magic_func, argstring):
+ """ Parse the string of arguments for the given magic function.
+ """
+ args = magic_func.parser.parse_argstring(argstring)
@fperez Owner
fperez added a note

We can just return the value straight away, no need to create that intermediate variable.

Don't worry though: the pull request is in almost perfect shape, so I'll do the cleanup as part of the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fperez fperez commented on the diff
IPython/core/tests/test_magic_arguments.py
((54 lines not shown))
+
+@magic_arguments()
+@argument('-f', '--foo', help="an argument")
+def magic_magic_foo(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(magic_magic_foo, args)
+
+@magic_arguments()
+@argument('-f', '--foo', help="an argument")
+def foo(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(foo, args)
+
+def test_magic_arguments():
@fperez Owner
fperez added a note

we'll convert these to use @parametric. For all its flaws, it's still better than the impossible to debug situation produced by nose's yield tests, which return an alternate stack (the one from nose) when they fail.

In the future, please use @parametric for these situations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fperez fperez merged commit e0e8fbb into ipython:master
@fperez
Owner

Thanks a bunch, Robert! Sorry for the delay in merging.

@damianavila damianavila referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 3, 2010
  1. @rkern
Commits on Jan 26, 2011
  1. @rkern

    BUG: when given unicode inputs, arg_split should return unicode outpu…

    rkern authored
    …ts. Always use utf-8 to encode the string instead of relying on sys.stdin.encoding, which may not be able to accept the full range of Unicode characters. When given unicode strings, arg_split is probably not receiving input from a terminal.
  2. @rkern
This page is out of date. Refresh to see the latest.
View
220 IPython/core/magic_arguments.py
@@ -0,0 +1,220 @@
+''' A decorator-based method of constructing IPython magics with `argparse`
+option handling.
+
+New magic functions can be defined like so::
+
+ from IPython.core.magic_arguments import (argument, magic_arguments,
+ parse_argstring)
+
+ @magic_arguments()
+ @argument('-o', '--option', help='An optional argument.')
+ @argument('arg', type=int, help='An integer positional argument.')
+ def magic_cool(self, arg):
+ """ A really cool magic command.
+
+ """
+ args = parse_argstring(magic_cool, arg)
+ ...
+
+The `@magic_arguments` decorator marks the function as having argparse arguments.
+The `@argument` decorator adds an argument using the same syntax as argparse's
+`add_argument()` method. More sophisticated uses may also require the
+`@argument_group` or `@kwds` decorator to customize the formatting and the
+parsing.
+
+Help text for the magic is automatically generated from the docstring and the
+arguments::
+
+ In[1]: %cool?
+ %cool [-o OPTION] arg
+
+ A really cool magic command.
+
+ positional arguments:
+ arg An integer positional argument.
+
+ optional arguments:
+ -o OPTION, --option OPTION
+ An optional argument.
+
+'''
+#-----------------------------------------------------------------------------
+# Copyright (c) 2010, IPython Development Team.
+#
+# Distributed under the terms of the Modified BSD License.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#-----------------------------------------------------------------------------
+
+# Our own imports
+from IPython.external import argparse
+from IPython.core.error import UsageError
+from IPython.utils.process import arg_split
+
+
+class MagicArgumentParser(argparse.ArgumentParser):
+ """ An ArgumentParser tweaked for use by IPython magics.
+ """
+ def __init__(self,
+ prog=None,
+ usage=None,
+ description=None,
+ epilog=None,
+ version=None,
+ parents=None,
+ formatter_class=argparse.HelpFormatter,
+ prefix_chars='-',
+ argument_default=None,
+ conflict_handler='error',
+ add_help=False):
+ if parents is None:
+ parents = []
+ super(MagicArgumentParser, self).__init__(prog=prog, usage=usage,
+ description=description, epilog=epilog, version=version,
+ parents=parents, formatter_class=formatter_class,
+ prefix_chars=prefix_chars, argument_default=argument_default,
+ conflict_handler=conflict_handler, add_help=add_help)
+
+ def error(self, message):
+ """ Raise a catchable error instead of exiting.
+ """
+ raise UsageError(message)
+
+ def parse_argstring(self, argstring):
+ """ Split a string into an argument list and parse that argument list.
+ """
+ argv = arg_split(argstring)
+ return self.parse_args(argv)
+
+
+def construct_parser(magic_func):
+ """ Construct an argument parser using the function decorations.
+ """
+ kwds = getattr(magic_func, 'argcmd_kwds', {})
+ if 'description' not in kwds:
+ kwds['description'] = getattr(magic_func, '__doc__', None)
+ arg_name = real_name(magic_func)
+ parser = MagicArgumentParser(arg_name, **kwds)
+ # Reverse the list of decorators in order to apply them in the
+ # order in which they appear in the source.
+ group = None
+ for deco in magic_func.decorators[::-1]:
+ result = deco.add_to_parser(parser, group)
+ if result is not None:
+ group = result
+
+ # Replace the starting 'usage: ' with IPython's %.
+ help_text = parser.format_help()
+ if help_text.startswith('usage: '):
+ help_text = help_text.replace('usage: ', '%', 1)
+ else:
+ help_text = '%' + help_text
+
+ # Replace the magic function's docstring with the full help text.
+ magic_func.__doc__ = help_text
+
+ return parser
+
+
+def parse_argstring(magic_func, argstring):
+ """ Parse the string of arguments for the given magic function.
+ """
+ args = magic_func.parser.parse_argstring(argstring)
@fperez Owner
fperez added a note

We can just return the value straight away, no need to create that intermediate variable.

Don't worry though: the pull request is in almost perfect shape, so I'll do the cleanup as part of the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ return args
+
+
+def real_name(magic_func):
+ """ Find the real name of the magic.
+ """
+ magic_name = magic_func.__name__
+ if magic_name.startswith('magic_'):
+ magic_name = magic_name[len('magic_'):]
+ arg_name = getattr(magic_func, 'argcmd_name', magic_name)
+ return arg_name
+
+
+class ArgDecorator(object):
+ """ Base class for decorators to add ArgumentParser information to a method.
+ """
+
+ def __call__(self, func):
+ if not getattr(func, 'has_arguments', False):
+ func.has_arguments = True
+ func.decorators = []
+ func.decorators.append(self)
+ return func
+
+ def add_to_parser(self, parser, group):
+ """ Add this object's information to the parser, if necessary.
+ """
+ pass
+
+
+class magic_arguments(ArgDecorator):
+ """ Mark the magic as having argparse arguments and possibly adjust the
+ name.
+ """
+
+ def __init__(self, name=None):
+ self.name = name
+
+ def __call__(self, func):
+ if not getattr(func, 'has_arguments', False):
+ func.has_arguments = True
+ func.decorators = []
+ if self.name is not None:
+ func.argcmd_name = self.name
+ # This should be the first decorator in the list of decorators, thus the
+ # last to execute. Build the parser.
+ func.parser = construct_parser(func)
+ return func
+
+
+class argument(ArgDecorator):
+ """ Store arguments and keywords to pass to add_argument().
+
+ Instances also serve to decorate command methods.
+ """
+ def __init__(self, *args, **kwds):
+ self.args = args
+ self.kwds = kwds
+
+ def add_to_parser(self, parser, group):
+ """ Add this object's information to the parser.
+ """
+ if group is not None:
+ parser = group
+ parser.add_argument(*self.args, **self.kwds)
+ return None
+
+
+class argument_group(ArgDecorator):
+ """ Store arguments and keywords to pass to add_argument_group().
+
+ Instances also serve to decorate command methods.
+ """
+ def __init__(self, *args, **kwds):
+ self.args = args
+ self.kwds = kwds
+
+ def add_to_parser(self, parser, group):
+ """ Add this object's information to the parser.
+ """
+ group = parser.add_argument_group(*self.args, **self.kwds)
+ return group
+
+
+class kwds(ArgDecorator):
+ """ Provide other keywords to the sub-parser constructor.
+ """
+ def __init__(self, **kwds):
+ self.kwds = kwds
+
+ def __call__(self, func):
+ func = super(kwds, self).__call__(func)
+ func.argcmd_kwds = self.kwds
+ return func
+
+
+__all__ = ['magic_arguments', 'argument', 'argument_group', 'kwds',
+ 'parse_argstring']
View
113 IPython/core/tests/test_magic_arguments.py
@@ -0,0 +1,113 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2010, IPython Development Team.
+#
+# Distributed under the terms of the Modified BSD License.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#-----------------------------------------------------------------------------
+
+from nose.tools import assert_equal, assert_true
+
+from IPython.external import argparse
+from IPython.core.magic_arguments import (argument, argument_group, kwds,
+ magic_arguments, parse_argstring, real_name)
+
+
+@magic_arguments()
+@argument('-f', '--foo', help="an argument")
+def magic_foo1(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(magic_foo1, args)
+
+@magic_arguments()
+def magic_foo2(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(magic_foo2, args)
+
+@magic_arguments()
+@argument('-f', '--foo', help="an argument")
+@argument_group('Group')
+@argument('-b', '--bar', help="a grouped argument")
+@argument_group('Second Group')
+@argument('-z', '--baz', help="another grouped argument")
+def magic_foo3(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(magic_foo3, args)
+
+@magic_arguments()
+@kwds(argument_default=argparse.SUPPRESS)
+@argument('-f', '--foo', help="an argument")
+def magic_foo4(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(magic_foo4, args)
+
+@magic_arguments('frobnicate')
+@argument('-f', '--foo', help="an argument")
+def magic_foo5(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(magic_foo5, args)
+
+@magic_arguments()
+@argument('-f', '--foo', help="an argument")
+def magic_magic_foo(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(magic_magic_foo, args)
+
+@magic_arguments()
+@argument('-f', '--foo', help="an argument")
+def foo(self, args):
+ """ A docstring.
+ """
+ return parse_argstring(foo, args)
+
+def test_magic_arguments():
@fperez Owner
fperez added a note

we'll convert these to use @parametric. For all its flaws, it's still better than the impossible to debug situation produced by nose's yield tests, which return an alternate stack (the one from nose) when they fail.

In the future, please use @parametric for these situations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ # Ideally, these would be doctests, but I could not get it to work.
+ yield assert_equal, magic_foo1.__doc__, '%foo1 [-f FOO]\n\nA docstring.\n\noptional arguments:\n -f FOO, --foo FOO an argument\n'
+ yield assert_equal, getattr(magic_foo1, 'argcmd_name', None), None
+ yield assert_equal, real_name(magic_foo1), 'foo1'
+ yield assert_equal, magic_foo1(None, ''), argparse.Namespace(foo=None)
+ yield assert_true, hasattr(magic_foo1, 'has_arguments')
+
+ yield assert_equal, magic_foo2.__doc__, '%foo2\n\nA docstring.\n'
+ yield assert_equal, getattr(magic_foo2, 'argcmd_name', None), None
+ yield assert_equal, real_name(magic_foo2), 'foo2'
+ yield assert_equal, magic_foo2(None, ''), argparse.Namespace()
+ yield assert_true, hasattr(magic_foo2, 'has_arguments')
+
+ yield assert_equal, magic_foo3.__doc__, '%foo3 [-f FOO] [-b BAR] [-z BAZ]\n\nA docstring.\n\noptional arguments:\n -f FOO, --foo FOO an argument\n\nGroup:\n -b BAR, --bar BAR a grouped argument\n\nSecond Group:\n -z BAZ, --baz BAZ another grouped argument\n'
+ yield assert_equal, getattr(magic_foo3, 'argcmd_name', None), None
+ yield assert_equal, real_name(magic_foo3), 'foo3'
+ yield assert_equal, magic_foo3(None, ''), argparse.Namespace(bar=None, baz=None, foo=None)
+ yield assert_true, hasattr(magic_foo3, 'has_arguments')
+
+ yield assert_equal, magic_foo4.__doc__, '%foo4 [-f FOO]\n\nA docstring.\n\noptional arguments:\n -f FOO, --foo FOO an argument\n'
+ yield assert_equal, getattr(magic_foo4, 'argcmd_name', None), None
+ yield assert_equal, real_name(magic_foo4), 'foo4'
+ yield assert_equal, magic_foo4(None, ''), argparse.Namespace()
+ yield assert_true, hasattr(magic_foo4, 'has_arguments')
+
+ yield assert_equal, magic_foo5.__doc__, '%frobnicate [-f FOO]\n\nA docstring.\n\noptional arguments:\n -f FOO, --foo FOO an argument\n'
+ yield assert_equal, getattr(magic_foo5, 'argcmd_name', None), 'frobnicate'
+ yield assert_equal, real_name(magic_foo5), 'frobnicate'
+ yield assert_equal, magic_foo5(None, ''), argparse.Namespace(foo=None)
+ yield assert_true, hasattr(magic_foo5, 'has_arguments')
+
+ yield assert_equal, magic_magic_foo.__doc__, '%magic_foo [-f FOO]\n\nA docstring.\n\noptional arguments:\n -f FOO, --foo FOO an argument\n'
+ yield assert_equal, getattr(magic_magic_foo, 'argcmd_name', None), None
+ yield assert_equal, real_name(magic_magic_foo), 'magic_foo'
+ yield assert_equal, magic_magic_foo(None, ''), argparse.Namespace(foo=None)
+ yield assert_true, hasattr(magic_magic_foo, 'has_arguments')
+
+ yield assert_equal, foo.__doc__, '%foo [-f FOO]\n\nA docstring.\n\noptional arguments:\n -f FOO, --foo FOO an argument\n'
+ yield assert_equal, getattr(foo, 'argcmd_name', None), None
+ yield assert_equal, real_name(foo), 'foo'
+ yield assert_equal, foo(None, ''), argparse.Namespace(foo=None)
+ yield assert_true, hasattr(foo, 'has_arguments')
+
+
View
10 IPython/utils/process.py
@@ -114,11 +114,17 @@ def arg_split(s, posix=False):
# http://bugs.python.org/issue1170
# At least encoding the input when it's unicode seems to help, but there
# may be more problems lurking. Apparently this is fixed in python3.
+ is_unicode = False
if isinstance(s, unicode):
- s = s.encode(sys.stdin.encoding)
+ is_unicode = True
+ s = s.encode('utf-8')
lex = shlex.shlex(s, posix=posix)
lex.whitespace_split = True
- return list(lex)
+ tokens = list(lex)
+ if is_unicode:
+ # Convert the tokens back to unicode.
+ tokens = [x.decode('utf-8') for x in tokens]
+ return tokens
def abbrev_cwd():
View
3  IPython/utils/tests/test_process.py
@@ -66,6 +66,9 @@ def test_arg_split():
"""Ensure that argument lines are correctly split like in a shell."""
tests = [['hi', ['hi']],
[u'hi', [u'hi']],
+ ['hello there', ['hello', 'there']],
+ [u'h\N{LATIN SMALL LETTER A WITH CARON}llo', [u'h\N{LATIN SMALL LETTER A WITH CARON}llo']],
+ ['something "with quotes"', ['something', '"with quotes"']],
]
for argstr, argv in tests:
nt.assert_equal(arg_split(argstr), argv)
Something went wrong with that request. Please try again.