Subcommands and subparsers

[averrin]

like http://docs.python.org/library/argparse.html#sub-commands

[keleshev]

I plan support for sub-commands, but I'm still not sure if it should be implemented via sub-parsers (each having a docstring with its options), or via some kinda pattern-matching in usage message:

Usage: program.py add [options]
                  rm [options]

[yaph]

An implementation via sub-parsers seems more natural to me, as it provides better separation of sub-commands.

[keleshev]

I agree that if every command has huge amount of options, then using sub-parsers (and thus splitting usage-massage) is better. But for simple programs sub-parsers are often an overkill, pattern-matching should do the job.

So, I plan to implement both. I started first with pattern-matching (in a branch), because I see a big future in it, in general (stuff like specifying argument patterns, like N [N ...], "required options", mutually-exclusive options (-a | -b), cool stuff!).

[brentp]

why can't it just use a decorator and the function's docstring?
then the user can append common docs to the function's docstring -- or use another decorator.

@command
def afunc(args):
     ....

will show

Usage: program.py afunc [options]
       ...

[keleshev]

@brentp I wanted to make API as simple as possible, preferably keep it as 1 function. What I was recently thinking: what if docopt took a list of docstrings and returned a tuple commands, options, arguments, and you could do:

commands, options, arguments = docopt(...)
if commands.add:
    prog_add(options, arguments)
elif commands.rm:
    prog_rm(options, arguments)

The problem with decorators:

• what will be the entry point that will run one of the decorated functions?
• how to handle valid case when there are no commands?

I want to come up with best API possible, but can't see any obviously-better-than-others API candidate.

Anyway, more discussion on this issue is welcome. v0.2 that will be released soon will not have subcommands, but the only reason—can't decide on API yet.

[keleshev]

Now that I wrote the example above I can see it's obviously bad :-)

[brentp]

It's possible to do this with docopt as-is. Here's a hacked together example:
https://gist.github.com/2788612

It would be nice if there were something that made this simpler and with better
error handling.

[keleshev]

Subcommands are now implemented in 0.3.0!

[brentp]

As my 0.02, the implementation of subcommands is very nice, but to actually use them to call sub-functions is not simple with 0.3.0. One has to take the dictionary returned from docopt, check the subcommands, then extract the arguments that are actually needed for a particular function.
If I'm missing something, please let me know.

[keleshev]

I agree totally. Now you need to:

args = docopt(__doc__)
if args['ship'] and args['new']:
    ship_new(args['<name>'])
elif args['ship'] and args['move']:
    ship_move(args['<name>'], args['--speed'])
elif ...

And that is not DRY. Any API ideas? @brentp could you elaborate on you decorator example?

Another question: docopt now returns dict—what do you think of that change?

[brentp]

It was nice to get back opts, args, but I can see cases where it's simpler to just have the dict.

I think this: https://gist.github.com/2788612 is a decent start, but looks ugly without some infrastructure.
I will elaborate by tomorrow.

I think it would be nice to be able to do:

ship_new(**args)

or

ship_move(**args)

after the dispatch to the correct function.

[brentp]

I have updated the gist:
https://gist.github.com/2788612

It includes the command() class that I think demonstrates a simple way to register sub-commands.
Given the new changes in 0.3, it's still some work to dispatch to the functions, now we have to strip the "--" ourselves?
I left that un-addressed at line 24-25, but, it gives and Idea what I had in mind. I'm sure it can be improved.

[keleshev]

what about just passing args to each function?

[brentp]

The downside to that is the function is then only useful when used with docopt.
That's why it was nice to get options, arguments from the docopt() call, then the function
could be called via:

func(*arguments, **options)

[keleshev]

But *arguments should also have names, how would you distinguish them?

I decided to use 1 dict, because otherwise I would need to return tuple (arguments, options, commands) since the commands were introduced. That would be make it more complicated for those who don't use commands. It looked as dict would scale better, also if there is need to add more stuff to it.

[brentp]

arguments are ordered. If it's a class, just have iter return them in the order they are specified in the docstring.

[keleshev]

As I wrote in #22:

I no longer think that the proposed above APIs are good because of this flaw:

• Real, big, programs (like git) consist (behind the scene) of many sub-programs, which are often written in different languages.

The above APIs are python-only. Fortunately there are a bunch of docopt implementations in different languages—the ideal API would allow a central executable to make dispatch to different other executables. Maybe even to executables that don't make use of docopt.

So I propose for your feedback the following solution. It is not fully supported in docopt to be useful (options do not work correctly), but it's a good proof of concept:

"""Example dispatching multiple scripts with several help-screens.

Usage:
    multi <command> [options] [<arguments>...]
    multi (-h | --help)

Commands:
    add
    remove

See multi help <command> for help on each command.

"""
from docopt import docopt


add_help = """Add command.

Usage: multi add <this> [<and-that>]

"""


remove_help = """Remove command.

Usage: multi_example.py remove <that> <from-this>

"""


args = docopt(__doc__)
if args['<command>'] == 'add':
    args = docopt(add_help)
    print(args)
elif args['<command>'] == 'remove':
    args = docopt(remove_help)
    print(args)

$ python multi_example.py -h
Example dispatching multiple scripts with several help-screens.

Usage:
    multi <command> [options] [<arguments>...]
    multi (-h | --help)

Commands:
    add
    remove

See multi help <command> for help on each command.

$ python multi_example.py add
Usage: multi add <this> [<and-that>]

$ python multi_example.py add -h
<this does not work correctly yet>

$ python multi_example.py add this
{'<and-that>': None,
 '<this>': 'this',
 'add': True}

$ python multi_example.py add this that
{'<and-that>': 'that',
 '<this>': 'this',
 'add': True}

$ python multi_example.py remove
Usage: multi_example.py remove <that> <from-this>

$ python multi_example.py remove this that
{'<from-this>': 'that',
 '<that>': 'this',
 'remove': True}

Isn't it sweet? docopt shows correct usage for subcommands (but not correct help yet), and subcommands are parsed correctly. (Although options don't work, since any option will be an error in during first docopt invocation, but meaning of [options] can be changed for that).

Also, note, that different help-screens could be in different files, in different languages, even supporting dispatching into non-docopt option-parsers. This is possible if you dispatch like that:

args = docopt(__doc__)
if args['<command>'] == 'add':
    # how to safely pass argv probably needs more work
    os.system('perl multi_add.pl ' + ' '.join('%r' % s for s in sys.argv[1:])) 
elif args['<command>'] == 'remove':
    os.system('ruby multi_remove.rb ' + ' '.join('%r' % s for s in sys.argv[1:])) 

What do you think?

Edit: This is a more proper example than the one above

args = docopt(__doc__)
if args['<command>'] == 'add':
    exit(subprocess.call(['perl', 'multi_add.pl'] + sys.argv[2:]))
elif args['<command>'] == 'remove':
    exit(subprocess.call(['ruby', 'multi_remove.rb'] + sys.argv[2:]))

[keleshev]

Just wanted to paste it here that it looks like parse_known_args was tackling the same problem of trying to delegate some options/arguments to another script:

http://docs.python.org/dev/library/argparse.html#argparse.ArgumentParser.parse_known_args

[keleshev]

The above idea is now implemented in any-options branch, which allowed to implement git CLI subset (including commands: add, branch, commit, push, remote) with dispatch script that looks like this:

#! /usr/bin/env python
"""
usage: git [--version] [--exec-path=<path>] [--html-path]
           [-p|--paginate|--no-pager] [--no-replace-objects]
           [--bare] [--git-dir=<path>] [--work-tree=<path>]
           [-c name=value]
           <command> [options] [<args>...]
       git [--help]

The most commonly used git commands are:
   add        Add file contents to the index
   branch     List, create, or delete branches
   commit     Record changes to the repository
   push       Update remote refs along with associated objects
   remote     Manage set of tracked repositories

See 'git help <command>' for more information on a specific command.

"""
import sys
from subprocess import call

from docopt import docopt


if __name__ == '__main__':

    args = docopt(__doc__,
                  version='git version 1.7.4.4',
                  help=False,
                  _any_options=True)

    # Handle -h|--help manually.
    # Otherwise `subcommand -h` would trigger global help.
    if args['<command>'] is None:
        print(__doc__.strip())
        exit()

    # Make argv for subparsers that does not include global options:
    i = sys.argv.index(args['<command>'])
    sub_argv = sys.argv[i:]

    if args['<command>'] == 'add':
        # In case subcommand is implemented as python module:
        import git_add
        print(docopt(git_add.__doc__, argv=sub_argv))
    elif args['<command>'] == 'branch':
        # In case subcommand is a script in some other programming language:
        exit(call(['python', 'git_branch.py'] + sub_argv))
    elif args['<command>'] == 'commit':
        exit(call(['python', 'git_commit.py'] + sub_argv))
    elif args['<command>'] == 'push':
        exit(call(['python', 'git_push.py'] + sub_argv))
    elif args['<command>'] == 'remote':
        exit(call(['python', 'git_remote.py'] + sub_argv))
    elif args['<command>'] == 'help':
        exit(call(['python', 'git.py'] + args['<args>'] + ['--help']))
    else:
        exit("%r is not a git.py command. See 'git help'." % args['<command>'])

I would love to hear some feedback. Should this be in 0.6.0?

[Julian]

I don't like having to write your own dispatch. docopt has enough information, or should be provided enough information, to do the dispatch to subparsers / commands itself.

I'd like to write something like

commands = dict(
    branch=branch.__doc__,
    commit=commit.__doc__,
    whatever=whatever.__doc__,
)

arguments = docopt.dispatch(__doc__, version="Hey hey hey v1.0", subcommands=commands)

where all the dispatch is now done for me, as is handling things like printing the global usage message or command usage message as appropriate, or erroring out for an unrecognized command. Supporting creating a help subcommand within the usage note automatically, like --help and --version would also be nice.

Of course an approach like this means that you can only have a single level of subparser, but that seems quite reasonable for sanity's sake :).

[keleshev]

@Julian your proposed API is DRYer than mine, but it would work for python code only. I was looking into making a universal solution for programs built with many languages (like git: C, bash, Perl, ...).

I never had a need to build such a complex CLI that would require subparsers—isn't it something that is done once in a lifetime? Are there any other examples apart from SCM like git/hg/svn which have need for that?

[Julian]

Hm. What would be Python specific about it? It should work anywhere you have associative arrays, and on that list your implementation of docopt for C has something that'd work too.

As for how often, I use it really often. Pattern matching is nice when you have a bunch of commands with the same potential options, otherwise it's really cluttered and bad, you get everything mixed together with things that aren't related. easy_install, pip, and setup.py are more examples of things with subparsers, as long as we're talking about Python. I'm sure I can think of some more if need be :).

[keleshev]

What would be Python specific about it?

I mean that with your approach you can't have a Python program that would call programs in different languages. Please give an example, if I'm wrong with this one.

You are right, though, about pip, setup.py (also hg) are pure-Python programs with sub-parsers. But note that they are one-of-a-kind programs. You don't do this kind of things every day. Prove me wrong :-)

That is why I don't want to clutter docopt API with a shortcut for something that is done so rarely, and could be done quite easily with docopt if needed (with introduction of any_options parameter).

The approach I presented above could be shortened in pure-Python case from this:

if args['<command>'] == 'add':
        import git_add
        exit(git_add.main(argv=sub_argv))
elif args['<command>'] == 'branch':
        import git_branch
        exit(git_branch.main(argv=sub_argv))
elif args['<command>'] == 'commit':
        import git_commit
        exit(git_branch.main(argv=sub_argv))
else: ...

To something like this:

if args['<command>'] in ('add', 'branch', 'commit'):
    m = __import__('git_' + args['<command>'])
    exit(m.main(argv=sub_argv))
else: ...

[keleshev]

I also agree that making global help manually is not perfect, but I think I can make global help work with this approach automatically. The problem is that if global parser has usage:

Usage: prog <command> [options] [<arguments>...]
       prog (-h | --help)

Then because of <command> [options] part, where [options] of course allows -h and --help, parser will trigger it's help. But if the meaning of [options] can be changed to "any options, except ones in the pattern", then global help might work automatically.

[Julian]

Looking back at my original example, I see that I forgot to write the final line as command, arguments =, since the proposed API would need to return the command that was selected (which was the reason I had a different function in the first place).

I think calling out to other processes is a different issue, and one even rarer than subparsers, which you maintain are quite rare :). If I needed to do that, then I'd probably either do what you're doing, or recreate argv myself from the arguments if I was in control of both programs. I don't think docopt can do much to help here, since docopt's whole function is to parse a usage string and return back a native mapping -- calling out to other programs with the argument list seems contrary to that goal, and so would need to be handled differently. But the most important thing to me is that the simple case should be simple.

Essentially that's my only objection I guess, otherwise as I have it would be exactly as your git example, other than manually needing to print the global usage, and the need to mess with argv. (Which would mean that at least in Python I could further trim down your example by using a dispatch dict rather than an if.. elif.. elif.. elif.. chain.

Oh and, if you'd like another example, my current usage for docopt is a console RSS reader, which has as subcommands foo get to fetch an rss feed from the internet, foo list to list the currently watched feeds, foo add, foo rm to add and delete... and each have their own command line options, which are not numerous, but for cleanliness deserve to be kept separate.

[keleshev]

Is you RSS reader open-source? If yes—where can I take a peek?

I will think more about APIs. I think docopt(__doc__, any_options=True) is a move in right direction—it makes hard case possible. Now you can do anything with docopt. This also allows arbitrary options at runtime:

>>> docopt('usage: prog [options]', '--hai --bye=there -sqwr', _any_options=True)
{'--bye': 'there',
 '--hai': True,
 '-q': True,
 '-r': True,
 '-s': True,
 '-w': True}

Thought corner-cases (like repeating options w and w/o argument) are still in development.

It is painful for me to add even 1 new parameter (any_options), not speaking of introducing decorators or classes. You should totally experiment if you want—like making a wrapper around docopt with API of your liking—I will incorporate it if it can make hard case—simpler, while keeping the common-case as simple as it is now (i.e. not adding extensive API).

[Julian]

It's nowhere near complete, but the foundation is at https://github.com/NYCPython/RSS/.

As for any_options, I don't have any idea what the argument means from just looking at it, so I think that's a minus. But I guess it sounds like you have different goals in mind than I do. I want good option parsing in languages I use in a way that's native to each, so to me that means using the most idiomatic things in each language's implementation. So, a non-uniform interface to a uniform data (usage string) format.

It'd certainly not be hard to build what I want on top of docopt essentially that's what I've set the foundation for in that repository. I appreciate the project, certainly, and will be keeping an eye on this issue anyhow :).

[keleshev]

If no other proposals or objections, the future of docopt sub-parsing will be around any_options parameter (described above).

any-options branch implements it:

https://github.com/docopt/docopt/tree/any-options

From future README:

Sub-parsers, multi-level help and huge applications (like git)

If you want to split your usage-patter in several, implement multi-level help (whith separate help-screen for each subcommand), want to interface with existing scripts that don't use docopt, or you're building the next "git", you will need the new any_options parameter (described in API section above). To get you started quickly we implemented a subset of git command-line interface as an example:

Feedback is welcome. I would be very happy to know if there is a better API idea.

[shabbyrobe]

I have no proposal or objection. A big +1 from me.

[kennethreitz]

I'm really looking forward to this.

[duncanphillips]

any_options is going to be a big win

[keleshev]

This is closed in favor of #64

[jck]

For people who are interested in using commands and subcommands, but not something full blown like git; I've implemented a simple decorator based approach here: https://github.com/jck/docopt-cmd

Example:

"""Do somethings

Usage:
    example.py [-vfr] do (a|b)

Options:
    -v --verbose    Be verbose.
    -f --force      Force.
    -r --random     Huh?.
"""
from docopt import docopt
from docopt_cmd import cmd

def main():
    args = docopt(__doc__)
    #print(args)

    cmd.dispatch(args)

#explicitly specify spec and options to pass as arguments
@cmd('do a', force='--force')
def something(force):
    print('doing a')
    print(force)

#or use magic
@cmd
def do_b(random):
    print 'doing b'
    print random

if __name__ == '__main__':
    main()