Tab completion flags #104

Closed
bitprophet opened this Issue Dec 22, 2013 · 23 comments

Projects

None yet

4 participants

@bitprophet
Member

See e.g. https://github.com/hhatto/zsh_completions/blob/master/_invoke for a zsh completion somebody wrote for Invoke as-is. My ideas for this (on top of whatever is in Fabric #6) is to use Invoke's existing parser to print easily computer readable info, e.g.:

[12:56:44] instead, output the flags/opts in a manner that is easier to parse, ditto for subcommand list
[12:56:51] and, I assume zsh supports this, the flags for each subcommand as well
[12:57:28] So one can: inv <tab, see flags/commands> mycom<tab, completes 'mycommand'> <tab, see 'mycommand' specific flags>
[12:58:17] So as long as I can make it easy to do stuff like 'inv --opts' or 'inv --opts ' or a more computer-formatted 'inv --list'...anybody who knows their shell's completion setup can adapt that info

@jthigpen
jthigpen commented Feb 6, 2014

So in my use case, I don't need command option completion (--blah), I just wanted to be able to tab complete the task names. I came up with this: https://gist.github.com/8853914. This works on my mac and an ubuntu 12.04 box.

@IvanMalison

Is this issue only for the tab completion of flag (i.e. -- / part of invoke arguments) or is it also referring to the completion of commands? I just wrote a zsh completion for tasks at
https://github.com/IvanMalison/dotfiles/blob/master/dotfiles/lib/completions/_invoke

Its similar to @jthigpen's solution but it doesn't get tripped up by aliases.

I'd be happy to integrate the work done by @hhatto to provide a complete zsh completion library.

@IvanMalison

https://github.com/kislyuk/argcomplete might be worth looking at.

@bitprophet
Member

@IvanMalison It's for both, the general point is "tab completion setups need to get info out of the tool as to what valid completion values are" (true for any completion at all - core flags, task names, task flags, etc) and so this is the ticket for "have core flags to easily spit out this info so completion scripts can just ask invoke directly instead of hardcoding".

Argcomplete (at a glance) seems argparse specific, so I dunno if it'd be useful for us? I'm expecting that our implementation would be pretty small and just tie into the parser classes.

@IvanMalison

yeah the version at https://github.com/IvanMalison/dotfiles/blob/master/dotfiles/lib/completions/_invoke is now working for both subcommands and core flags, but it has to do a lot of nasty things to parse the output of invoke -l.

Task flags would require some way to access that information (which is not currently available as far as i know). Is there anything blocking the merge of #163 ? I'd like to make accessing the available subcommands (and their arguments) easier.

@bitprophet
Member

Nothing blocking it besides time/priorities :( since someone (you) is currently active & willing to poke at the shell side of things, I'll see if I can put some time aside this week to review #163 and do whatever else is necessary to get this stuff exported in a friendly manner. Thanks!

@bitprophet
Member

Looking over #163 raises the following more specific questions we need to answer for this to be "done well":

  • What formats work best for machine-readable output? Space separated, line separated, other?
    • Guessing line-separated as this tends to work best with most arbitrary shell tools.
  • Do the most common shells' completion mechanisms work best with individual 'types' of output (eg one flag for each of tasks, core flags, per-task flags, etc) or would a single "spit everything out" flag be better?
    • From what I recall, absolutely the former - we'd probably need one for "list all tasks", one for "list all core options", and then could build it out with contextualized completion lists for individual tasks, or even for namespaces (think inv <tab>do<tab>bu<tab> to complete inv docs.build)
  • What actual names to use for these?
    • #163 calls the completion oriented task list --shortlist but this doesn't work as well for flags (--shortflags is confusing and has its own meaning anyways (-b vs --build).
    • --complete-<x>? eg --complete-tasks, --complete-flags, etc? maybe s/complete/completion?
    • Gets more complex if we add contextualization/args, e.g. inv --flags-for mytask?
    • So maybe --tasks, --flags, --tasks-for <namespace> and --flags-for <task>?
@IvanMalison

Looking over #163 raises the following more specific questions we need to answer for this to be "done well":

What formats work best for machine-readable output? Space separated, line separated, other?
Guessing line-separated as this tends to work best with most arbitrary shell tools.

Agreed, although space separated is not THAT hard to work with.

Do the most common shells' completion mechanisms work best with individual 'types' of output (eg one flag for each of tasks, core flags, per-task flags, etc) or would a single "spit everything out" flag be better?
From what I recall, absolutely the former - we'd probably need one for "list all tasks", one for "list all core options", and then could build it out with contextualized completion lists for individual tasks, or even for namespaces (think inv dobu to complete inv docs.build)
When I first read this, I was inclined to agree, but now that I've thought about it a little bit, I think it might be better to have a single command because there is fixed startup cost involved in starting a python process and then having invoke run tasks.py that could be large enough that it would make invoke completion feel laggy. In my experience, it is bad enough that it makes sense to use zsh completion caching to avoid calling it over and over again every time you are attempting completion.

What actual names to use for these?
#163 calls the completion oriented task list --shortlist but this doesn't work as well for flags (--shortflags is confusing and has its own meaning anyways (-b vs --build).
--complete-? eg --complete-tasks, --complete-flags, etc? maybe s/complete/completion?
Gets more complex if we add contextualization/args, e.g. inv --flags-for mytask?
So maybe --tasks, --flags, --tasks-for and --flags-for ?

perhaps:
--completion-info
--machine-readable

@bitprophet
Member

A wrinkle in task listing: some users might expect "full" listings, e.g. a deep namespace tab-completing entire task paths, e.g. a "task list" might be deploy, docs.build, docs.clean. This is what I and #163 were assuming.

But others might prefer a more "build it up" approach where the first tab is the first level of the namespace (deploy, docs) (perhaps with trailing dots denoting non-leaf paths) and subsequent tabs added depth. So e.g. inv <tab>do<tab>b<tab> would first display deploy and docs., then docs.build and docs.clean, and finally docs.build would be the last item to complete (with a space after it).

I'd expect the split between these two options to depend on the size of one's task namespace, with small ones preferring the former and anything bigger, the latter. This implies that if we pick just one, we go the latter, more 'intelligent' route.

Having both might be overdoing it, but given we need these handful of flags either way, probably not a huge deal.

@IvanMalison

A wrinkle in task listing: some users might expect "full" listings, e.g. a deep namespace tab-completing entire task paths, e.g. a "task list" might be deploy, docs.build, docs.clean. This is what I and #163 were assuming.

But others might prefer a more "build it up" approach where the first tab is the first level of the namespace (deploy, docs) (perhaps with trailing dots denoting non-leaf paths) and subsequent tabs added depth. So e.g. inv dob would first display deploy and docs., then docs.build and docs.clean, and finally docs.build would be the last item to complete (with a space after it).

I'd expect the split between these two options to depend on the size of one's task namespace, with small ones preferring the former and anything bigger, the latter. This implies that if we pick just one, we go the latter, more 'intelligent' route.

Having both might be overdoing it, but given we need these handful of flags either way, probably not a huge deal.

Well, zsh, for example, sort of handles this automatically in that it will complete up to the first dot if there is an ambiguity, and then cycle through the available choices that matched the original input if tab is pressed again.

@bitprophet
Member

Finally dug into this enough to realize the approach outlined above (flags for printing tasks and/or options) doesn't really work - any completion requires use of the Invoke cli parser to figure out what the current task context is. Having --tasks and --flags and --flags-for only makes sense if the completion script using them knows what that context is, and it can't without giving Invoke the full command string being completed.

For example, yes, a naive completion of inv <tab> could use --tasks, but inv task1 --arg1 task2 --arg2 <tab> needs to know that it's "within" task2 and that valid completions are a combination of task2 flags and other task names. The only way to know whether the current context is task1 (which it would be if --arg1 took a value) or task2 (if --arg1 was a boolean) is to run the parser.

On one hand this means fewer new flags, on the other it means I need to make sure we can execute the parser in a mode that works well for this. Suspect it won't be that hard, since we already have options for running the parser in a "ignore everything you can't understand" mode - just need a way for it to 'export' its current state.

@bitprophet
Member

Of course, this also means we need a way of getting a possibly-complex Invoke command line into an Invoke parameter argument itself, but that should be doable either by using STDIN or something like invoke --complete -- <command line here>.

@bitprophet
Member

Also need to clarify behavior re: 'bare' completion - should inv <tab> complete task names only, or task names and core options?

One of the tab completion scripts submitted for Fabric only completes core options when the partial word starts with -, and this also appears to be how e.g. my Git completion module (for zsh) behaves. Feels like a reasonable compromise.

@bitprophet bitprophet added a commit that referenced this issue Mar 12, 2015
@bitprophet bitprophet Changelog re #104 0354ca8
@bitprophet
Member

Have the basics working (took some detours through general testing related junk, sigh) as well as a bash script using it. Turns out, bash's compgen works just fine for dotted task names due to how substrings work (in retrospect, "duh") so we don't need to really worry about how that behaves.

@bitprophet
Member

Another wrinkle occurring to me as I build this out, folks loading nondefault collections with -c will have an issue, because the tab complete script/function can't easily know this, when it is asking Invoke for valid completions.

I guess this would require more logic in the completion scripts themselves, e.g. "if you see -c xxx in $COMP_WORDS, add it to your --complete call". Probably not super awful?

@bitprophet
Member

Another wrinkle (boy it's funny how computers have so many wrinkles) is that when completing technically-invalid command lines, sometimes you actually want it to "work".

For example, completing inv task_requiring_a_posarg <tab>: trying to parse everything up to the tab normally fails because task_requiring_a_posarg won't parse happily unless it has some value after it. At the very least, we shouldn't have completion explode messily, it should return no completion options (such that e.g. a shell's default filename completion kicks in).

Another option is to allow completing flags anyways, because task_requiring_a_posarg might have other non-positional arguments (or the user may be intending to give the positional argument explicitly via a flag instead).

EDIT: rad, the idea I had (include failed context in ParseError) is already a thing. Go, past me!

@bitprophet
Member

It's working for bash now, including most of the edge cases I could come up with trying to use it on invoke's own tasks.py.

Found one more edge case/bug (task names aren't including aliases), and also realizing that in practice it is almost annoyingly slow, so having the scripts themselves cache might not be a bad idea. Could just use the modification timestamp on the tasks.py for that?

Also need to mention it in the docs outside of just the changelog.

In checklist form,

  • Print aliases in task name lists
  • Try caching real quick
  • Mention in docs
  • zsh
@bitprophet
Member

Right, forgot caching was skipped at first because it's poorly defined here. The naive scripts that think of completion as a single set of task names and/or flags, can afford to just cache a single shell array in a file. In our case, the completion may differ wildly depending on the command line being completed, which is why we went the way we did.

I did say the completion as-is is almost annoyingly slow; it's not super duper bad, just not as lightning fast as things that do cache well.

We could potentially do something crazier like move all the logic into a shell script (hrrgh) and have --complete just shit out the raw data (and then have the script cache that) but I don't know if that's worth it, especially since it would duplicate (triplicate, even) the number of codebases that care about parser innards.

@IvanMalison

Did you look at what I did with caching in zsh in https://github.com/hhatto/zsh_completions/blob/master/_invoke? It's not perfect, but It makes completion really snappy.

@bitprophet
Member

@IvanMalison Maybe I'm not reading it hard enough, but I did look at it and while it's more advanced than the bash stuff, it still only looks like it completes a) task names and b) core options - and the stuff I just merged to master (!) also does the much harder c) per-task options.

I reckon we could just cache those first two categories and leave the rest uncached, but for now at least I've spent enough time on it I'd rather wait to see if anybody complains about speed. It's really not that slow as-is on my Macbook Air, at least.

@bitprophet
Member

Closing this as 'done' (but that is not intended to make @IvanMalison stop talking :D :D please do reply!)

@bitprophet bitprophet closed this Mar 14, 2015
@nehalecky

I noticed that the checkbox above for mention in docs was never checked, and my searches for tab autocomplete pyinvoke only lead to this thread along with some references to the code base of the pyinvoke repo.

I'd like to enable autocomplete for use in bash / zsh—any pointers greatly appreciated. :)

@nehalecky

And of course, just after commenting, I stumble across this:
https://github.com/pyinvoke/invoke/blob/master/sites/docs/cli.rst#shell-tab-completion

I will try out this setup, but also, I'd recommend that this be added to docs at http://www.pyinvoke.org!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment