Dynamic completion support

[Manishearth]

Maintainers notes:

• Assuming this is custom completion on top of clap-driven completion, this is blocked on Stablize Rust-Native Completion Engine Tracking Issue #3166
• Open question: does the completion function accept just the arg and current value or does it have access to ArgMatches for completing based on other data like in Compute argument possible values based on other arguments #1910

Like with PossibleValue::help, we should support help text

---

Currently clap does basic completion of arguments, however it can't be used to do things like the dynamic completion of git checkout <branch>.

It would be really neat if this could be handled by clap itself -- AFAICT nothing like this currently exists for any language and it would just be magical since you can then do everything in Rust code.

The rough idea is that the Arg/Subcommand builder can take a .with_completion(|args, str| ... ) closure. We pass the
closure the partial parsed list of other args, and the partial string being completed for this arg (may be empty). The closure returns a list of completions.

When you attempt to do something like git branch -D foo<tab>, the completion script will call git --secret-completion-param 10 -- branch -D foo, where 10 is the index in the arguments of where <tab> was pressed, and after the -- we pass all the arguments. Clap parses these arguments except the partial one, and passes this down to the with_completion closure, which will return a list of completions which we print out (and feed back to bash or whatever).

A simpler version doesn't handle parsing all the other arguments, instead your closure just takes the partial string, so you just provide a closure operating on a string.

If this could be made to work neatly, it would be pretty great.

cc @killercup

[kbknapp]

Sorry for the late reply, it's been a busy week!

This is related to #568 and something I very much want. The implementation you've outlined is pretty close to what we're thinking in #568 and probably almost exactly what will end up being implemented. I haven't worked out the final details yet as I've had other issues as as priority and the v3 release is a prerequisite for me to implement this (although I want this, or an initial implementation in the v3-alpha1 release).

[sportfloh]

hey, I was recently looking for such a feature and stumbled over clangs autocompletion feature (http://blog.llvm.org/2017/09/clang-bash-better-auto-completion-is.html). Maybe their solution is somehow inspiring.
Can I help with this issue?

[kinnison]

To reiterate a call for this, we'd love to have this in rustup so we could offer more localised completions (e.g. for +foo to complete over installed toolchains).

[woodruffw]

I'd also like to register interest in this. It's not as conceptually clean, but an approach that injects the results of a subshell would be sufficient for many CLIs.

Borrowing the with_completion terminology above:

subcommand(
  App::new("dump")
  .arg(
    Arg::with_name("thing")
        .with_completion("foobar list")
  )
)

would go from something like this in the bash completions:

opts=" -h -V  --help --version  <thing> "

to this:

opts=" -h -V  --help --version  $(foobar list) "

(N.B. that this approach applies to Args and not the subcommands themselves).

[CreepySkeleton]

I must admit I like the "special option to autocomplete" approach very much, but I've also came up with a number of tricky questions.

Let me summarize the prior discussion and outline the desired solution that should work and satisfy everybody.

We want this facility somewhere in clap:

fn dynamic_completion<F>(hook: F) -> Variants 
where 
    F: FnOnce(Input) -> Variants;

Where to squeeze the fn in? I believe it should be a method in App, and this method should only affect the current subcommand by default. Developers are supposed to call this method for every subcommand separately. We may also provide a "global" variant.

The hook here is effectively a piece of developer-written code whose job is to look at the args that have already been supplied and provide the completion variants back to clap.

How it works: presence of this kind of hook tells clap to generate a special --autocomplete option. The binary simply prints the completion variants, one per line (see example below).

How does clap supply the variants back to the shell so the later can use it? It doesn't. What it does is printing the values to stdin. This option is supposed to be called from inside the completions script, like that. The script will handle the rest in a shell-specific way.

What does Variants look like and why not using simple Vec<OsString>? Because we need to leave the ability for devs to fall back to the default completion script, see also #1793 to get what I mean by "default completion".

// we may want to extend the system in future
#[non_exhaustive]
enum Variants {
    /// Return this when the completion hook couldn't decide on the completion.
    /// This will fall back to the default completion system which should work sensibly.
    Fallback,
    
    /// Return when the hook did found some good options to complete.
    Success(Vec<OsString>) 
}

Some examples:

# What user does
cargo +<TAB>

# the call 
cargo --autocompletion 1 '+'

# the output (OK, pack the values and return early)
SUCSESS
nightly
stable
beta

# ATTENTION! There's a space between `+` and `<TAB> 
cargo + <TAB>

# the call
cargo --autocompletion 2 '+' 

# the output (Oops, let's just cross fingers and let the script handle the rest)
FALLBACK

The next question is: what is Input. Well, we need to

1. Allow devs to specify the name of the long option
2. Trim everything prior to the actual args for the current subcommand, if any
3. Tell user if there was a space between the last arg and TAB

fn dynamic_completion<F>(name: &str, hook: F) -> Variants 
where 
    F: FnOnce(bool, &[OsString]) -> Variants;

.dynamic_completion(|has_space, args| { /* ... */ })

// alternatively, see |args, str| design in the first comment

Does everybody agree with this design? Any unresolved questions?

[Manishearth]

Yeah, this is essentially what I proposed above, with the small tweak that I was using -- as an argument separator instead of quoting the entire thing (I find quoting to just be very messy), and also I was havng the completion function live on the Arg. I do think having a completion function on the Arg (and one that lives on the whole command) is good: this way it is very easy to write dynamic completion for just one argument, but you can also write it for the whole function if you want.

I'm less fond of the has_space, args approach because the user needs to reparse the args in this case. IMO we should parse the rest of the args and surface them to the user in a structured way, and then surface the partial string being tab-completed.

Trim everything prior to the actual args for the current subcommand, if any

I think it should be all args except the one currently being tabbed. It should be okay to tab-complete in the middle of an arg list

[CreepySkeleton]

instead of quoting the entire thing

No quoting the entire string here (we simply don't know what the string was with most shells). We pass the args array as is, quoting each arg to guard against spaces. But using -- is probably a good idea so, if user used -- on his own, it would be accounted for.

# what we do
app arg1 --flag --opt1 val1 'val 2' -- something <TAB>

# what we get
app --autocomplete 7 -- '--flag' '--opt' 'val1' 'val 2' '--' 'something'

user needs to reparse the args in this case

User needs to reparse all args in any case. The "partial parsing" problem was reported as #1880, and I think this could work as a viable option. But anyway, the hook should work on top of bare strings and expect partially_parsed_matches to be incomplete and/or not quite correct.

So, maybe:

// or maybe index instead of has_space?
.dynamic_completion(|has_space, partially_parsed_matches, args_as_os_strings| { /* ... */ })

[kenoss]

A few weeks ago I saw completion approaches of git, kubectl, pyenv and clap_generate and I found that dynamic completion will be suitable for clap.

I agree with th CreepySkeleton's first observation. But, I vote Manishearth's point. I prefer the completion function live on the Arg.
I think forcing users to parse args themselves decrease clarity of clap. We should provide structural way if we can.

I expect normalization works for #1880 .

Is there a case one have to parse raw string? Let me classify the cases.

Simple cases.

# Candidates are options of `branch` subcommand.  `App("branch")` is responsible for completion.
$ git branch -<TAB>

# Candidates are short options of `branch` subcommand.  `App("branch")` is responsible.
$ git branch -a<TAB>

# Candidates are branches.  `Arg` with index of `App("branch")` is responsible.
$ git branch -a <TAB>

# No candidates.  `Arg` with short name `m` is responsible.
$ git commit -m <TAB>

More complex cases.
Note that completion of git 4 uses variables:

• cur: current word, word arround
• prev: previous word
• words: array of arguments
• cword: num of arguments

# `--strategy` and `--strategy-option`
$ git cherry-pick --strategy<TAB>

# Candidates are strategies.  `--strategy` is responsible.
$ git cherry-pick --strategy <TAB>
$ git cherry-pick --strategy=<TAB>
$ git cherry-pick --strategy <TAB> foo
$ git cherry-pick --strategy=<TAB> foo

The code is here.
This uses cur and prev.

Clap version will be done easily because parser will remove ambiguity of and = and we get candidates by asking Arg("strategy").

$ git fetch <TAB>
benches/        clap_derive/    etc/            ip6-localhost   localhost       src/            tests/          upstream
clap-perf/      clap_generate/  examples/       ip6-loopback    origin          target/         trochilidae

$ git fetch --mirror <TAB>
cow          destringify  master-orig  staging      trying       v2-master
debug        master       release      staging.tmp  v1-master    v3-dev

$ git fetch origin:
(nothing)

$ git fetch --mirror origin:
HEAD    debug   master

The code is here.
This uses subcommand's name and investigates its options. Note that git has subcommands at most depth 1.

Consider the last case. Clap version will be done by querying subcommand's name and options from Arg.

---

Seeing git's cases, I feel parsing by clap + Arg.with_completion + some query mechanism is sufficiently powerful.

[kenoss]

Note that clap can automatically derive completion for short/long options in many cases, i.e., if one doesn't need special treatment.

[PoignardAzur]

Is there progress on this issue? The approach suggested by @CreepySkeleton seems pretty good to me.

If the issue is stalled for need of a contributor, I'd be interested in spending a few days on it (especially if I can get some mentoring). I'd really like to improve cargo's autocompletion.

[pksunkara]

This is still in design phase. According to the current suggested design, this needs more foundational work regarding partial parsing (#1880).

[epage]

@AndreasBackx I've got the start for #3166 in #3656. Once I have it merged, want to take over flushing it out so it sufficient for existing shell completions (cargo's handwritten or existing clap_complete users)? Once we are at that point, we can look at resolving this issue.

Additionally, are there any other communication channels that are more direct? The old Gitter chat does not seem to be active, there doesn't happen to be some chat channel elsewhere?

Sorry, forgot to respond to this one. We do have a zulip instance that we've not really advertised past the maintainers yet. I can also be reached on the rust-lang zulip strean "wg-cli". I'm also on both rust related discord instances but I tend to miss notifications there.

EDIT: I've also gone ahead and joined "kbknapp/clap-rs" on gitter