Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

completions: Support adding additional code to complete values #568

Open
joshtriplett opened this issue Jul 4, 2016 · 43 comments
Open

completions: Support adding additional code to complete values #568

joshtriplett opened this issue Jul 4, 2016 · 43 comments

Comments

@joshtriplett
Copy link
Contributor

@joshtriplett joshtriplett commented Jul 4, 2016

For some argument values, the bash-completions may want to include additional logic for what type of value to complete, rather than allowing arbitrary strings. For instance, an option might accept a path to a file that exists; bash has a mechanism for that. Or, an option might accept the name of a git ref that exists; that's something more easily implemented in the program in Rust. Either way, it makes sense to augment clap's completion logic.

This also argues for implementing the completions by calling the program at runtime, rather than via shell; that way, arbitrary logic in Rust can determine what to complete, rather than providing a template system to feed in shell code (ick).

@kbknapp
Copy link
Member

@kbknapp kbknapp commented Jul 4, 2016

Perhaps adding something like was discussed in in #376 where there is a "completer" function? I'm all for this, but figured it's also addable in a backwards compatible way once I had the base implementation complete.

I'm just not sure which would be the best way to add this so I'm open to all ideas.

@joshtriplett
Copy link
Contributor Author

@joshtriplett joshtriplett commented Jul 4, 2016

I'm honestly not sure either. I'm really hesitant to suggest inlining shell script snippets into Rust code as strings; I'd rather see those written in a separate shell file and included from there (not least of which to get the right filetype and highlighting). bash (via compgen) and bash-completion (via functions in /usr/share/bash-completion/bash_completion) have some built-in helpers, and it'd be nice to support those for common cases like hostnames, users, and files (with glob patterns). Someone might also want to write arbitrary shell code to enumerate argument values. It'd also be nice to support using arbitrary Rust code by invoking the program.

I think what I'd suggest is that the .completer function should take an enum argument, where that enum has values like User, File, FileGlob("*.ext"), BashFunc("__comp_function_name"), and RustFunc(...). Those would then translate into appropriate calls to compgen, calls to the specified function, or invocations of the program to run Rust code. (That last one would also require something like a global_setting to enable a --clap-complete option or similar.)

This is turning out to be a remarkably hairy yak.

@mathstuf
Copy link

@mathstuf mathstuf commented Oct 2, 2016

In zsh at least, clap could generate completion function calls such as:

(( $+functions[_appname_clap_complete_ARG] )) || _appname_clap_complete_ARG () {
}

Which can then be overridden in a supplemental file included before this one (via source if the file exists). Bash probably has some mechanism that works similarly.

@emk
Copy link

@emk emk commented Oct 10, 2016

I've just converted cage to use clap, and I'm very happy with the results. Basic completion works under both bash and fish. Great code!

But cage would benefit enormously from being able to dynamically complete the names of docker services for commands like:

cage test $SERVICE_NAME

If the project in the current directory has the ervices foo and frontend/bar I would like to be able to do the following:

> cage test f<TAB>
foo
frontend/bar

I would be happy to add an extra argument to the app, something like:

> cage --_complete-service f
foo
frontend/bar

And declare this as:

- SERVICE:
    value_name: "SERVICE"
    required: true
    complete_with: "_complete-service"
    help: "The name of the service in which to run a command"

Obviously the details could vary a bit, but we would ultimately have --_complete-pod, --_complete-service, --_complete-pod-or-service and --_complete-repo-alias, among others. Also note that many different subcommands would share each completion hook, which might mean we want these to be potentially global.

emk added a commit to faradayio/cage that referenced this issue Oct 10, 2016
Note that our completion support is incomplete pending solutions for
these two issues:

clap-rs/clap#568
clap-rs/clap#685
@kbknapp
Copy link
Member

@kbknapp kbknapp commented Oct 10, 2016

@emk Your post has me thinking about this more and more, I'm thinking some sort of hybrid between what @joshtriplett listed above and what you're proposing.

My schedule is pretty busy this week, but I should be able to at least test some ideas and see the feasibility. Stay tuned to this thread for updates!

@emk
Copy link

@emk emk commented Oct 10, 2016

@emk
Copy link

@emk emk commented Oct 11, 2016

Ah, here we go. Some docs on how several Python arg parsing libraries handle --_completion.

selfcompletion is a layer on top of argparse to take the fine-grained model
argparse builds of the arguments your program accepts and automatically
generate an extra '--_completion' argument that generates all possible
completions given a partial command line as a string.

The '--_completion' argument in turn is used by a generic bash programmable
completion script that tries '--_completion' on any program that doesn't have
its own completion already available, renders the output of the program's
built-in completion if available, and otherwise silently falls back to the
shell default.

Here is the generic completion function for bash:

_foo()
{
    prog="$1"
    while read glob_str; do
        case $prog in
        $glob_str)
            return 1;;
        esac
    done < <( echo "$SELFCOMPLETION_EXCLUSIONS" )
    which "$prog" >/dev/null || return 1
    _COMP_OUTPUTSTR="$( $prog --_completion "${COMP_WORDS[*]}" 2>/dev/null )"
    if test $? -ne 0; then
        return 1
    fi
    readarray -t COMPREPLY < <( echo -n "$_COMP_OUTPUTSTR" )
}

complete -o default -o nospace -D -F _foo

The advantage of this approach is that the per-shell code can be written only once, and all the hard work can be done directly by the application itself. Obviously, there might be disadvantages as well. But I figured it was worth tracking down all the existing attempts to standardize this to see if any of them had helpful ideas. :-)

@joshtriplett
Copy link
Contributor Author

@joshtriplett joshtriplett commented Oct 24, 2016

@kbknapp Any updates on this mechanism? I have someone asking after completions, and I'd love to beta-test this.

@jcreekmore
Copy link

@jcreekmore jcreekmore commented Oct 25, 2016

@kbknapp I would be interested in this as well. I am currently post-processing my completions to substitute in _filedir for filename completion, but that is less than ideal.

@kbknapp
Copy link
Member

@kbknapp kbknapp commented Oct 25, 2016

@joshtriplett @jcreekmore

Now that the ZSH implementation is complete I've got a better handle on this. The biggest issue I see holding this up is that completions are done differently between all three (so far) supported shells.

I'm all for some sort of enum with variants that allow things like, Files, Directories, Globs(&str), Code(&str) or something to that effect. But some shells support those things verbatim, others only in arbitrary ways that clap doesn't use when gen'ing the completion code.

I'm just unsure of the best way to expose this. Perhaps on an Arg::complete_with(enum)?

I guess, first what is the shell you're trying to support, and what particular portions are you wanting to inject into the completion code?

@emk
Copy link

@emk emk commented Oct 25, 2016

@kbknapp For cage, we'd like to be able to complete custom "types" of values, such as Docker container names, "pod" names, target environments, and so on. The legal values can only be determined by asking our executable at runtime, since they vary from project to project.

This is pretty much how git-completion handles origin names, branch names, etc.

The problem with an enum is that it would limit us to just a few built-in types such as Files, Directories, etc., right?

@joshtriplett
Copy link
Contributor Author

@joshtriplett joshtriplett commented Oct 25, 2016

@kbknapp I don't think you need to support embedding arbitrary shell code from Rust. My suggestion would be to support the lowest-common-denominator bits (filenames, usernames, filenames matching one of these patterns, etc), and then have a "call this Rust function" variant that invokes the program itself with some internal hidden --clap-complete option that dispatches to that Rust function. That makes it easy to do things like "a git ref matching this pattern", by calling a Rust function implementing that.

For those common categories like filenames or usernames, use the shell built-in mechanisms if available, or provide and use code implementing those simple completions if the shell doesn't have them.

If people want "invoke this shell code", I'd suggest adding a Rust variant to call a named shell function, and then letting people add that shell function to the resulting generated completions for any shell they support. That seems preferable to embedding many variants of shell code directly.

enum Completion<F: Fn(...) -> ...> {
    File,
    User,
    Hostname,
    Strings(&[str]),
    Globs(&[str]),
    ShellFunction(&str), // maybe
    RustFunction(F),
}
@kbknapp
Copy link
Member

@kbknapp kbknapp commented Oct 25, 2016

@emk Yes, and no. It would be extensible, so more variants could be added. But Some of the variants could also take additional parameters, and ultimately (possibly) injecting arbitrary shell script via something like ,Code(&str) which of course isn't super great, but perhaps a fallback if a particular variant doesn't quite fit the bill. (Or perhaps not...it could end up being massively unsafe 😜 )

At the same time, I haven't looked into exactly where this code would be injected and ultimately if it's even feasible yet. This is just straight of the top of my head right now.

Also, if the "types" are know prior to runtime ZSH already supports this just by using the Arg::possible_values

@joshtriplett
Copy link
Contributor Author

@joshtriplett joshtriplett commented Oct 25, 2016

That's true, "one of these fixed strings" should be an option as well. Updating the type in my previous comment.

@kbknapp
Copy link
Member

@kbknapp kbknapp commented Jan 14, 2017

From @Xion in #816

In Python, there is a package called argcomplete which provides very flexible autocompletion for apps that use the standard argparse module. What it allows is to implement a custom completion provider: essentially a piece of your own code that's executed when the the binary is invoked in a Special Way (tm) by the shell-specific completion script.
For an example, see here. The code is preparing completions dynamically from the filesystem, or even from a remote API (if certain flag isn't passed (flags are partially parsed at this point)).
Having something like this in clap would be very nice. I know this is a potentially complex subsystem so it'd be unreasonable to expect it implemented anytime, but I wanted to at least put this feature on the radar.

@kbknapp
Copy link
Member

@kbknapp kbknapp commented Jan 14, 2017

After reading through some of the argcomplete python module the hardest part will be figuring out how to call a Rust function from the shell.

@kbknapp
Copy link
Member

@kbknapp kbknapp commented Jan 14, 2017

I'm guessing what'll end up happening is some sort of double run with hidden args.

@kbknapp
Copy link
Member

@kbknapp kbknapp commented Jan 19, 2017

Expanding on the ideas (from #818)

The problem with implementing this is I just haven't had a good time to sit down and think about how (because of work, holidays, family, etc.). I want a way to specify this that abstracts well enough to work for all shells. The easiest way is to say, "Put your arbitrary completion shell script here inside this Arg::complete_with(&str)" but that feels super hacky to me, and potentially unsafe. What I'd like to do is provide a Arg::complete_with(Fn(&str, &str)->String) (and a Arg::complete_with_os(Fn(&str,&OsStr)->OsString) where an arbitrary Rust function is called...but herein lies the problem; shell completions are run before the program executes. This has led to some people using hidden args or something like, $ prog --complete me<tab> calling a shell completer that actually runs $ prog --_complete_arg "complete" --_complete_prefix "me" which generates the possible completions and returns them to the shell. I'm not against doing that, but again feels strange because you're injecting hidden args into a CLI. Although typing this out right now does make me lean towards this solution.

@softprops
Copy link

@softprops softprops commented Jun 5, 2018

Here's some inspiration from how golang kingpin package handles dynamic tab completion using the bash COMPREPLY bash protocol

Kingpin supports dynamic completions via it's HintAction interface

I imagine this should be possible providing a fn interface to clap's Arg type for dynamic completions

Under the covers kingpin forks program control when in completion mode to invoke that completion func then exits the process.

It switches modes of operation based on a flag that the completion script passes.

@adamtulinius
Copy link

@adamtulinius adamtulinius commented Sep 6, 2018

Here's some inspiration from how golang kingpin package handles [..]

Please not that kingpin currently can't complete file paths, which need something like alecthomas/kingpin@master...DBCDK:hint-files to fix. I'm just mentioning this, because it required work on the compreply stuff, and might be useful here as well.

@NotBad4U
Copy link

@NotBad4U NotBad4U commented Nov 21, 2018

Hello everyone 😄
Any plans to progress on this feature? I could give it a try, but I would need a bit of mentoring / pointing to the right places.

@joshtriplett
Copy link
Contributor Author

@joshtriplett joshtriplett commented Nov 21, 2018

@NotBad4U
@kbknapp is interested in seeing that as clap gets closer to 3.0, and we're likely to brainstorm how it'll work in the next few months. We'll ping this bug when doing so.

@IssueHuntBot
Copy link

@IssueHuntBot IssueHuntBot commented Dec 3, 2018

@issuehuntfest has funded $50.00 to this issue. See it on IssueHunt

@Dylan-DPC
Copy link
Contributor

@Dylan-DPC Dylan-DPC commented Mar 23, 2019

@NotBad4U are you still interested in working on this issue? If yes, then we are happy to mentor you

@NotBad4U
Copy link

@NotBad4U NotBad4U commented Mar 26, 2019

yes 😄 I'm still interested @Dylan-DPC

@IssueHuntBot
Copy link

@IssueHuntBot IssueHuntBot commented Mar 27, 2019

@cben has funded $10.00 to this issue.


@Dylan-DPC
Copy link
Contributor

@Dylan-DPC Dylan-DPC commented Mar 27, 2019

@NotBad4U you can start working on it. if you need any help you can ask us in #wg-cli channel on discord.

@zx8
Copy link

@zx8 zx8 commented Jun 10, 2019

https://github.com/posener/complete is a library dedicated to dynamic completion written in Go, if you're looking for some ideas/inspiration. Despite the repo's description, it supports bash, zsh & fish.

@benjumanji
Copy link

@benjumanji benjumanji commented Apr 4, 2020

Here is a proposal that I think would be relatively simple. I haven't checked how zsh completion is done, but it's my understanding that it understands bash so you could just re-use it?

The generated scripts for both fish and bash already support dynamic completions. Both compgen for bash and complete for fish honor parameter expansion, including shell substitution and word split. That is to say if you throw $(cat words) for bash or (cat words) for possible_value into an arg it totally works! Right up until clap validation kicks in and says it's not a valid value.

Proposal: add dynamic to arg which adds the shell code and disables value validation. I can't justify working on this right now, but if this is an acceptable proposal I might be able to circle back around to it in a few weeks when I might need it.

@8573
Copy link

@8573 8573 commented Apr 4, 2020

I haven't checked how zsh completion is done, but it's my understanding that it understands bash so you could just re-use it?

I've forgotten almost all I once knew about the Zsh completion system, but I recall that it's much more powerful/expressive(/fancy) than Bash's such that, if I recall correctly, telling it to use Bash completion, while the easy way out, could provide a needlessly suboptimal user experience. That said, I would think that such a project could start with telling Zsh to use Bash completion and come back and write native Zsh completion later.

@pksunkara pksunkara added this to the 3.0 milestone Apr 9, 2020
@pksunkara
Copy link
Member

@pksunkara pksunkara commented Apr 9, 2020

#1793 is an attempt at fixing this in zsh.

@epage
Copy link
Contributor

@epage epage commented Jul 22, 2021

Are #1232 and this now dupes? Should we close one in favor of the other to make it easier to browse the backlog?

@pksunkara pksunkara removed this from the 3.0 milestone Jul 26, 2021
@pksunkara
Copy link
Member

@pksunkara pksunkara commented Jul 26, 2021

I kept them separate because this issue is more concentrated on leveraging shell completions while the other is for a full fledged solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet