Make completions for subcommands easier to add

[sudormrfbin]

The current way to add completions for subcommands of a command, as can be seen from the default completions in share/completions, is to define functions like __fish_cmdname_needs_subcommand and __fish_cmdname_using_subcommand and then use them as conditions like so:

# main subcommands
complete -c cmdname -n "__fish_cmdname_needs_subcommand" -a "subcmd1"
complete -c cmdname -n "__fish_cmdname_needs_subcommand" -a "subcmd2"
# completions for options of subcommands
complete -c cmdname -n "__fish_cmdname_using_subcommand subcmd1" -l subcmd1-specific-option

This leads to a lot of duplicated code and could be buggy. Adding a --subcommand flag to the complete command to specify subcommands and options exclusive to them could be very useful. Something like this:

complete -c cmdname --subcommand subcmd1
complete -c cmdname --subcommand subcmd1 -l subcmd1-specific-option

Or maybe the -c flag itself can take care of the subcommands, which IMO is more readable and intuitive, especially in the case for subcommands of subcommands:

complete -c 'git remote' -d 'Manage set of tracked repositories'
complete -c 'git remote' -s v -l verbose -d 'Be verbose'
complete -c 'git remote add' -d 'Add new remote'

[krobelus]

One complication is that fish needs knowledge of available command options to know what is the subcommand in a commandline like git -c color.diff.old=blue diff.

The code duplication could be avoided by using a function to generates the repetitive complete lines.

[sudormrfbin]

@krobelus I'm not sure I completely understand the complication; are you referring to the ambiguity of eithercolor.diff.old=blue or diff being interpreted as subcommands by fish ? If so, wouldn't that be solved by the --require-parameter option which can be specified when writing complete commands ? It can clearly specify that the token following a particular switch is to be considered as an argument to the switch itself and not as a subcommand.

I was referring to the code duplication due to __fish_cmdname_needs_subcommand and __fish_cmdname_using_subcommand being defined repeatedly in every completion file that needed subcommands.

[krobelus]

If so, wouldn't that be solved by the --require-parameter option which can be specified when writing complete commands?

Yes, you're right that would work.

I was referring to the code duplication due to __fish_cmdname_needs_subcommand and __fish_cmdname_using_subcommand being defined repeatedly in every completion file that needed subcommands.

I see. I don't think this really hurts, but of course the completions are unsightly and not as easy to write as they could be.

I think this could turn out to be a reasonable addition.

Nested subcommands would work as well:
complete -c git --subcommand remote --subcommand add -s f

[zanchey]

Some previous discussion of one approach in #4226.

[mqudsi]

I'm all for this.

Note that a lot of the current completions that deal with subcommands are a little iffy. There are different approaches taken, but the most common one to determine whether or not to suggest a subcommand depends on !seen_subcommand_from, while the definition of seen_subcommand_from just loops over all parameters and does not take their location/index into account.

Aside from being inefficient, it's also pretty wrong since the literal subcommand keyword may easily appear as a payload option to another command. Other completions are using __fish_is_first_arg to determine whether to suggest a subcommand, but that's wrong because it's common to have an optional switch for the command that modifies its (or a future subcommand's) behavior, such as git --paginate [subcommand].

If you stick with the common definition of subcommand as being necessarily exclusive of any other subcommands (i.e. you cannot invoke two subcommands in one execution), and additionally confine a subcommand to being the first non-switch argument, it's a lot easier and more correct to test for whether to complete a subcommand with __fish_is_nth_token 1 and then redefine __fish_seen_subcommand_from to be __fish_is_first_token_from 1 foo or similar.

I ended up adding the mis-named is_first_token to handle this case, but it's almost (or entirely?) impossible to introspect the completions from within the completion file to determine if an argument that doesn't start with - is a payload to a previous switch, but that's something that's easily doable from within the complete builtin because it knows (or at least can know) when a -s or -l argument is expecting an argument and when it's not.

To be honest, there are so many issues with the current approach to completions (not the least of which is their eval nature) that to truly optimize for anything beyond single-level completions it might make more sense to rethink the paradigm and to support some sort of tree-based declarative syntax for basically defining an argument parser that additionally supplies completions. Just look at how much repetition there is in any given completions file because we're trying to map a literal argument tree to a flat argument-by-argument structure. (But maybe just adding --subcommand will be a good start!)

[ridiculousfish]

Maybe it is worth resurrecting the docopt approach, which knows about subcommands. I think my first run at it could be simplified. However I don't know if it will be sufficient for very advanced cases like git.

Here is what the ssh completions looked like under docopt - nice, huh.

[mqudsi]

That is cool, although I'm not a fan of having it all whitespace-delimited in an unstructured quoted string.

docopt completions are static, right? I think it makes more sense to build whatever approach we want to take around fish's support for dynamic completions, which is where it really shines as compared to the statically-generated completions that power other shells or are automatically derived from man pages. (e.g. supporting commandline introspection)

[canton7]

I was having a think about this, related to #8266, and I've written up some thoughts below.

I'm going to use git as the motivating example here, as it has a huge number of completions using multiple levels of subcommands. The ideas of course apply to any command which has subcommands.

Motivation

Completions with subcommands are, in the current design, pretty inefficient. Take for example a few lines from git's completion:

complete -f -c git -n __fish_git_needs_command -a checkout -d 'Checkout and switch to a branch'
complete -f -c git -n '__fish_git_using_command checkout' -s b -d 'Create a new branch'
complete -f -c git -n '__fish_git_using_command checkout' -s t -l track -d 'Track a new branch'

Subcommands are implemented by providing a completion for the entire git command, with a condition which determines whether the subcommand is present. Each completion which applies to a subcommand needs to repeat the same condition, and there can be many of these: git's completion file has just under 1000.

This results in fish effectively needing to perform a linear search through all completions to find the ones which are applicable to a particular given subcommand, repeating the same tests over and over again. This can add up to significant cost: each condition currently replicates the work of parsing commandline -opc, meaning git's completion currently takes around 12 seconds on Windows with most of that time taken by repeatedly testing those conditions (although #8266 improves on this significantly).

The complete statements also get quite repetative to author.

An easy solution

It turns out that complete almost has pretty nice subcommand support built-in already.complete currently supports --condition and --wraps, but if these are passed together, the condition doesn't apply to the --wraps. All that needs to happen is for --condition to also apply to --wraps, that is, a command only wraps another command if the given condition passes.

This would then let you write something like:

complete -c git -n '__fish_git_using_command checkout' --wraps 'git#checkout'
complete -f -c 'git#checkout' -s b -d 'Create a new branch'
complete -f -c 'git#checkout' -s t -l track -d 'Track a new branch'

That is, we create a "command" called git#checkout (the name is arbitrary, and I chose # as a separator simply because it doesn't appear in normal commands). This will never be completed directly (unless the user tries particularly hard). We then define another completion on git with the condition __fish_git_using_command checkout, which wraps this.

This still results in a linear search through all of the possible subcommands, which is a significantly smaller space than all possible completions for all of git. Only when a subcommand completion condition matches do we start evaluating which of that subcommand's completions apply.

This also extends to nested subcommands. To take git remote as an example:

complete -f -c 'git#remote' -n '__fish_git_using_command add' --wraps 'git#remote#add'
complete -f -c 'git#remote#add' -s f -d 'Once the remote information is set up git fetch <name> is run'
# ...

It also works well with e.g. git aliases. Currently there's a horrible hack here involving global variables. This proposal would instead let you write something like:

git config -z --get-regexp 'alias\..*' | while read -lz alias cmdline
    set -l command (__fish_git_aliased_command $cmdline)
    string match -q --regex '\w+' -- $command; or continue
    # Git aliases can contain chars that variable names can't - escape them.
    set -l alias (string replace 'alias.' '' -- $alias | string escape --style=var)
    complete -c git -n "__fish_git_using_command $alias" --wraps "git#$command"
end

Downsides

I've identified the following downsides with this approach:

• It's not very discoverable: completion authors would need to read and digest the docs.
• This is a (small) compatibility break: if users currently pass both --condition and --wraps, this change could break them.
• We still need to do a linear search through all possible subcommands.
• Those git#checkout commands enter the global namespace of all commands. The user could try and complete e.g. 'git#checkout', and get confused.

A more considered solution

A second higher-effort but neater proposal is to add two new options to complete: --subcommand and --subcommand-selector.

--subcommand takes an arbitrary string, and can be specified one or more times. Completions which specify a subcommand are not automatically considered. E.g.:

complete -f git --subcommand checkout -s b
complete -C 'git -'
# Does not suggest 'git -b'

--subcommand-selector takes a shell command, which prints either a string, or nothing. If it prints a string, then all completions which passed that same string to --subcommand are considered for completion.

This then lets you write completions along the lines of:

function __fish_git_find_command
    set -l cmd (commandline -opc)
    set -e cmd[1]
    argparse -s (__fish_git_global_optspecs) -- $cmd 2>/dev/null
    or return

    if set -q argv[1]
        echo $argv[1]
    end
end

complete -f git --subcommand-selector '__fish_git_find_command'

complete -f git --subcommand checkout -s b -d 'Create a new branch'
# Completions can apply to multiple subcommands
complete -c git --subcommand diff --subcommand show --subcommand range-diff -l color -d 'Show colored diff'

--subcommand and --subcommand-selector can be specified together, to support nested subcommands:

complete -f git --subcommand-selector '__fish_git_find_command'
complete -f git --subcommand 'remote' --subcommand-selector '__fish_git_find_subcommand' # __fish_git_find_subcommand returns e.g. 'remote add'
complete -f git --subcommand 'remote add' # ...

This can still work with aliases, although it isn't quite as neat. This will involve two calls to __fish_git_find_subcommand, one to parse the alias from git <alias>, and a second to parse the command from the resulting git <command>:

complete -c git --subcommand "$alias" --wraps "git $command"

This could be improved by adding an e.g. --wraps-subcommand option, but I'm not sure it's worth the effort.

This approach reduces the subcommand lookup complexity to O(1), and also nicely namespaces subcommands under their command. It's also a lot more obvious and discoverable.

Note that an important aspect of both of these approaches if that they don't involve complete learning how to parse subcommands out of a command-line, or really even knowing what a subcommand is (other than opaque identifier). That logic is left to the condition in the first proposal, or --subcommand-selector in the second: the user can insert whatever extra logic a particular command needs, or fall back to using a fish-provided helper function for common cases. This makes the implementation easier, and means that weird corner-cases can be handled by user functions and don't need to affect complete.

That said, to make this more user-friendly, complete could default to usnig a built-in --subcommand-selector which just takes argv[1], and subsequent calls to complete --subcommand-selector would overwrite this with the user-specified shell command. That way:

complete -c command --subcommand subcommand # ...

Would work pretty well out of the box, while retaining the ability for more advanced cases (such as git's) to provide their own more complex --subcommand-selector.

[krobelus]

One problem is that it's not so trivial to detect the subcommand; it depends on the specific command, for example

git --git-dir .git status

I think this is the canonical way right now (using argparse to correctly detect the subcommand):

complete git -xa '(__fish_complete_git)'
function __fish_complete_git
	argparse [...]
	set -l subcommand $argv[1]
	switch $subcommand
		case ''
			printf %s\t%\n \
				remote "Manage set of tracked repositories" \
				fetch "Download objects and refs from another repository"
		case remote
			switch $argv[2]
				case ''
					printf %s\t%\n \
						add "Adds a new remote" \
						remove "Removes a remote"
				case remove
					__fish_git_remotes
				[...]
			end
		[...]
	end
end