Completion script syntax

[remkop]

I just found yori and it looks wonderful! Congrats on a beautiful product!

I am interested in generating completion scripts for picocli-based command line applications. (Picocli is a library for easily building CLI apps in languages that run on the JVM.) The picocli library currently supports completion for bash and zsh, and I'm looking at providing support for fish, native zsh, and clink. Now that I found yori I am thinking to add yori to that list and point Windows users at your project as well as clink.

I have looked at the completion scripts in the project and the documentation on completion scripts, but I have not found a description of the completion script syntax. I may be able to figure it out from the example scripts but it would help a lot if there was some docs, even just some notes would be useful. Perhaps this already exists and I just haven't found it yet?

[malxau]

Can I ask what about the documentation is unclear? Obviously I'm a little terse, since I don't want to create documentation that's so large that it's not read.

The first argument to a completion script is the argument number being completed; the second argument is the argument string that has been entered so far requiring completion.

So, if you type:

foobarbaz onething twothing three<TAB>

The script foobarbaz.exe.ys1 would be invoked, and the first argument would be "3" because it's completing the third argument, and the second argument would be "three" being the text that is being completed.

For the git completion script I recently extended it so the script receives each of the arguments that are currently entered in the command line, meaning that the first argument on the command line is the third argument to the script, etc. However, that change is checked in but not in the stable release, so I didn't want to update the documentation to refer to it just yet.

The high level thing though is these scripts are far more limited than bash, but far more flexible than CMD. I always wanted to add regex support, but without it, these are always going to be a bit limited and hacky.

[remkop]

In trying to answer your question, I spent some time on this 😄 and I was able to figure out some things from the examples. Here is how I believe the documentation can be improved.

Mention echo

The first part I did not understand was how to generate completion candidates. I would mention that:

• completion scripts can generate completion candidates by calling echo -- /<switch> [args]. (That is what "output" means in the manual: it means calling echo. This was not obvious to me.)
• after the script generate completion candidates by calling echo, the candidates are then automatically filtered by yori: from the generated list, only those candidates are displayed that match the text that is being completed in the user input. (That is what the "Matches (against)" bit in the switch description means. I did not understand that at first.)

The doc jumps straight into the available values for <switch>:

About the options for automatically generating completion candidates

A completion script can output:

switch	description
/commands	Matches executables and builtin commands
/directories	Matches directories only (not files)
/executables	Matches executables only
/files	Matches files and directories
/filesonly	Matches files (not directories)
/insensitivelist	Matches against an explicitly specified list, case insensitively
/sensitivelist	Matches against an explicitly specified list, case sensitively

Questions about these values:

• yori automatically generates completion candidates for certain switches like /commands, /directories, /files (etc). Are the completion candidates generated by the /directories and /files switches dependent on the current directory of the user?
• It is unclear how yori obtains the list of commands and executables, and whether this list is customizable.

Add Some Brief Examples

It would be good to show the source for a few example completion scripts in the documentation. (If this can be done in a few lines for each example.)

Mention syntax / programming language

It may be good to state explicitly that:
Completion scripts are essentially Windows batch files and can use the full syntax of Windows batch files. (Please correct me if I'm wrong.)

My understanding now is that:
As long as the script eventually calls echo, you will have a working completion script.

This is a key insight that took me a while. Either mentioning this explicitly or having a few brief examples would have helped me get there sooner.

How to access the full user input

For the git completion script I recently extended it so the script receives each of the arguments that are currently entered in the command line, meaning that the first argument on the command line is the third argument to the script, etc. However, that change is checked in but not in the stable release, so I didn't want to update the documentation to refer to it just yet.

Thank you for mentioning that. I did not realize that the full user input is currently not available to completion scripts.

The full input is necessary to correctly generate candidates for commands with subcommands. Completion scripts need to be able to distinguish between input like the below:

git add -<TAB>
git remote add -<TAB>

Parsing support?

As an aside, it would be nice if there was some parsing support.

In a concrete example for a command with subcommands, suppose the user entered git remote add -<TAB>, we want our completion script to generate only the options for git remote add: -t, -m, -f, --tags, --no-tags, --mirror.
To accomplish this, our script needs to use the batch syntax to parse the tokens that the user entered, and eventually call echo -- /sensitivelist -t -m -f --tags --no-tags --mirror.
There is no mechanism to help with this token parsing. This is where clink's idea to use lua completion scripts and provide a simple parser abstraction really shines. See the example code here.

[malxau]

Note this part in the guide:

These scripts can be Yori scripts or any other executable.

The supplied scripts are Yori scripts, which are similar to batch scripts except they are interpreted by Yori rather than CMD. The differences can be seen in things like the argument specifiers (ie., %2% vs %2) and in the if statements, which are quite different. But there's no technical reason why completion scripts can't be written in perl, python or even powershell. The use of echo is really specific to Yori scripts or CMD - any other language will have its own way to output text.

For the commands and executables list, executables refer to files that Yori can find in the user's %PATH% that have an extension specified in %PATHEXT%. Commands includes all executables as well as builtin commands, which are known to the shell. Unlike most other shells, Yori allows builtin commands to be added and removed dynamically without needing to be statically linked (see http://www.malsmith.net/yori/guide/#modules .) But at all times, a builtin command is either already known to the shell, or is capable of being registered by a binary that will be searched for via %PATH% and %PATHEXT% .

Files and directories are matched based on the current directory, but they can be fully specified paths as well. Yori is effectively assuming that a program which can take a file as input can take either a relative or absolute path to that file. Note though that Ctrl+Tab will expand the full path to a file.

I agree that non-trivial completion scripts need the full command, which is why I recently added that, although what's currently there is still a bit painful to use. I also agree that a richer parsing language would make this much better, and hoped that regex would be a large part of that. Although it looks like what you're referring to here may also be a big part of it, because arguments can be specified in many orders and in many ways so direct string compares are probably insufficient to capture the user's intention. What's frustrating in Windows is that command strings are passed to a receiving process which is free to interpret them however it wants, so there's really no standard for argument parsing, which makes a fully generic approach very difficult.

[remkop]

Output

The use of echo is really specific to Yori scripts or CMD - any other language will have its own way to output text.

Understood. Thank you for the explanation! That clears up a lot.

• Completion scripts are run as an external process, and can only communicate to Yori via the standard output stream. (Other completion systems I have seen are in-process and call built-in commands; this is why I had trouble understanding what "output" meant in the manual.)
• Yori generates completion candidates, the completion script merely gives instructions.

May I suggest the following text for the user manual:

A completion script can write one of the following instructions to the standard output stream:

/commands Matches executables and builtin commands
/directories Matches directories only (not files)
/executables Matches executables only
/files Matches files and directories
/filesonly Matches files (not directories)
/insensitivelist Matches against an explicitly specified list, case insensitively
/sensitivelist Matches against an explicitly specified list, case sensitively

Based on the specified instruction, Yori will then generate completion candidates and display them to the user.

Identifying the completion script

Note this part in the guide:

These scripts can be Yori scripts or any other executable.

I see now! (Again, I did not expect that Yori executes completion scripts in an external process. Hence communication via the standard streams. Makes sense now.)

May I suggest adding a few words to explain how Yori matches the command in the user input that is being completed with the correct completion script in %YORICOMPLETEPATH%? Something like:

These scripts can be Yori scripts or any other executable whose name starts with the command under completion. (somecommand and somecommand.exe are considered different commands and require separate completion scripts.)

Parser

It is an interesting approach to allow completion scripts written in any language. However, I suspect that many people will stick to scripts in CMD or Yori script since those are guaranteed to work. Looking at the example scripts I am discouraged to imagine writing a non-trivial completion script for something like git using a CMD-like syntax.

I hope you will give consideration to my suggestion in #47. Clink provides a very pleasant API, that allows extremely terse and to-the-point syntax for writing complex completion scripts like this:

local parser = clink.arg.new_parser
local git_parser = parser(
    {
        "add" .. parser({}, 
            "-v", "--verbose",
            "-f", "--force",
            "-i", "--interactive",
            "--refresh"
            ),
        "commit" .. parser(
            "-a", "--all",
            "-m", "--message=",
            "-v", "--verbose",
            "-q", "--quiet",
            "--"
        ),
        "remote"..parser({
            "add" ..parser(
                "-t"..parser({branches}), -- defined elsewhere
                "-m",
                "-f",
                "--mirror",
                "--tags", "--no-tags"
            ),
            "rename"..parser({remotes}),
            "remove"..parser({remotes}),
            "rm"..parser({remotes}),
            "set-head"..parser({remotes}, {branches},
                "-a", "--auto",
                "-d", "--delete"
            ),
            "set-branches"..parser("--add", {remotes}, {branches}),
            "set-url"..parser(
                "--add"..parser("--push", {remotes}),
                "--delete"..parser("--push", {remotes})
            ),
            "update"..parser({remotes}, "-p", "--prune")
            }, "-v", "--verbose"),
    },
    "--version",
    "--help",
    "-c",
    "--git-dir",
)

clink.arg.register_parser("git", git_parser)