Add string tokenize builtin command

[amosbird]

Output tokenized history commands. This is useful for tools like fzf to do further filtering and completion.

implemented in #3818 .

[krader1961]

The initial description could use more details. I searched the fzf issues and couldn't find anything relevant. Will fzf actually use such a feature?

In any event we've received a couple of questions in the recent past for how to produce tokens from a command line in the same way fish does internally. I recall one answer was to abuse commandline to do so but that can be characterized as a hack. It would be better to augment the string command with a new subcommand, tokenize, to do the job. That could then be combined with history output or any other source of command lines.

[amosbird]

Currently we can only rely on history-token-search-backward to do arguments completion. For long arguments, interactive fuzzy filtering is more usable than blindly back stepping history tokens one by one. Consider this example,

I use fzf-tmux with the following fish function,

  function fzf-history-token-widget -d "Show command history"
    set -q FZF_TMUX_HEIGHT; or set FZF_TMUX_HEIGHT 40%
    set str (commandline -jc)
    set tok (commandline -tc)
    begin
      set -lx FZF_DEFAULT_OPTS "--height $FZF_TMUX_HEIGHT $FZF_DEFAULT_OPTS +s --tiebreak=index --bind=ctrl-r:toggle-sort $FZF_CTRL_R_OPTS +m"
      if [ $str = $tok ]
        history | eval (__fzfcmd) -q '$str' | read -l result
        and commandline -- $result
      else
        string tokenize -a -n 1000 | eval (__fzfcmd) -- -q '$tok' | read -l result
        and commandline -tr -- $result
      end
    end
    commandline -f repaint
  end

[krader1961]

Please do not implement this until a firm design proposal is written. @faho and I just had a conversation on Gitter about the somewhat surprising behavior of commandline --tokenize. We do not want to implement string tokenize as a 100% compatible version of the former. @faho and I will provide more details on the problems with commandline --tokenize in subsequent comments.

[faho]

@krader1961 and I have just discussed some questionable behavior of commandline --tokenize on gitter that are of interest here.

Specifically, commandline --tokenize currently

• Ignores all "non-string" tokens - this includes pipes, semicolons and newlines

• Strips quotes from strings (so echo "wurst" becomes echo and wurst)

• Prints commandsubstitutions as a single token

The questions are:

• How should these be handled properly?

• Should we fix it in both string tokenize and commandline --tokenize or first in the former or keep the latter as-is for all time for backwards-compatibility?

I'd argue that ignoring "non-string" tokens is inherently broken, and causes anything that relies on it to be broken as well.

[krader1961]

Note that you can trivially explore the current tokenize behavior. Run

function t; echo; printf "|%s|\n" (commandline --tokenize); end
bind \cT t

Then type various commands that include pipes, semicolons, quoted strings, subcommands, etc. and press [ctrl-T] to see how what you have typed is tokenized.

[krader1961]

My recommendation is to leave the current commandline --tokenize behavior unchanged even in the next major release but with a warning that it will be changed in the subsequent major release. Implement string tokenize ASAP with whatever improved behavior we decide upon. Then reimplement all of our core uses of commandline --tokenize in terms of a combination of the two commands. Third-party extensions can then use string tokenize for dealing with both the current command and those from the command history.

[mqudsi]

@faho @ridiculousfish are we ditching $IFS or did we cancel that idea?

[faho]

are we ditching $IFS or did we cancel that idea?

I still want to. It's a terrible interface. read already gained a -d option that supersedes IFS, so the only missing piece to gain full feature-parity is adding some way to not split command substitutions.

[faho]

So, I now have a branch that implements string tokenize: https://github.com/faho/fish-shell/tree/string-tokenize

It keeps "special" tokens like redirections and pipes, so you should get a fairly complete representation of the string.

It doesn't go into command substitutions, because our tokenizer itself doesn't.

It uses the --no-quoted option to remove quotes, without it it keeps them (string tokenize -- "echo 'foo'" prints echo and 'foo' - add -n and the latter becomes foo).

But I'm not sure if that's the right way around or useful at all, because I'm not quite sure what this is used for. I know there was one case where I hacked a completion or binding to set commandline to be able to use commandline --tokenize, but I can't find it.

Essentially if you keep the quotes, you can know that it had quotes (useful for the magic paste stuff, I'd imagine). If you strip them, you can more easily pass it to a second tokenize pass, or to a hypothetical expand builtin (#5811)