"?" glob/wildcard considered annoying/confusing (with "-?" option)

[samdmarshall]

• Have you checked if problem occurs with fish 2.4.0?
• Tried fish without third-party customizations (check sh -c 'env HOME=$(mktemp -d) fish')?

fish version installed (fish --version):
fish, version 2.4.0

OS/terminal used:

ProductName:	Mac OS X
ProductVersion:	10.12.2
BuildVersion:	16C67

Terminal.app

Fish fails to interpret the some command line arguments correctly when they use -?

Reproduction steps

https://asciinema.org/a/duts17f0sz4npm91kfvyq0fwe
(two examples of commands that use -? are graphviz's dot (brew install graphviz) and aspell (brew install aspell) )

Results

No matches for wildcard '-?'.  (Tip: empty matches are allowed in 'set', 'count', 'for'.)
fish: dot -?
          ^

I am able to work around this by providing the command line argument as -\?, so that it gets properly escaped, but I feel that this should be automatically handled.

[krader1961]

Note you can also avoid the error by quoting the string; either '-?' or -'?'. The current behavior has been reached after extensive discussion. It's pretty unlikely we're going to change this again to match bash but let's leave this open a few days to see if any of the other core devs want to revisit the topic.

[krader1961]

Also, I'm curious about your wording of the subject of this issue: "Fish fails to interpret some command line arguments without escapes". Fish is in fact interpreting the argument as a glob that matches any file name beginning with a hyphen and followed by exactly one character. Since you don't have any file names matching that pattern fish tells you as much. The bash behavior is to pass the literal glob to the command if it doesn't match any files. On the other hand if you had, say, a file named -X then your example would be equivalent to dot -X which is most certainly not what you want. The bash behavior is dangerous because it makes people think it is safe, until it isn't, to type things like -? without quoting or escaping the glob.

[faho]

The behavior bash has here by default is what we called "passglob" in #2394.

In case a file exists that matches that glob, the filename is passed - so if you had a file named "-x" in the current directory, dot -x would be called. If it doesn't (like in most cases), the literal glob is passed through.

We came to the conclusion that this is the worst possible behavior - you don't know when it's safe to use, and it leads to e.g. for i in * needing an explicit check for the literal glob - something like if test "$i" = "*" && ! test -e "$i"; then continue; fi.

This can also be customized via the "failglob" or "nullglob" options.

Fish's current behavior is equivalent to "nullglob" with for, set and count and "failglob" everywhere else.

Personally I still feel that's the right compromise.

If there's anything to be done here, I'd prefer to remove the "?" glob altogether and instead find another way to generate more complicated existent filenames, since there's very little use for that one-char glob. In fact I can't find a single use of it in our codebase.

That would of course be a breaking change.

[samdmarshall]

if y'all want to categorize this more accurately by updating the title, I won't complain. I wasn't sure how to accurately describe this behavior beyond what I was seeing. I was expecting the ? in -? to be taken as a literal, and not interpreted. It seems highly unlikely to get any of the existing software that uses a -? for the equivalent of -h/--help to change their flag.

Maybe a compromise can be reached here for automatic handling where if there is a defined completion for the command to accept a -? flag then it gets interpreted as a literal, and not a glob? If not, at the very least a more informative error message from fish shell would be appreciated so that the user can modify their input accordingly.

[krader1961]

We definitely do not want to predicate treating -? literally on whether there is a command completion listing it as a short flag. The inconsistent behavior based on whether a completion was defined or not would drive people crazy.

We might want to consider special-casing that specific glob and always treating it as a literal rather than a glob. In my three decades of using UNIX I've never seen someone create a file name matching that specific glob other than by accident. I also agree with @faho. If we could go back in time to when fish was being created we should not implement the question-mark glob. It's used so seldom it would be better to implement it via some other mechanism such as a hypothetical expand_filename command that takes a regex (not a glob) and returns a list of matching file names. Unfortunately while we can, and perhaps should, implement such a command we cannot now change the meaning of an unquoted ? without creating a major release and even then doing so would be controversial.

[faho]

In my three decades of using UNIX I've never seen someone create a file name matching that specific glob other than by accident.

I've actually seen people recommend creating a file called "-i". For use with rm *. Yes, really.

[krader1961]

I've actually seen people recommend creating a file called "-i". For use with rm *.

That deserves a OMFG response. Long ago when I did UNIX tech support I would sometimes mutter under my breath "step slowly away from the keyboard" when a customer did something like that.

[samdmarshall]

¯\_(ツ)_/¯ I don't mind dealing with the work-around of escaping it; so long as the shell was to inform me what i should do either way. Adding verbosity to the existing error message to explain how to correctly escape it seems like an appropriate fix here and the entire usage debate and be avoided.

[ridiculousfish]

I like the idea of adding more smarts to the glob error text. Perhaps something like:

> curl http://www.example.org/?foo=bar
No matches for wildcard 'http://www.example.org/?foo=bar'. Did you mean to use \? instead of ?
fish: curl http://www.example.org/?foo=bar
                                  ^

where we would only show the last part if the glob is long and contains a ? character.

[krader1961]

Removing the RFC label since there seems to be consensus that simply improving the error message is sufficient.

[anordal]

Perhaps quoting is the best advice (for teaching a good habit when pasting strings with wildcards you may not be aware of).

> curl http://www.example.org/?foo=bar
fish: No matches for wildcard '?' in:
curl http://www.example.org/?foo=bar
                            ^
Tip: If you mean a literal question mark, remember to place it in quotes,
like this '?', such as by quoting the word it appears in.

[samdmarshall]

@anordal I like that approach a lot.

[cdCorey]

As a relatively new user of the Friendly Interactive Shell, I just want to add my support for removing this. I've been getting the error in @anordal's comment above from curl, and it took me awhile to figure out it was due to a "feature" of fish and not a bug in curl. This wildcard undoubtedly costs users a thousand times more time in annoyance/confusion than it saves the one or two people who might intentionally use it—and there's nothing "friendly" about that.

If it can't be removed entirely due to legacy concerns, is there at least a way for those of us who don't want it to disable it on our individual machines?

[anordal]

@cdCorey I feel with you, but

"feature" of fish

In case you might not know, we just copied the ? wildcard from bash – you should undoubtedly quote it in bash too. As others have said, the difference is in how we treat unmatched wildcards. The reasoning behind the "friendliness" of that can be summed up as not treacherously letting you do it the wrong way.

not a bug in curl

Right, I think the biggest problem with the current message is that it's not prefixed by fish: . My proposal would fix that.

disable it on individual machines?

Not recommended: Then you can't run others' scripts – won't be supported.

[anordal]

The curl case is also broken by #3878, so removing the ? wildcard would currently not suffice:

> curl http://localhost?a=b&c=d
No matches for wildcard 'http://localhost?a=b'.  (Tip: empty matches are allowed in 'set', 'count', 'for'.)
fish: curl http://localhost?a=b&c=d
           ^
Unsupported use of '='. In fish, please use 'set c d'.
fish: curl http://localhost?a=b&c=d
                                ^

[faho]

Note: There is one case where we use "?" as a wildcard - as an argument to case, e.g.

case '-??*'

to match grouped options in type.fish (and possibly others).

I feel like it would probably okay to allow this (as it's not a file-matching glob), and similar uses in string match too, since the main issue is with ? on the commandline and these uses need to be quoted anyway.

[anordal]

The one thing I use ? for on the commandline is ls /dev/sd? (lists block devices without their partitions). Are we contemplating a glob command again? Some say python -c 'import this' | grep implicit. Others say backward compatibility.

[anordal]

In any case, we have an error message, and it could probably be more beginner friendly. I have 2 commits here, with this result:

> curl http://localhost?a=b&c=d
fish: No matches for wildcard pattern 'http://localhost?a=b'.
(Tip: To avoid wildcarding, quote it. To allow no matches, expand it to a variable.)
curl http://localhost?a=b&c=d
     ^
fish: Unsupported use of '='. In fish, please use 'set c d'.
curl http://localhost?a=b&c=d
                          ^

• Commit 1: Move the fish: prefix. Needs consideration: This is for all errors we print, and the prefix is actually the name of the script, if applicable.
• Commit 2: Change the error message. Because of the length, I put a controversial newline in there.

RFC

[faho]

I have 2 commits here

@anordal: Submit a PR, then we can play around with it. At first glance, this seems fine to me.

[faho]

Unfortunately while we can, and perhaps should, implement such a command we cannot now change the meaning of an unquoted ? without creating a major release and even then doing so would be controversial.

@krader1961: Since we are doing fish 3.0, should we do this?

I've found two uses of "?" as a wildcard character in our codebase - in the "acidhub" and "sorin" prompts, and both are used with case. I suspect both of these could be replaced with * without any changes in behavior.

[krader1961]

@faho, I'd be okay with making "?" a non-glob character in 3.0 but we'll really need a glob command (or whatever we call it). There will no doubt be a few people who use it as a glob that will complain. But there will be far more people who are no longer surprised that this obscure glob character needs to be quoted or escaped.

[mqudsi]

One awesome benefit would be to no longer need to escape most URIs when interacting with commands like curl, wget, http, aria2c, etc.

Note that ZSH has (a plugin|an option) that automatically escapes glob characters under certain conditions (pate being one of them - perhaps our bracketed paste support can now make this possible) so that the behavior is more in-line with what a sure would intuitively suggest. But that's basically dark magic in the shell.

[faho]

One awesome benefit would be to no longer need to escape most URIs when interacting with commands like curl, wget, http, aria2c, etc.

Well..... not quite. "&" is still an issue, so you should still get into the habit of doing that.

Note that ZSH has (a plugin|an option) that automatically escapes glob characters under certain conditions (pate being one of them - perhaps our bracketed paste support can now make this possible)

We have discussed doing that, but, as you say

that's basically dark magic in the shell.

What we have implemented, is what I like referring to as "magic paste". When you are in a single-quoted context (i.e. there's an open single-quote), anything you paste will have both ' and \ escaped, so you can simply close the single-quote after.

E.g. enter curl ', paste the URL, press ' and enter.

[mqudsi]

I did not know that! This is quickly going to become my favorite feature.

[zanchey]

I always like to have a bit of data for these decisions (is any data better than no data? perhaps not) so I had a look through my shell history. In the fourteen years of history on my personal user profiles, I can see ten or so uses of ? as a glob - usually in cases like diff -ur pcre3-8.3?/debian, where I think a * would be a reasonable replacement.

[krader1961]

If we did this I think we want string match to continue to recognize * and ? as globs. Also, if at all possible it would be nice if case continued to recognize ? as a glob. Ideally this change would only stop treating ? as a glob when doing pathname expansion. And, finally, if at all possible inhibit pathname expansion in the context of a case statement. It should be safe to write constructs like the following without having to remember to quote the asterisk:

switch $var
case ?
   handle single letter
case abc
   do something
case def
   do something
case *
   do catchall
end