enhance wildcard syntax to support matching hidden files (those beginning with a period)

[mamiu]

See problem described here:
http://stackoverflow.com/questions/24841723/how-to-use-recursive-wildcards-in-fish-shell-for-dotfiles-and-folders

[KamilaBorowska]

Wildcards are designed to be a simple API not designed for more complex searches. See http://stackoverflow.com/a/24844696/736054.

[faho]

So, to get the actual information on here and save people an extra click:

The issue is that recursive wildcards (or "globs") never match "hidden" files, i.e. ls -a **.txt never matches "subfolder/.haddock.txt".

The solution is to explicitly match dotfiles, like

ls **/.*.txt **.txt

however, this means that you'd also need to explicitly match dotfiles in the current directory, like

ls **/.*.txt **.txt .*.txt

This should test it nicely:

mkdir sub
touch .flounder.txt halibut.txt sub/haddock.txt sub/.hagfish.txt
ls **.txt # will print all non-hidden files
ls .*.txt # will print .flounder.txt
ls **.*.txt # won't match
ls **/.*.txt # will only match sub/.hagfish.txt
ls **.txt **/.*.txt .*.txt # will match all of them

(Also, I have now decided all examples should be fish)

[mamiu]

@faho:
Thanks for your detailed information and tests!

@ridiculousfish:
What do you think about the following approach?

One asterisk to get all non-hidden files and folders from the current directory:

    ls *

Two asterisks to get all non-hidden files and folders recursively:

    ls **

Three asterisks to get all files and folders recursively, including hidden ones:

    ls ***

(These examples use ls, but naturally it should work with all commands.)

[faho]

It's important to note that all of this clashes with the existing documentation. That says:

** matches any string of characters. This includes matching an empty string. The string may include the / character but does not need to.

There's no mention of hidden files being excluded. So the question is if it should do that, and for that the root question is "is it valuable to do something to just non-hidden files", and I'd argue it is.

Then the question becomes how to do something to hidden files, so you can nicely do something to both by specifying both.

Since I don't want to have new syntax for just that very special case, I'd like to see if the existing syntax can be adapted, and the most likely candidates for that are **.* and **/.*.

If **.* would be changed to only match hidden files/folders recursively (i.e. in the current directory and all subdirectories) the meaning of ** would need to be changed, since, if it matches any string, it would also match "fish/flounder" which means **.* would also match "fish/flounder.txt".

**/.* on the other hand, would only need to additionally match "./.flounder.txt" for it to now work with all my examples, and ** matching "." is completely in line with the documentation and IMHO also logical. (Edit: Nope, according to the doc the ** could also be empty, so it would also match dotfiles in "/")

So, TL;DR: Make **/.* match all hidden files and folders recursively (including current dir).

[ridiculousfish]

It seems like .?** ought to work.

[faho]

Why? That's "a dot, then something that's not a "/" and then maybe something else, including a "/"" or in regexp \\.[^/].*. That matches everything in hidden directories and all the hidden files in the current directory:

mkdir .fish
touch .fish/halibut .flounder catfish
echo .?**

.fish/halibut .flounder

[strannik19]

so far this is the most confusing thing for me with fish. if wildcards are supposed to be simple, why is it that rm ./* doesn't simply return all files in the current directory? not returning a subset of files in my directory is not expected and definitely not simple, as I need to figure out new ways to delete everything in current directory.

[braham-snyder]

@strannik19 I might agree with you to some extent, but I think backwards- and bash-compatibility mean that changing that wouldn't be a good idea (not that I'm an authority on the issue). Regardless, this helps: rm {.,}*long_suffix_I_dont_want_to_type_again.

For the recursive case, something along the lines of {**/,}{,.}*.txt (splitting that into two might be more human-readable) works, but of course it's still clumsy.

Also, it's probably worth noting that the stackoverflow question linked in the OP has been viewed over one thousand times as of today — I think an enhanced syntax here could improve fish quite a bit.

It seems like .?** ought to work.

Why? That's "a dot, then something that's not a "/" and then maybe something else, including a "/"" or in regexp \\.[^/].*. That matches everything in hidden directories and all the hidden files in the current directory.

He may have been alluding to how ? works in regexps.

[strannik19]

it would def improve fish, just because its really counterintuitive that * doesn't match everything, hence the people going to stackoverflow. plus there are other threads on SO around the same topic.
{.,}* is the best option really when you need to get everything at the moment, but it does feel very hacky (and slightly awkward to type :-) )

[Nnarol]

@strannik19 Changing the behavior of * compared to other shells would not make sense in my opinion. If * matched files starting with a dot, the whole concept of hidden files would be thrown out the window.
Probably the only thing to still respect the convention would be ls, and that's it.

{.,} is indeed hackish, because it is not globbing, but a shell expansion that happens before it (at least in Bash), and generates two globbing patterns using the preamble and postfix.
However, Bash's extended globbing syntax does accommodate this use case: ?(<pattern>), which is the same as <atomic_pattern>? in regex.
E.g.: echo ~/?(.)* would print all hidden and non-hidden files in your home directory.
I do not know whether fish supports this extended globbing scheme, but if it does, this is pretty straightforward to do.

[cjbassi]

It seems like there are some people (including me) who would prefer to have dotfiles globbed by default so I think a configuration option would work nicely in this case. In Zsh, setopt globdots enables dotfiles to be globbed with *. Would adding that configuration option to fish be allowed?

[faho]

Would adding that configuration option to fish be allowed?

It would not, as it would be quite against fish's philosophy.

In essence, adding this configuration option makes things worse, because it now means if you have code that is affected by it you need to consider the option (e.g. if you share scripts you need to make sure the option is the value you expect), instead of just writing things one way, always.

[xiruizhao]

(If you start with a dot, then it will suggest hidden directories.)

@ridiculousfish in #88 (comment)

1. But many dot files today are configuration files and there is no intention to hide them.
2. POSIX does not mention "hidden" files. They are called filenames beginning with a <period> ('.') (IEEE Std 1003.1TM-2017 p.2383)
3. ls does not mention "hidden" files. They are called entries starting with .(https://www.man7.org/linux/man-pages/man1/ls.1.html)

I'm fine with globbing/pattern matching being POSIX compliant. But for interactive usage, fish should allow interactive tab completion of dot files. Requiring to specify the leading . also prevents non-leading matching. For example, ig cannot expand to .gitignore, you have to input .giti to disambiguate.

[faho]

So, TL;DR: Make */. match all hidden files and folders recursively (including current dir).

This now works, presumably after #7222

[faho]

@yochem The reason this fails for you is because one of the globs doesn't match, and so fish errors out instead of running something that might be nonsensical (think ls *.foo - if that didn't match, it would run ls and print your entire current directory).

So, the way to do this is what the docs tell you: Save it in a variable and check it.

set -l activate {.,}*/bin/activate.fish
set -q activate[1]
and source $activate[1]

Which also neatly sidesteps a problem in your original code if more than one file matches (if you do source foo bar, that reads the file "foo" with $argv set to "bar").

If you have to do it in one line you can use a variable override:

activate={.,}*/bin/activate.fish source $activate[1]

[yochem]

Awesome, thank you very much for your detailed explanation and solution! :)