path/filepath: Glob should support ** for zero or more directories

[ascarter]

Go version 1.4.2
Mac OS X 10.10

Example:

package main

import "fmt"
import "path/filepath"
import "os"

func main() {
    files, err := filepath.Glob("/usr/local/go/src/**/*.go")
    if err != nil {
            fmt.Print(err)
            os.Exit(1)
    }
    fmt.Printf("files: %d\n", len(files))
    for _, f := range files {
        fmt.Println(f)
    }
}

Expected:

% ls /usr/local/go/src/**/*.go | wc -l
    1633

Actual:

files: 732

It seems that ** is equivalent to *. The extended ** pattern is common in shells and is supported in Rust and Java for example.

[ascarter]

For reference, here is Rust's implementation. I think these rules would be a good description of how Go's filepath.Glob should work:

https://github.com/rust-lang/glob/blob/master/src/lib.rs#L369

[ianlancetaylor]

For reference, this is supported by bash when invoked as "bash -O globstar" or after running "shopt -s globstar".

[nodirt]

OK to implement? @adg

[nodirt]

@bradfitz for opinion

[ianlancetaylor]

I think this is OK to implement.

[adg]

Yep, I agree.

[gopherbot]

CL https://golang.org/cl/16280 mentions this issue.

[dotcomputercraft]

all, I wanted to check in to see if this issue (#11862) was fixed. Please let me know status of this ticket. Is there a work around to this problem? Talk to you soon. Thanks in advance.

[nodirt]

Status: currently I'm not working on this. Feel free to unassign.

[rsc]

I certainly see the utility here but adding ** has various knock-on effects that may not be obvious. For example, * matches symlinks to other directories. ** cannot, at least not without extra work to avoid file system cycles. Also it's not obvious this should be considered a backwards compatible change, both for Glob and for Match. And you'd probably want to do path.Match as well. There's a lot to consider here, even though it looks trivial.

[slimsag]

@rsc closed the most recent CL for this with the comment:

I really don't think we should do any of this. This is much less understandable than the current code. If you need a very fast, very complex filepath.Glob, it is of course possible to write one outside the standard library.

Should this issue remain open?

[rsc]

This issue does remain open. What's confusing is that GitHub has linked just above your comment to issue Shopify/themekit#102, which is closed.

[slimsag]

@rsc that is because someone linked to it in a Markdown link / comment: Shopify/themekit#102 (comment)

What can be done here? A better implementation? It's not clear to me.

[extemporalgenome]

I think it's useful, but I wouldn't consider it backwards compatible. ** already works and does something (which is the same something as a single star).

[ascarter]

@extemporalgenome At best, I'd argue that ** is undefined and it just happens to do the same thing as *. I don't think this should represent an issue in regards to backwards compatibility.

[ascarter]

@slimsag the shopify issue I think is different. They chose to walk the tree and that solved their case - I don't think that applies to the general problem. I think @rsc was saying that the solution here is non trivial (and it's not just filepath.Glob but also path.Match).

I'd like it to stay on the list. Coming from other languages, I think this is a pretty common feature for glob syntax.

Note that the downside of attempting to write your own implementation is that I think you would wind up re-implementing glob entirely. I'm trying to figure out how to solve this for http://github.com/jstemmer/gotags since it won't be solved in the standard library anytime soon.

[extemporalgenome]

@ascarter * is very well defined as matching any sequence (including an empty sequence) of non-separator characters. Thus ** is equally well defined as matching zero or more non-separator characters, followed by a match of zero characters on the tail of the same filename.

The proposed special-casing of ** will certainly not match less than it does now, but may match more, and the question is whether existing programs could become broken through what constitutes a change in behavior.

[mattn]

I'm thinking this feature should be provided from third-party products not core.
for example, you can use https://github.com/mattn/go-zglob

[coolaj86]

Stopgap Workaround

This may not be a perfect solution, but weighing in at only 60 LoC it's a good start for those willing to use something outside of the standard library:

https://stackoverflow.com/a/40181899/151312

[jba]

In general I agree that a function's documentation should say what it does and not what it doesn't do. But using ** is a pitfall, and we do document pitfalls, as in this lengthly caveat from fs.Sub:

Note that Sub(os.DirFS("/"), "prefix") is equivalent to os.DirFS("/prefix") and that neither of them guarantees to avoid operating system accesses outside "/prefix", because the implementation of os.DirFS does not check for symbolic links inside "/prefix" that point to other directories. That is, os.DirFS is not a general substitute for a chroot-style security mechanism, and Sub does not change that fact.

We could word the doc so that it describes a consequence of the function's behavior, instead of describing non-behavior. For example, "Note that multiple consecutive *s behave like a single *".

[soypat]

Could this be needed to match only main.go, hey.go files in the following case?

- unwanted.go
- dir1
| - main.go
| - dir2
   | - hey.go

• Glob("dir1/*.go") only matches main.go
• Glob("dir1/*/*.go") only matches hey.go
• Glob("*.go") matches them all, including the unwanted unwanted.go

[mih-kopylov]

It's a shame such an famous Glob feature not implemented in Go for 7 years.

@adg @rsc is there a chance to prioritize it please?

[adg]

@mih-kopylov There are some concerns that Russ outlined in an earlier comment and also there are compatibility promises that we must heed. So if someone wants to address this issue they need should probably write it up as a proposal that addresses all these potential issues.

In the meantime, if there is a (seemingly, from my brief research) mature doublestar package that provides a glob that handles **. It doesn't have any dependencies and has extensive unit tests. Give it a shot.