🐞 Directory.glob with recursive patterns (**/*.go) returns duplicates and is very slow

[vito]

What is the issue?

It looks like Buildkit returns weird results when you ReadDir with a glob like **/*.go. It actually seems to find everything recursively, but then trim each result to only have the first path segment. So for a/1.go a/2.go b/3.go it'll return a a b. We then recurse into a twice, which will also have this issue, so it quickly explodes into a huge number of results.

We could work around this issue by deduping, but it would probably be better to fix this upstream in Buildkit. It would be perfect if it just returned all the full paths, which it seems capable of doing. Then we wouldn't need to recurse ourselves. While we're there, maybe we could add support for exclude filters too (edit: interestingly it seems to support ! prefixes for exclusions - maybe we should be phasing out include: ["a"], exclude: ["b"] in favor of globs: ["a", "!b"]?).

cc @jedevc @TomChv for continuity from #5824 (comment)

Dagger version

v0.9.1+ (when the feature was added; not a regression)

Steps to reproduce

{
  git(url:"https://github.com/dagger/dagger") {
    branch(name:"main"){
      tree {
        glob(pattern:"**/dagger.json")
      }
    }
  }
}

Log output

{
  "data": {
    "git": {
      "branch": {
        "tree": {
          "glob": [
            "ci/dagger.json",
            "core/integration/testdata/modules/go/basic/dagger.json",
            "core/integration/testdata/modules/go/broken/dagger.json",
            "core/integration/testdata/modules/go/ifaces/dagger.json",
            "core/integration/testdata/modules/go/ifaces/impl/dagger.json",
            "core/integration/testdata/modules/go/ifaces/test/dagger.json",
            "core/integration/testdata/modules/go/namespacing/dagger.json",
            "core/integration/testdata/modules/go/namespacing/sub1/dagger.json",
            "core/integration/testdata/modules/go/namespacing/sub2/dagger.json",
            "core/integration/testdata/modules/go/basic/dagger.json",
            "core/integration/testdata/modules/go/broken/dagger.json",
            "core/integration/testdata/modules/go/ifaces/dagger.json",
            "core/integration/testdata/modules/go/ifaces/impl/dagger.json",
            "core/integration/testdata/modules/go/ifaces/test/dagger.json",
            "core/integration/testdata/modules/go/namespacing/dagger.json",
            "core/integration/testdata/modules/go/namespacing/sub1/dagger.json",
            "core/integration/testdata/modules/go/namespacing/sub2/dagger.json",
            "core/integration/testdata/modules/go/basic/dagger.json",
            "core/integration/testdata/modules/go/broken/dagger.json",
            "core/integration/testdata/modules/go/ifaces/dagger.json",
            "core/integration/testdata/modules/go/ifaces/impl/dagger.json",
            "core/integration/testdata/modules/go/ifaces/test/dagger.json",
            "core/integration/testdata/modules/go/namespacing/dagger.json",
            "core/integration/testdata/modules/go/namespacing/sub1/dagger.json",
            "core/integration/testdata/modules/go/namespacing/sub2/dagger.json",
            // you get the idea