glob support in core

[boneskull]

I'm curious if anyone thinks adding glob support to core would be a good idea?

Globs (globspecs? what's the right term?) are natively supported in POSIX (?) shells, which means you can do this:

$ some-cli-app **/*.foo.js

This is expanded by the shell before execution of some-cli-app into a list of files matching the glob.

But we get in to trouble when our scripts (e.g., in package.json) use globs, because they will not be expanded, and the scripts will fail (or otherwise exhibit unexpected behavior) on Windows.

To support globs in a portable way, any CLI app would need to pull in glob or a similar module, and pass the glob as a quoted string in the arguments:

$ some-cli-app "**/*.foo.js"

(On Windows, you must use double-quotes above, but using single-quotes in POSIX shells means env variables [e.g., $FOO] will be treated as literals; env variables are another can of worms)

The utility of glob has been well-established, and it's used in enough CLI apps that it may be a good candidate for vendoring or some other implementation. I haven't looked at the glob source, so it's unclear whether vendoring would be the best route, but I do know it contains several other dependencies which would also need to be pulled in (right @iansu?).

Thoughts?

cc @isaacs

[iansu]

I do think this is a good idea but as you point out glob does have 6 dependencies, including minimatch so it's unclear to me how we would vendor it.

[coreyfarrell]

globs are not supported by bash v5 on Fedora 30. Running some-cli-app **/*.foo.js will produce the same result as some-cli-app */*.foo.js - the double-star is not supported.

As for pulling glob into the core I'm not sure. I think it might be more appropriate to first ask if minimatch functionality belongs in the core. If so I think it would be important to evaluate pros/cons of multiple modules with similar functionality to determine which one would be best for the core (minimatch and micromatch each have about 15 million downloads per week).

[ljharb]

Glob support would be great precisely because it’s not consistent across platforms, but is needed for things like git/npm/eslint ignore, eslint overrides, etc.

[jonschlinkert]

As often as globbing is used, and since most developers really don't realize how many edge cases and platform-specific issues need to be considered to produce a consistent experience, IMHO it would be great to add native support.

If Node.js does decide to include this functionality, I would be willing to put some time into helping (assuming that's of any interest). Either way, my recommendation is that you look at the matching and fs operations as completely separate chunks of logic.

• minimatch and picomatch are both focused on the former (just the matching)
• glob is focused on the latter (but is currently coupled to minimatch)

If it's decided that this functionality should be re-written directly in Node.js, I'd be happy to take a stab at contributing that logic, starting with just the matching. I think @isaacs has more experience and knowledge of the fs logic and cross-platform considerations than I do. But I'd be willing to help with that as well.

If you decide to use external libs for any of this, I would of course also be willing and interested in collaborating on making any adjustments, improvements, or changes to our libs that are necessary to make them meet your requirements.

[sindresorhus]

I would recommend also taking a look at fast-glob which is both faster and better maintained than glob.

// @mrmlnc

[ljharb]

Let's not throw subjective shade at other packages, though.

[jonschlinkert]

sorry just saw your comment @ljharb (after I did thumbs up)

[boneskull]

We're meeting tomorrow and this will be on the agenda. To everyone offering comments: please attend if you're available!

[boneskull]

@sindresorhus It sounds like you'd be in support of pulling fast-glob into core, then?

[jonschlinkert]

@boneskull I'd be in support of that decision too :)

[sindresorhus]

@ljharb It was not my intention to "throw a subjective shade". I'll be better at justifying my opinions in the future. I just wanted to mention an alternative.

Here are benchmarks comparing glob and fast-glob: https://gist.github.com/mrmlnc/f06246b197f53c356895fa35355a367c#file-fast_glob_benchmark_server-txt

By "better maintained", I meant more actively maintained. Poor word use on my part. glob has almost not had any real code commits in 3 years.

For context, I built globby (15M weekly downloads now) some years ago to provide a subjectively nicer API for glob with a Promise API, support for multiple patterns, negated patterns, etc. Then fast-glob came along which fixed a lot of bugs found in glob and I eventually switched globby over to use fast-glob. In my subjective opinion, fast-glob has a better API and codebase.

[sindresorhus]

It sounds like you'd be in support of pulling fast-glob into core, then?

Yes

[boneskull]

OK, we discussed this at today's meeting

• Both glob and the underlying matcher should be available for consumption
• To start, let's create a PR which adds matching support (a la minimatch or micromatch)
• Suggested API: path.match()
• "gitignore-style" matching is also widely used, and this should be a use case
• @iansu and/or myself will provide a comparison of micromatch vs. minimatch vs. bash vs. gitignore to find the intersection of features; we can narrow down an API from there
• After this, there are several options w/r/t adding glob support to fs, and we can cross that bridge later

gimme feedback