feat: Swap out Globby for custom globbing solution.

[nzakas]

Removes globby due to numerous issues with ignoring files and returning the correct files.

Fixes #16354
Fixes #16340
Fixes #16300

Prerequisites checklist

• I have read the contributing guidelines.

What is the purpose of this pull request? (put an "X" next to an item)

[ ] Documentation update
[ ] Bug fix (template)
[ ] New rule (template)
[ ] Changes an existing rule (template)
[ ] Add autofix to a rule
[ ] Add a CLI option
[x] Add something to the core
[ ] Other, please explain:

What changes did you make? (Give an overview)

I removed globby and replaced it with a custom solution built upon @nodelib/fs.walk, which was used by fast-glob to walk the directory and look for matching filenames. Instead of forcing globby to reinterpret ignores, this solution uses FlatConfigArray#isFileIgnored() as part of the traversal so it exactly mirrors the ignoring behavior inside of a config array. This should fix a lot of the outstanding issues related to ignoring certain files.

Is there anything you'd like reviewers to focus on?

If you've seen weird behavior, please check out this branch and try it out to see if it fixes the issue. I'm still adding tests to this PR to make sure everything is fixed.

Also, I marked as "feat" because it seemed worthy of a minor release, but totally fine to mark as "fix" if others feel differently.

[netlify]

✅ Deploy Preview for docs-eslint ready!

Name	Link
🔨 Latest commit	5e699df
🔍 Latest deploy log	https://app.netlify.com/sites/docs-eslint/deploys/6344795953957200083449d0
😎 Deploy Preview	https://deploy-preview-16369--docs-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[nzakas]

deepFilter determines whether or not we descend into a directory. So, we want to cut off things like node_modules here to avoid the traversal.

[nzakas]

entryFilter determines which entries will show up in the results, so I filter out directories and check patterns here.

[nzakas]

Since we are removing ignorePath, not a big deal to fix this test up to make sure it works correctly.

[mdjermanovic]

I'm not sure why timeouts happen only on Node 18, but what's common for all failing tests is that only flat config tests are failing and they're all testing behavior with unmatched patterns, so perhaps this indicates that handling unmatched patterns should be optimized in eslint-helpers.js.

[nzakas]

Ah good call! I've optimized the search so that it's excluding more directories by default and that's making some of the tests run noticeably faster locally. I'll look more deeply into the unmatched branch.

[mdjermanovic]

For example, this test:

eslint/tests/lib/cli.js

Lines 494 to 507 in 720ff75

	it(`should throw an error on unmatched glob pattern with configType:${configType}`, async () => {
	let filePath = getFixturePath("unmatched-patterns");
	const globPattern = "unmatched*.js";
	if (useFlatConfig) {
	filePath = filePath.replace(/\\/gu, "/");
	}
	await stdAssert.rejects(async () => {
	await cli.execute(`"${filePath}/${globPattern}"`, null, useFlatConfig);
	}, new Error(`No files matching '${filePath}/${globPattern}' were found.`));
	});
	});

In eslintrc mode, it traverses only fixtures/unmatched-patterns directory.

In flat config mode, it traverses the whole fixtures, and once again the whole fixtures.

Maybe we could keep the search logic from eslintrc?

eslintrc (ESLint class & FileEnumerator):

• Runs multiple searches, one for each pattern. A downside is that it may traverse the same directories multiple times if the patterns have intersections, but that should be rare.
• Starts search from glob parent directory.
• Determines whether the search matches ignored files during the search.

flat config (FlatESLint class & eslint-helpers):

• Runs one search with all patterns at once. A downside is that it doesn't know if each of the patterns has matched something (Bug: [new config system] unmatched individual patterns are not reported #16275).
• Always starts search from cwd. It's less optimal than starting from the glob parent, and patterns like ../*.js don't work.
• If search returned nothing, runs another search to see if it matches only ignored files.

[nzakas]

So interestingly when I log through all off the filtering, it appears that the traversal completes and then just hangs. Whatever cleanup is being done, either from the package or from Node.js, is clearly getting stuck.

Basically, if entryFilter never returns true, we get this behavior. I can't figure out where it's hanging as it seems to be in between callbacks and if I try to use the debugger it steps through just fine.

At this point I'm out of ideas aside from manually creating my own filesystem walker. ☹️

[mdjermanovic]

Maybe fswalk.walkStream hangs, would it work if we use doFsWalk instead? (we're forcing an early exit with found = true anyways).

[nzakas]

After I napped I also realized that walkStream was the likely culprit. 👍

It looks like it's not an issue with streams per se, but rather an issue with using async interation via the for await-of loop. It seems like the stream isn't properly closed when we exit early from the loop. If I use a stream without async iteration then it works fine.

[nzakas]

Okay, now other errors are happening, that's progress. :) That's all for today.

[nzakas]

Okay, I believe this should work now. Undoubtedly there are perf improvements that can be made, but I think those should be handled in another PR. From my local testing, this code looks pretty close to what we had with globby.

[mdjermanovic]

Starting from cwd makes it impossible to lint files outside cwd.

For example, do cd lib and then npx eslint "../*.js".

Expected result is to lint Makefile.js, karma.conf.js, and other files in the root of the eslint/eslint project. It used to work before this change, and it works with eslintrc.

Actual result:

Oops! Something went wrong! :(

ESLint: 8.24.0

No files matching the pattern "../*.js" were found.
Please check for typing mistakes in the pattern.

[mdjermanovic]

Issue #16413

[mdjermanovic]

File should be ignored if any of the ancestor directories are ignored.

For example:

.
├── foo/
│   └── bar/
│       └── a.js
└── eslint.config.js

// eslint.config.js
module.exports = [{
    ignores: ["foo"]
}];

eslint foo/bar/a.js -f tap

Expected result:

TAP version 13
1..1
ok 1 - D:\tmp\eslint-ignore-patch\foo\bar\a.js
  ---
  message: >-
    File ignored because of a matching ignore pattern. Use "--no-ignore" to
    override.
  severity: warning
  data:
    line: 0
    column: 0
    ruleId: ''
  ...

Actual result:

TAP version 13
1..1
ok 1 - D:\tmp\eslint-ignore-patch\foo\bar\a.js

(with globs, it works as expected because traversal doesn't descend into ignored foo)

Note: eslint "foo/**/*.js" works as expected because it doesn't enter ignored foo. But cd foo && eslint "bar/*.js" doesn't work as expected as it enters foo/bar although that directory should be ignored because it's in an ignored directory foo.

Edit: #16414

[mdjermanovic]

It should be possible to unignore a directory.

For example:

.
├── foo/
│   └── bar/
│       └── a.js
└── eslint.config.js

// eslint.config.js
module.exports = [{
    ignores: ["foo/*", "!foo/bar"]
}];

eslint "foo/**/*.js" -f tap

Expected result:

TAP version 13
1..1
ok 1 - D:\tmp\eslint-ignore-patch\foo\bar\a.js

Actual result:

Oops! Something went wrong! :(

ESLint: 8.24.0

You are linting "foo/**/*.js", but all of the files matching the glob pattern "foo/**/*.js" are ignored.

If you don't want to lint these files, remove the pattern "foo/**/*.js" from the list of arguments passed to ESLint.

If you do want to lint these files, try the following solutions:

* Check your .eslintignore file, or the eslintIgnore property in package.json, to ensure that the files are not configured to be ignored.
* Explicitly list the files from this glob that you'd like to lint on the command-line, rather than providing a glob as an argument.

Edit: #16415

[mdjermanovic]

If we want .gitignore-like behavior, then the last pattern that matches the file should determine whether the file is ignored (unless the file is under an ignored directory; in that case, the file is ignored).

For example:

.
├── a.js
└── eslint.config.js

// eslint.config.js
module.exports = [{
    ignores: [
        "*.js",
        "!a*.js",
        "a.js"
    ]
}];

eslint a.js -f tap

Expected result:

TAP version 13
1..1
ok 1 - D:\tmp\eslint-ignore-patch\a.js
  ---
  message: >-
    File ignored because of a matching ignore pattern. Use "--no-ignore" to
    override.
  severity: warning
  data:
    line: 0
    column: 0
    ruleId: ''
  ...

Actual result:

TAP version 13
1..1
ok 1 - D:\tmp\eslint-ignore-patch\a.js

Edit: #16416

[nzakas]

Thanks for the detailed review. It looks like each of the problems you've mentioned actually already exists with the globby implementation (on main), and because this PR was just intended to replace globby and not solve every bug in the current implementation, I'd like to suggest that we merge this as-is and ask you to open separate issues for each of those problems.

The three bugs this PR fixes are more happy side effects of swapping out globby rather than intentional. :)

What do you think?

[mdjermanovic]

I was mostly concerned about #16369 (comment), but that's not a common use case and we'll have time to fix that now.

[mdjermanovic]

LGTM, thanks!

I'll open issues about the problems mentioned here.

[silverwind]

Found one regression: /bin/**.js no longer matches bin/foo.js, but did before this PR. I've updated my config to just /bin/ which works.