perf: improve worker count calculation for "auto" concurrency

[fasttime]

Prerequisites checklist

• I have read the contributing guidelines.

What is the purpose of this pull request? (put an "X" next to an item)

[ ] Documentation update
[ ] Bug fix (template)
[ ] New rule (template)
[ ] Changes an existing rule (template)
[ ] Add autofix to a rule
[ ] Add a CLI option
[ ] Add something to the core
[x] Other, please explain:

Improve worker count calculation for "auto" concurrency.

refs #20040 (comment)

What changes did you make? (Give an overview)

This pull request makes some improvements to the thread count calculation that is done when the concurrency is "auto".

1. Ignored files are no longer considered in the thread count calculation. Determining if a file is ignored can be done very efficiently now because config arrays are pre-loaded for all files when the files are enumerated.
2. When using the cache with the default "metadata" strategy, cached files with unchanged results are no longer considered in the thread count calculation. This means that only files which need to be read, parsed and processed by the linter will be counted. If all cached results are valid, multithreading will not be used.
3. Increased the AUTO_FILES_PER_WORKER constant from 35 to 50. This constant value affects the thread count calculation when only a few files are linted. The most visible effect is that multithreading will be no longer turned on when linting less than 51 files (this is an empiric value and possibly still too low on some machines). I realized that some of the projects I used in my early tests were biased by ignored files. After repeating the tests with ignored files not counted, the larger value seems more appropriate.

Is there anything you'd like reviewers to focus on?

The additional time to count files that need processing (excluding ignored and cached files) is printed in a debug line, for example:

  eslint:eslint 351 file(s) to process counted in 18 ms +19ms

In practice, the effective overhead seems to be even lower, possibly because of cross-thread caching by the OS, which results in subsequent access to file metadata being faster.

Benchmarks

I tested the implementation on a fork of the OpenUI5 repo which contains 12,500 lintable files and uses the --cache option: https://github.com/mauriciolauffer/openui5/tree/eslint-v9.

These are the results when the cache file is outdated:

> time DEBUG=eslint:eslint eslint ./src --quiet --cache --concurrency=auto
  eslint:eslint Using config loader LegacyConfigLoader +0ms
  eslint:eslint Using file patterns: ./src +0ms
  eslint:eslint 12500 file(s) found in 4009 ms +4s
  eslint:eslint 151 file(s) to process counted in 3 ms +3ms
  eslint:eslint Linting using 4 worker thread(s). +0ms
DEBUG=eslint:eslint eslint ./src --quiet --cache --concurrency=auto  139.43s user 8.25s system 526% cpu 28.039 total

It takes just 3 milliseconds to determine that the maximum number of threads (i.e. 4 for an 8 core CPU) should be used.

Now, here are the results when the cache file is present and all cached results are valid:

> time DEBUG=eslint:eslint eslint ./src --quiet --cache --concurrency=auto 
  eslint:eslint Using config loader LegacyConfigLoader +0ms
  eslint:eslint Using file patterns: ./src +1ms
  eslint:eslint 12500 file(s) found in 3987 ms +4s
  eslint:eslint 0 file(s) to process counted in 56 ms +56ms
  eslint:eslint Linting in single-thread mode. +0ms
DEBUG=eslint:eslint eslint ./src --quiet --cache --concurrency=auto  4.46s user 0.36s system 108% cpu 4.452 total

In this case, it takes 56 milliseconds to determine that no concurrency should be used. The process is slower compared to the previous run because a much larger number of file metadata is checked, until it is determined that there aren't enough files to process for multithread mode.

I also repeated the same tests on a slower Windows machine with 16 cores, with comparable results:

No files cached

> $env:DEBUG="eslint:eslint"; Measure-Command { eslint ./src --quiet --cache --concurrency=auto }
  eslint:eslint Using config loader LegacyConfigLoader +0ms
  eslint:eslint Using file patterns: ./src +3ms
  eslint:eslint 12500 file(s) found in 8384 ms +8s
  eslint:eslint 351 file(s) to process counted in 8 ms +9ms
  eslint:eslint Linting using 8 worker thread(s). +0ms


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 39
Milliseconds      : 356
Ticks             : 393564490
TotalDays         : 0.000455514456018518
TotalHours        : 0.0109323469444444
TotalMinutes      : 0.655940816666667
TotalSeconds      : 39.356449
TotalMilliseconds : 39356.449

All files cached

> $env:DEBUG="eslint:eslint"; Measure-Command { eslint ./src --quiet --cache --concurrency=auto }
  eslint:eslint Using config loader LegacyConfigLoader +0ms
  eslint:eslint Using file patterns: ./src +2ms
  eslint:eslint 12500 file(s) found in 8150 ms +8s
  eslint:eslint 0 file(s) to process counted in 197 ms +199ms
  eslint:eslint Linting in single-thread mode. +0ms


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 10
Milliseconds      : 574
Ticks             : 105746003
TotalDays         : 0.000122391207175926
TotalHours        : 0.00293738897222222
TotalMinutes      : 0.176243338333333
TotalSeconds      : 10.5746003
TotalMilliseconds : 10574.6003

As already mentioned, the actual time overhead seems to be even lower than the figures suggests because of the reduced time for other operation, to the extent that there is no noticeable difference between --concurrency=auto and --concurrency=off when most of the files are cached:

> hyperfine "./bin/eslint.js --cache --concurrency=auto" -n 5 
Benchmark 1: 5
  Time (mean ± σ):     577.8 ms ±   9.5 ms    [User: 762.1 ms, System: 65.9 ms]
  Range (min … max):   562.8 ms … 597.1 ms    10 runs
 
> hyperfine "./bin/eslint.js --cache --concurrency=off" -n 5 
Benchmark 1: 5
  Time (mean ± σ):     582.3 ms ±  11.0 ms    [User: 769.0 ms, System: 66.4 ms]
  Range (min … max):   558.4 ms … 600.4 ms    10 runs

[netlify]

✅ Deploy Preview for docs-eslint canceled.

Name	Link
🔨 Latest commit	16d8260
🔍 Latest deploy log	https://app.netlify.com/projects/docs-eslint/deploys/68d0d9fe3220040008b2fd0d

[fasttime]

This is the same as the debug utility's main export, but it makes sure that the %t formatter is registered. I'm using this to format differences of high-resolution timer values (process.hrtime.bigint()). I'm not sure if this should be made more explicit with a different name, a comment, etc.

[fasttime]

I moved these tests here so they will be repeated with and without the v10_config_lookup_from_file flag. This is because calculateWorkerCount now uses the config loader.

[fasttime]

This was causing test failures because of ESLint objects being created from a mocked version of lib/eslint/eslint.js. With noPreserveCache set, a mocked version of the lib/eslint/eslint.js module was being kept in the require cache, with its own version of privateMembers, and that did no longer match the privateMembers in the scope of the caclulateWorkerCount function, which is imported from the original module.

[fasttime]

As above, this was causing inconsistencies in Node.js' require cache, resulting in surprising test failures.

[github-actions]

Hi everyone, it looks like we lost track of this pull request. Please review and see what the next steps are. This pull request will auto-close in 7 days without an update.

[fasttime]

This is ready for review.

[mdjermanovic]

Just a few minor suggestions, then LGTM.

[mdjermanovic]

LGTM, thanks! Would like @nzakas to review before merging.

[nzakas]

LGTM. Just waiting for the patch release window to close.