feat: Rule Performance Statistics for flat ESLint (#17850)

* Add stats option * Remove resetTimes, stats per file * Add suggestions from review * Update types & docs * Update times description * Add Stats Data page * Fix file verification issues * Use named exports * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Add suggestions and new data * Update lib/shared/stats.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update lib/shared/stats.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update lib/shared/types.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/integrate/nodejs-api.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/extend/stats.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update lib/linter/linter.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update lib/linter/linter.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update lib/linter/linter.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update lib/linter/linter.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update lib/linter/linter.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update tests/lib/eslint/eslint.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update tests/fixtures/stats-example/file-to-fix.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/integrate/nodejs-api.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/use/command-line-interface.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Fix missing bracket * Update docs/src/use/command-line-interface.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update stats tests * Update tests/lib/eslint/eslint.js Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Update docs/src/integrate/nodejs-api.md Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> * Fix missing properties * Remove trailing spaces * Update nodejs-api.md * Update docs/src/extend/stats.md Co-authored-by: Nicholas C. Zakas <nicholas@humanwhocodes.com> * Update docs/src/extend/stats.md Co-authored-by: Nicholas C. Zakas <nicholas@humanwhocodes.com> * Update docs/src/extend/stats.md Co-authored-by: Nicholas C. Zakas <nicholas@humanwhocodes.com> * Update docs/src/extend/stats.md Co-authored-by: Nicholas C. Zakas <nicholas@humanwhocodes.com> * Update stats.md * Add more docs --------- Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com> Co-authored-by: Nicholas C. Zakas <nicholas@humanwhocodes.com>
eslint · Mar 25, 2024 · de40874 · de40874
1 parent d85c436
commit de40874
Show file tree

Hide file tree

Showing 16 changed files with 521 additions and 19 deletions.
diff --git a/docs/src/extend/custom-formatters.md b/docs/src/extend/custom-formatters.md
@@ -98,6 +98,7 @@ Each object in the `results` array is a `result` object. Each `result` object co
 * **messages**: An array of [`message`](#the-message-object) objects. See below for more info about messages.
 * **errorCount**: The number of errors for the given file.
 * **warningCount**: The number of warnings for the given file.
+* **stats**: The optional [`stats`](./stats#-stats-type) object that only exists when the `stats` option is used.
 * **source**: The source code for the given file. This property is omitted if this file has no errors/warnings or if the `output` property is present.
 * **output**: The source code for the given file with as many fixes applied as possible. This property is omitted if no fix is available.
 

diff --git a/docs/src/extend/custom-rules.md b/docs/src/extend/custom-rules.md
@@ -938,3 +938,5 @@ quotes |    18.066 |   100.0%
 ```
 
 To see a longer list of results (more than 10), set the environment variable to another value such as `TIMING=50` or `TIMING=all`.
+
+For more granular timing information (per file per rule), use the [`stats`](./stats) option instead.
diff --git a/docs/src/extend/stats.md b/docs/src/extend/stats.md
@@ -0,0 +1,139 @@
+---
+title: Stats Data
+eleventyNavigation:
+    key: stats data
+    parent: extend eslint
+    title: Stats Data
+    order: 6
+---
+
+While an analysis of the overall rule performance for an ESLint run can be carried out by setting the [TIMING](./custom-rules#profile-rule-performance) environment variable, it can sometimes be useful to acquire more *granular* timing data (lint time per file per rule) or collect other measures of interest. In particular, when developing new [custom plugins](./plugins) and evaluating/benchmarking new languages or rule sets. For these use cases, you can optionally collect runtime statistics from ESLint.
+
+## Enable stats collection
+
+To enable collection of statistics, you can either:
+
+1. Use the `--stats` CLI option. This will pass the stats data into the formatter used to output results from ESLint. (Note: not all formatters output stats data.)
+1. Set `stats: true` as an option on the `ESLint` constructor.
+
+Enabling stats data adds a new `stats` key to each [LintResult](../integrate/nodejs-api#-lintresult-type) object containing data such as parse times, fix times, lint times per rule.
+
+As such, it is not available via stdout but made easily ingestible via a formatter using the CLI or via the Node.js API to cater to your specific needs.
+
+## ◆ Stats type
+
+The `Stats` value is the timing information of each lint run. The `stats` property of the [LintResult](../integrate/nodejs-api#-lintresult-type) type contains it. It has the following properties:
+
+* `fixPasses` (`number`)<br>
+  The number of times ESLint has applied at least one fix after linting.
+* `times` (`{ passes: TimePass[] }`)<br>
+  The times spent on (parsing, fixing, linting) a file, where the linting refers to the timing information for each rule.
+    * `TimePass` (`{ parse: ParseTime, rules?: Record<string, RuleTime>, fix: FixTime, total: number }`)<br>
+    An object containing the times spent on (parsing, fixing, linting)
+        * `ParseTime` (`{ total: number }`)<br>
+          The total time that is spent when parsing a file.
+        * `RuleTime` (`{ total: number }`)<be>
+          The total time that is spent on a rule.
+        * `FixTime` (`{ total: number }`)<be>
+          The total time that is spent on applying fixes to the code.
+
+### CLI usage
+
+Let's consider the following example:
+
+```js [file-to-fix.js]
+/*eslint no-regex-spaces: "error", wrap-regex: "error"*/
+
+function a() {
+    return / foo/.test("bar");
+}
+```
+
+Run ESLint with `--stats` and output to JSON via the built-in [`json` formatter](../use/formatters/):
+
+```bash
+npx eslint file-to-fix.js --fix --stats -f json
+```
+
+This yields the following `stats` entry as part of the formatted lint results object:
+
+```json
+{
+    "times": {
+        "passes": [
+            {
+                "parse": {
+                    "total": 3.975959
+                },
+                "rules": {
+                    "no-regex-spaces": {
+                        "total": 0.160792
+                    },
+                    "wrap-regex": {
+                        "total": 0.422626
+                    }
+                },
+                "fix": {
+                    "total": 0.080208
+                },
+                "total": 12.765959
+            },
+            {
+                "parse": {
+                    "total": 0.623542
+                },
+                "rules": {
+                    "no-regex-spaces": {
+                        "total": 0.043084
+                    },
+                    "wrap-regex": {
+                        "total": 0.007959
+                    }
+                },
+                "fix": {
+                    "total": 0
+                },
+                "total": 1.148875
+            }
+        ]
+    },
+    "fixPasses": 1
+}
+```
+
+Note, that for the simple example above, the sum of all rule times should be directly comparable to the first column of the TIMING output. Running the same command with `TIMING=all`, you can verify this:
+
+```bash
+$ TIMING=all npx eslint file-to-fix.js --fix --stats -f json
+...
+Rule            | Time (ms) | Relative
+:---------------|----------:|--------:
+wrap-regex      |     0.431 |    67.9%
+no-regex-spaces |     0.204 |    32.1%
+```
+
+### API Usage
+
+You can achieve the same thing using the Node.js API by passing`stats: true` as an option to the `ESLint` constructor. For example:
+
+```js
+const { ESLint } = require("eslint");
+
+(async function main() {
+    // 1. Create an instance.
+    const eslint = new ESLint({ stats: true, fix: true });
+
+    // 2. Lint files.
+    const results = await eslint.lintFiles(["file-to-fix.js"]);
+
+    // 3. Format the results.
+    const formatter = await eslint.loadFormatter("json");
+    const resultText = formatter.format(results);
+
+    // 4. Output it.
+    console.log(resultText);
+})().catch((error) => {
+    process.exitCode = 1;
+    console.error(error);
+});
+```
diff --git a/docs/src/integrate/nodejs-api.md b/docs/src/integrate/nodejs-api.md
@@ -150,6 +150,8 @@ The `ESLint` constructor takes an `options` object. If you omit the `options` ob
   Default is `null`. The plugin implementations that ESLint uses for the `plugins` setting of your configuration. This is a map-like object. Those keys are plugin IDs and each value is implementation.
 * `options.ruleFilter` (`({ruleId: string, severity: number}) => boolean`)<br>
   Default is `() => true`. A predicate function that filters rules to be run. This function is called with an object containing `ruleId` and `severity`, and returns `true` if the rule should be run.
+* `options.stats` (`boolean`)<br>
+  Default is `false`. When set to `true`, additional statistics are added to the lint results (see [Stats type](../extend/stats#-stats-type)).
 
 ##### Autofix
 
@@ -367,6 +369,8 @@ The `LintResult` value is the information of the linting result of each file. Th
   The modified source code text. This property is undefined if any fixable messages didn't exist.
 * `source` (`string | undefined`)<br>
   The original source code text. This property is undefined if any messages didn't exist or the `output` property exists.
+* `stats` (`Stats | undefined`)<br>
+  The [Stats](../extend/stats#-stats-type) object. This contains the lint performance statistics collected with the `stats` option.
 * `usedDeprecatedRules` (`{ ruleId: string; replacedBy: string[] }[]`)<br>
   The information about the deprecated rules that were used to check this file.
 
@@ -715,6 +719,14 @@ const Linter = require("eslint").Linter;
 Linter.version; // => '9.0.0'
 ```
 
+### Linter#getTimes()
+
+This method is used to get the times spent on (parsing, fixing, linting) a file. See `times` property of the [Stats](../extend/stats#-stats-type) object.
+
+### Linter#getFixPassCount()
+
+This method is used to get the number of autofix passes made. See `fixPasses` property of the [Stats](../extend/stats#-stats-type) object.
+
 ---
 
 ## RuleTester

diff --git a/docs/src/use/command-line-interface.md b/docs/src/use/command-line-interface.md
@@ -126,6 +126,7 @@ Miscellaneous:
   -h, --help                      Show help
   -v, --version                   Output the version number
   --print-config path::String     Print the configuration for the given file
+  --stats                         Add statistics to the lint report - default: false
 ```
 
 ### Basic Configuration
@@ -811,6 +812,20 @@ This option outputs the configuration to be used for the file passed. When prese
 npx eslint --print-config file.js
 ```
 
+#### `--stats`
+
+This option adds a series of detailed performance statistics (see [Stats type](../extend/stats#-stats-type)) such as the *parse*-, *fix*- and *lint*-times (time per rule) to [`result`](../extend/custom-formatters#the-result-object) objects that are passed to the formatter (see [Stats CLI usage](../extend/stats#cli-usage)).
+
+* **Argument Type**: No argument.
+
+This option is intended for use with custom formatters that display statistics. It can also be used with the built-in `json` formatter.
+
+##### `--stats` example
+
+```shell
+npx eslint --stats --format json file.js
+```
+
 ## Exit Codes
 
 When linting files, ESLint exits with one of the following exit codes:

diff --git a/lib/cli.js b/lib/cli.js
@@ -131,6 +131,7 @@ async function translateOptions({
     resolvePluginsRelativeTo,
     rule,
     rulesdir,
+    stats,
     warnIgnored,
     passOnNoPatterns,
     maxWarnings
@@ -222,6 +223,7 @@ async function translateOptions({
 
     if (configType === "flat") {
         options.ignorePatterns = ignorePattern;
+        options.stats = stats;
         options.warnIgnored = warnIgnored;
 
         /*

diff --git a/lib/eslint/eslint-helpers.js b/lib/eslint/eslint-helpers.js
@@ -685,6 +685,7 @@ function processOptions({
     overrideConfig = null,
     overrideConfigFile = null,
     plugins = {},
+    stats = false,
     warnIgnored = true,
     passOnNoPatterns = false,
     ruleFilter = () => true,
@@ -791,6 +792,9 @@ function processOptions({
     if (Array.isArray(plugins)) {
         errors.push("'plugins' doesn't add plugins to configuration to load. Please use the 'overrideConfig.plugins' option instead.");
     }
+    if (typeof stats !== "boolean") {
+        errors.push("'stats' must be a boolean.");
+    }
     if (typeof warnIgnored !== "boolean") {
         errors.push("'warnIgnored' must be a boolean.");
     }
@@ -818,6 +822,7 @@ function processOptions({
         globInputPaths,
         ignore,
         ignorePatterns,
+        stats,
         passOnNoPatterns,
         warnIgnored,
         ruleFilter

diff --git a/lib/eslint/eslint.js b/lib/eslint/eslint.js
@@ -84,6 +84,7 @@ const LintResultCache = require("../cli-engine/lint-result-cache");
  *      doesn't do any config file lookup when `true`; considered to be a config filename
  *      when a string.
  * @property {Record<string,Plugin>} [plugins] An array of plugin implementations.
+ * @property {boolean} [stats] True enables added statistics on lint results.
  * @property {boolean} warnIgnored Show warnings when the file list includes ignored files
  * @property {boolean} [passOnNoPatterns=false] When set to true, missing patterns cause
  *      the linting operation to short circuit and not report any failures.
@@ -465,6 +466,7 @@ async function calculateConfigArray(eslint, {
  * @param {boolean} config.fix If `true` then it does fix.
  * @param {boolean} config.allowInlineConfig If `true` then it uses directive comments.
  * @param {Function} config.ruleFilter A predicate function to filter which rules should be run.
+ * @param {boolean} config.stats If `true`, then if reports extra statistics with the lint results.
  * @param {Linter} config.linter The linter instance to verify.
  * @returns {LintResult} The result of linting.
  * @private
@@ -477,6 +479,7 @@ function verifyText({
     fix,
     allowInlineConfig,
     ruleFilter,
+    stats,
     linter
 }) {
     const filePath = providedFilePath || "<text>";
@@ -497,6 +500,7 @@ function verifyText({
             filename: filePathToVerify,
             fix,
             ruleFilter,
+            stats,
 
             /**
              * Check if the linter should adopt a given code block or not.
@@ -528,6 +532,13 @@ function verifyText({
         result.source = text;
     }
 
+    if (stats) {
+        result.stats = {
+            times: linter.getTimes(),
+            fixPasses: linter.getFixPassCount()
+        };
+    }
+
     return result;
 }
 
@@ -808,6 +819,7 @@ class ESLint {
             fix,
             fixTypes,
             ruleFilter,
+            stats,
             globInputPaths,
             errorOnUnmatchedPattern,
             warnIgnored
@@ -922,6 +934,7 @@ class ESLint {
                             fix: fixer,
                             allowInlineConfig,
                             ruleFilter,
+                            stats,
                             linter
                         });
 
@@ -1010,7 +1023,8 @@ class ESLint {
             cwd,
             fix,
             warnIgnored: constructorWarnIgnored,
-            ruleFilter
+            ruleFilter,
+            stats
         } = eslintOptions;
         const results = [];
         const startTime = Date.now();
@@ -1034,6 +1048,7 @@ class ESLint {
                 fix,
                 allowInlineConfig,
                 ruleFilter,
+                stats,
                 linter
             }));
         }