--experimental-test-coverage empty since 23.5.0

[ChALkeR]

Version

v24.1.0

Platform

Darwin macbook-air.taile5a6b.ts.net 24.4.0 Darwin Kernel Version 24.4.0: Fri Apr 11 18:34:14 PDT 2025; root:xnu-11417.101.15~117/RELEASE_ARM64_T8122 arm64

Subsystem

test

What steps will reproduce the bug?

import { test } from 'node:test'

console.log(process.versions.node)

test('foo', () => {
  if (false) {
    throw new Error('Failed')
  } else {
    // pass
  }
})

% node --experimental-test-coverage --test i.js    
24.1.0
✔ foo (0.502334ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 57.841208
ℹ start of coverage report
ℹ ----------------------------------------------------------
ℹ file      | line % | branch % | funcs % | uncovered lines
ℹ ----------------------------------------------------------
ℹ ----------------------------------------------------------
ℹ all files | 100.00 |   100.00 |  100.00 | 
ℹ ----------------------------------------------------------
ℹ end of coverage report

Since 23.5.0:

% nvm use 23.5
Now using node v23.5.0 (npm v10.9.2)
% node --experimental-test-coverage --test i.js
23.5.0
✔ foo (0.330042ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 64.134167
ℹ start of coverage report
ℹ -----------------------------------------------------
ℹ file | line % | branch % | funcs % | uncovered lines
ℹ -----------------------------------------------------
ℹ -----------------------------------------------------
ℹ …les | 100.00 |   100.00 |  100.00 | 
ℹ -----------------------------------------------------
ℹ end of coverage report

23.4.0 was fine:

% nvm use 23.4                                 
Now using node v23.4.0 (npm v10.9.2)
% node --experimental-test-coverage --test i.js
23.4.0
✔ foo (0.326084ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 62.499375
ℹ start of coverage report
ℹ -------------------------------------------------------
ℹ file   | line % | branch % | funcs % | uncovered lines
ℹ -------------------------------------------------------
ℹ i.js   |  91.67 |    66.67 |  100.00 | 7
ℹ -------------------------------------------------------
ℹ …files |  91.67 |    66.67 |  100.00 | 
ℹ -------------------------------------------------------
ℹ end of coverage report

How often does it reproduce? Is there a required condition?

Every time

What is the expected behavior? Why is that the expected behavior?

Output on 23.4.0 (see above) was fine
Non-empty coverage report, reporting line 7 in the above example as uncovered

What do you see instead?

Empty coverage

Additional information

No response

[ChALkeR]

Ah, #56060
But --test-coverage-include i.js doesn't help this, so I was confused

[ChALkeR]

No, this still is confusing

In this setup, i0.js isn't a test file, but is ignored:

i.js:

import { test } from 'node:test'
import { foo } from './i0.js'

console.log(process.versions.node)

test('foo', () => {
  foo(2)
})

i0.js:

export function foo(arg) {
  if (!arg) {
    throw new Error('Failed')
  } else {
    return arg * 2
  }
}

Before 23.5:

23.4.0
✔ foo (0.345917ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 74.277709
ℹ start of coverage report
ℹ --------------------------------------------------------
ℹ file    | line % | branch % | funcs % | uncovered lines
ℹ --------------------------------------------------------
ℹ i.js    | 100.00 |   100.00 |  100.00 | 
ℹ i0.js   |  85.71 |    66.67 |  100.00 | 3
ℹ --------------------------------------------------------
ℹ … files |  93.33 |    80.00 |  100.00 | 
ℹ --------------------------------------------------------
ℹ end of coverage report

Since 23.5:

23.5.0
✔ foo (6.384833ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 164.013708
ℹ start of coverage report
ℹ -----------------------------------------------------
ℹ file | line % | branch % | funcs % | uncovered lines
ℹ -----------------------------------------------------
ℹ -----------------------------------------------------
ℹ …les | 100.00 |   100.00 |  100.00 | 
ℹ -----------------------------------------------------
ℹ end of coverage report

--test-coverage-include=i0.js doesn't help
--test-coverage-exclude='' doesn't work
--test-coverage-exclude='some-invalid-pattern' helps, but is... strange

[ChALkeR]

... ok, this apparently excludes files based on their global location, and if the project is inside a subdir of a .../test/... dir, everything is excluded

which looks confusing, but not very significant to care

[pmarchini]

Hey @ChALkeR, as per documentation the latest versions are excluding from the coverage the matching test files:

"By default all the matching test files are excluded from the coverage report. Specifying this option will override the default behavior." (https://nodejs.org/docs/latest/api/cli.html#--test-coverage-exclude)

[ChALkeR]

@pmarchini I see that, the issue was that those were not test files
E.g. if an os user is named test, everything in their projects would likely be excluded, as this operates on global paths, not on relative paths

This is unfortunate, but (as I already said) not very significant

[pmarchini]

Sorry @ChALkeR, I think I didn’t express myself clearly.

You're right about the match pattern.

At the moment, we're excluding, by default, everything that matches the inclusion patterns, which may need clarification.

It might be worth checking whether the documentation makes this clear, since what I shared definitely isn't clear enough

[ChALkeR]

Or perhaps matching should instead be performed on paths relative to the current project root / closest package.json file / cwd as fallback
Or use the files that are matched by node --test

node --test is also documented to include tests matching inclusion patterns, but it does not treat global paths as matches

I would have expected this to exclude the exact same files that are matched by node --test

[targos]

I think this is a bug.