Skip to content

[Test] Bun reports the All-files coverage row as an unweighted mean over files and bunfig.toml never sets coverageSkipTestFiles, so 319 test files sit in the badge at 100 percent and the real src figure is about 5 points lower #1016

Description

@pathosDev

Problem

The ~91 % on the README badge, the number test:coverage:gate enforces, and the number #541 proposes to ratchet to 88–90 % are all the same value: the % Lines cell of Bun's All files row. That value is not the line coverage of src/. Two independent mechanics separate it from what everyone reads it as.

1. It is an unweighted mean over files, not Σ hit / Σ found. Measured, not assumed — see Evidence. A 3 000-line module at 40 % and a 6-line barrel at 100 % contribute equally to the aggregate. Coverage therefore moves with the number of files, and a large under-tested file is masked by a handful of small fully-covered ones.

2. The test suite is inside the mean. bunfig.toml never sets coverageSkipTestFiles, and Bun's default is false, so every *.test.ts file is a row in the table — at essentially 100 %, by construction, because a test file that did not execute is a failing test. There are 319 such files (60 382 lines) against 550 files in src/ (78 712 lines). Roughly a third of the rows in the mean are pinned at 100 % and can never move.

The combination is why the floor does not bite in the way #541 assumes. Solving for the src-only figure from the reported 91 %: with 319 test files at 100 % removed from a mean over ~967 loaded files, the remainder lands at ≈ 86 %, not 91 %. The 80 % floor therefore permits src/ to fall to roughly the mid-60s before CI notices — and the ratchet #541 proposes would ratchet a number that is ~5 points optimistic and structurally insensitive to file size.

Evidence

bunfig.toml in full — the [test] block sets one key, and it is not this one:

bunfig.toml:12-17
[test]
coveragePathIgnorePatterns = [
  "src/testkit/ParallelMultiNodeSpec.ts",
  "src/testkit/internal/MultiNodeBroker.ts",
  "src/testkit/internal/parallel-multi-node-bootstrap.ts",
]

Both consumers read the same cell. CI:

.github/workflows/test.yml:130-135
          # Line coverage from the "All files" aggregate row.  Bun's
          # --coverage table is `File | % Funcs | % Lines | Uncovered`, so
          # the LINE column is awk field $3 ($2 is % Funcs).
          LINES=$(echo "$OUTPUT" | grep "^All files" | awk -F'|' '{print $3}' | tr -d ' ')
          LINES_INT=${LINES%.*}
          [[ -z "$LINES_INT" ]] && LINES_INT=0

and the local gate:

scripts/coverage-gate.mjs:42-46
// Parse the "All files" aggregate row.  Bun's `--coverage` table is
//   File ... | % Funcs | % Lines | Uncovered Line #s
// so LINE coverage is the SECOND numeric column (the first is % Funcs).
const m = output.match(/^All files\s+\|\s+[0-9]+(?:\.[0-9]+)?\s+\|\s+([0-9]+(?:\.[0-9]+)?)/m);

A/B, one small test file. bun test --coverage tests/unit/Supervision.test.ts, first with bunfig.toml as committed, then with an otherwise identical config that adds coverageSkipTestFiles = true:

A — bunfig.toml as committed
All files                                                 |   25.14 |   47.99 |
 tests\unit\Supervision.test.ts                           |   87.67 |  100.00 |

B — same, plus coverageSkipTestFiles = true
All files                                                 |   24.35 |   47.33 |
 (tests\unit\Supervision.test.ts row absent)

The test file is a first-class row and it sits at 100.00 %. In lcov terms from the same run, tests/unit/Supervision.test.ts reports LF=217, LH=217.

The aggregate is an unweighted mean. The same run, reduced from its lcov output three ways:

run 1 — tests/unit/Supervision.test.ts (80 files)
  bun "All files" % Lines ............................ 47.99
  unweighted mean of per-file line % ................. 47.99   <- matches
  weighted (sum LH / sum LF) ......................... 40.97
  unweighted mean excluding *.test.ts ................ 47.33   <- matches run B

run 2 — MailboxVariants + Supervision + CircuitBreaker (87 files)
  bun "All files" % Lines ............................ 53.65
  unweighted mean of per-file line % ................. 53.65   <- matches
  weighted (sum LH / sum LF) ......................... 46.59

Two runs, exact agreement with the unweighted mean and a 6–7 point disagreement with the weighted one. The reported figure is mean(per-file %).

Denominator sizes, find … | xargs wc -l:

src/**/*.ts          550 files    78 712 lines
tests/**/*.ts        417 files    71 690 lines
  of which *.test.ts 319 files    60 382 lines

And the number the badge carries:

README.md:9
  <a href="#"><img alt="coverage" src="https://img.shields.io/badge/coverage-~91%25-22c55e?style=flat-square"/></a>

Proposal

  • Set coverageSkipTestFiles = true in bunfig.toml. It is one line, and it removes 319 permanently-100 % rows from the mean. Test helpers under tests/ are not matched by it, so add tests/ to coveragePathIgnorePatterns as well — the harness is not the product.
  • Stop reading All files. Emit lcov (--coverage-reporter=lcov) and compute Σ LH / Σ LF over src/ in scripts/coverage-gate.mjs. That is the number everyone believes they are already reading, and it is size-weighted, so a large module going dark cannot be papered over by small files.
  • Re-baseline the floor after the metric changes, not before. Whatever number the corrected metric reports on develop is the new reality; [Test] Coverage gate: single parser, ratcheted floor, per-module floors #541's 88–90 % target was derived from the old one and will not apply.
  • Have the gate print both figures for one release cycle (old parser, new parser) so the discontinuity in the badge history is explainable rather than looking like a regression.

Acceptance sketch

  • bunfig.toml sets coverageSkipTestFiles = true and excludes tests/ from the coverage denominator.
  • scripts/coverage-gate.mjs computes line coverage as Σ LH / Σ LF restricted to src/, from lcov rather than from the All files text row.
  • .github/workflows/test.yml calls that script instead of re-deriving the number in bash (this is [Test] Coverage gate: single parser, ratcheted floor, per-module floors #541's first bullet; it becomes mandatory here because the parser is no longer a one-line awk).
  • The README badge and the floor are re-baselined against the corrected metric, in one commit, with the delta stated in the commit body.
  • A unit test pins the aggregation: given a synthetic lcov with two files of very different sizes, the computed figure equals the weighted one and not the mean.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: confirmed by execution. The A/B was run on the single file tests/unit/Supervision.test.ts — once with the committed bunfig.toml, once with bun --config=… pointing at a copy that adds coverageSkipTestFiles = true and is otherwise identical (the temporary file was deleted; the tree is unmodified). The unweighted-mean finding was not part of the original claim: it came out of reducing the same runs' lcov three ways and finding that only the unweighted mean reproduces Bun's printed figure, on two independent runs. The full suite was deliberately not run, so the ≈ 86 % src-only figure above is arithmetic from the reported 91 %, not a measurement — the direction and the mechanism are measured, the exact magnitude is not.

Adjacent: #541 proposes ratcheting the floor from 80 to 88–90 and unifying the two parsers. It is the right instinct against the wrong number — its own text says "actual coverage is ~91 %", which is precisely the figure this issue shows to be inflated by ~5 points and blind to file size. #541 should land after this one, with its target recomputed. #538 (nightly for the ACTOR_TS_SKIP_FLAKY_MNS quarantine) and #540 (gate typecheck:dev) are separate axes of the same "the green check is narrower than it looks" theme and do not overlap.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinginfrastructureCI / build / live-integration testspriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions