feat(ci): coverage reporting with a baseline regression gate - #1036
feat(ci): coverage reporting with a baseline regression gate#1036AlemTuzlak wants to merge 9 commits into
Conversation
Adds a `test:coverage` script to every package and a `Coverage` job to the PR workflow that fails when coverage drops. Coverage was previously unrunnable: `@vitest/coverage-v8` was pinned to 4.0.14 in all 50 packages while `vitest` resolved to 4.1.10, so vitest could not resolve the provider at all (`ERR_MODULE_NOT_FOUND`). The two existing `test:coverage` scripts (ai, ai-client) could not have worked. Aligning the provider on 4.1.10 fixes it. - every package gains `test:coverage`, measured over `src/**` with json-summary + text-summary reporters (flags live in the script so no per-package vitest config changes were needed) - `scripts/coverage-check.mjs` diffs the fresh summaries against `coverage-baseline.json` and exits non-zero on a >0.5pp drop in statements/branches/functions/lines; unmeasured packages are skipped rather than counted as 0% - root gains `test:coverage:all`, `test:coverage:check`, `test:coverage:update` - `coverage-baseline.json` seeded from a full run of the 48 packages whose suites pass on this machine - CONTRIBUTING.md documents the workflow and the two known gaps
The check previously only printed to the CI log, so the numbers were invisible unless the job went red. Writing the same per-package table with deltas to $GITHUB_STEP_SUMMARY puts it on the PR's Checks tab on every run, pass or fail, with no token and no `pull-requests: write` permission.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdded Vitest coverage commands across packages, a coverage summary comparator, pull request coverage validation against the merge base, a main-branch cache-warming workflow, and updated coverage and lint documentation. ChangesCoverage regression gate
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
🚀 Changeset Version Preview19 package(s) bumped directly, 32 bumped as dependents. 🟥 Major bumps
🟨 Minor bumps
🟩 Patch bumps
|
|
View your CI Pipeline Execution ↗ for commit 61ba972
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-acp
@tanstack/ai-angular
@tanstack/ai-anthropic
@tanstack/ai-bedrock
@tanstack/ai-claude-code
@tanstack/ai-client
@tanstack/ai-code-mode
@tanstack/ai-code-mode-skills
@tanstack/ai-codex
@tanstack/ai-devtools-core
@tanstack/ai-durable-stream
@tanstack/ai-elevenlabs
@tanstack/ai-event-client
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-grok-build
@tanstack/ai-groq
@tanstack/ai-isolate-cloudflare
@tanstack/ai-isolate-node
@tanstack/ai-isolate-quickjs
@tanstack/ai-mcp
@tanstack/ai-memory
@tanstack/ai-mistral
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-opencode
@tanstack/ai-openrouter
@tanstack/ai-persistence
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-sandbox
@tanstack/ai-sandbox-cloudflare
@tanstack/ai-sandbox-daytona
@tanstack/ai-sandbox-docker
@tanstack/ai-sandbox-local-process
@tanstack/ai-sandbox-sprites
@tanstack/ai-sandbox-vercel
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-utils
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/openai-base
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
The first CI run confirmed the gate works — 47 of 50 packages reproduced the locally generated numbers to the hundredth — and surfaced the three that could not be measured correctly on Windows: - ai-acp and ai-sandbox have tests that fail on Windows for reasons unrelated to coverage, so they were missing from the baseline entirely - ai-sandbox-local-process reads ~20pp higher on Windows because its process-spawn code branches on platform, which tripped the gate All three now carry the numbers from the Linux CI run. CONTRIBUTING.md spells out that the baseline must come from Linux, since CI is what enforces it.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CONTRIBUTING.md`:
- Line 55: Update the lint command in the contributing guide’s command table
from pnpm test:eslint to pnpm test:oxlint, matching the script exposed by the
root package configuration.
In `@packages/ai-acp/package.json`:
- Line 41: Quote the src/** glob in the test:coverage script’s
--coverage.include argument so the shell cannot expand it before Vitest
processes it. Apply this change at packages/ai-acp/package.json:41-41,
packages/ai-angular/package.json:49-49,
packages/ai-anthropic/package.json:56-56, packages/ai-preact/package.json:41-41,
packages/ai-react-ui/package.json:37-37, packages/ai-react/package.json:41-41,
packages/ai-sandbox-cloudflare/package.json:49-49,
packages/ai-sandbox-daytona/package.json:40-40,
packages/ai-sandbox-docker/package.json:40-40,
packages/ai-sandbox-local-process/package.json:39-39, and
packages/ai-sandbox-sprites/package.json:41-41.
In `@packages/ai-codex/package.json`:
- Line 44: Update the test:coverage script in packages/ai-codex/package.json at
lines 44-44, packages/ai-durable-stream/package.json at lines 39-39,
packages/ai-vue/package.json at lines 36-36, packages/ai/package.json at lines
68-69, and packages/openai-base/package.json at lines 39-39 to include
--passWithNoTests in each Vitest coverage command.
In `@packages/ai-mcp/package.json`:
- Around line 59-60: Add the --passWithNoTests flag to each test:coverage script
in packages/ai-mcp/package.json, packages/ai-mistral/package.json,
packages/ai-openai/package.json, packages/ai-opencode/package.json,
packages/ai-openrouter/package.json, and packages/ai-persistence/package.json,
preserving the existing Vitest coverage options.
In `@packages/ai-solid/package.json`:
- Line 68: Align the local vitest declaration with `@vitest/coverage-v8` at exact
version 4.1.10 in packages/ai-solid/package.json:68-68,
packages/ai-bedrock/package.json:49-49,
packages/ai-claude-code/package.json:58-58,
packages/ai-client/package.json:70-70,
packages/ai-code-mode-skills/package.json:87-87,
packages/ai-code-mode/package.json:72-72,
packages/ai-sandbox-vercel/package.json:53-53,
packages/ai-sandbox/package.json:77-77, packages/ai-solid-ui/package.json:73-73,
packages/ai-svelte/package.json:72-72, and packages/ai-utils/package.json:55-55,
ensuring each package keeps both dependencies on the same exact version.
In `@scripts/coverage-check.mjs`:
- Around line 99-102: Update the coverage comparison flow around the !before
branch to fail the command when a measured package lacks a baseline entry, while
preserving success only for packages in the explicit documented exclusion list.
Ensure the failure directs users to run pnpm test:coverage:update to add the
missing baseline entry, and keep the existing additions reporting for permitted
exclusions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 572be2b3-80e0-4c3b-8c20-febbbdd35918
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (56)
.github/workflows/pr.ymlCLAUDE.mdCONTRIBUTING.mdcoverage-baseline.jsonpackage.jsonpackages/ai-acp/package.jsonpackages/ai-angular/package.jsonpackages/ai-anthropic/package.jsonpackages/ai-bedrock/package.jsonpackages/ai-claude-code/package.jsonpackages/ai-client/package.jsonpackages/ai-code-mode-skills/package.jsonpackages/ai-code-mode/package.jsonpackages/ai-codex/package.jsonpackages/ai-devtools/package.jsonpackages/ai-durable-stream/package.jsonpackages/ai-elevenlabs/package.jsonpackages/ai-event-client/package.jsonpackages/ai-fal/package.jsonpackages/ai-gemini/package.jsonpackages/ai-grok-build/package.jsonpackages/ai-grok/package.jsonpackages/ai-groq/package.jsonpackages/ai-isolate-cloudflare/package.jsonpackages/ai-isolate-node/package.jsonpackages/ai-isolate-quickjs/package.jsonpackages/ai-mcp/package.jsonpackages/ai-memory/package.jsonpackages/ai-mistral/package.jsonpackages/ai-ollama/package.jsonpackages/ai-openai/package.jsonpackages/ai-opencode/package.jsonpackages/ai-openrouter/package.jsonpackages/ai-persistence/package.jsonpackages/ai-preact/package.jsonpackages/ai-react-ui/package.jsonpackages/ai-react/package.jsonpackages/ai-sandbox-cloudflare/package.jsonpackages/ai-sandbox-daytona/package.jsonpackages/ai-sandbox-docker/package.jsonpackages/ai-sandbox-local-process/package.jsonpackages/ai-sandbox-sprites/package.jsonpackages/ai-sandbox-vercel/package.jsonpackages/ai-sandbox/package.jsonpackages/ai-solid-ui/package.jsonpackages/ai-solid/package.jsonpackages/ai-svelte/package.jsonpackages/ai-utils/package.jsonpackages/ai-vue-ui/package.jsonpackages/ai-vue/package.jsonpackages/ai/package.jsonpackages/openai-base/package.jsonpackages/preact-ai-devtools/package.jsonpackages/react-ai-devtools/package.jsonpackages/solid-ai-devtools/package.jsonscripts/coverage-check.mjs
| "lint:fix": "oxlint src --type-aware --fix", | ||
| "test:build": "publint --strict", | ||
| "test:oxlint": "oxlint src --type-aware", | ||
| "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary", |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Add --passWithNoTests to test:coverage for consistency with sibling packages. Most packages in this cohort add --passWithNoTests to the test:coverage script (for example packages/ai-devtools/package.json Line 76, packages/ai-elevenlabs/package.json Line 56, packages/ai-vue-ui/package.json Line 38). These five packages omit the flag. Without it, vitest run --coverage exits non-zero when a package has no matching test files, and this can break the CI Coverage job for nx affected runs.
packages/ai-codex/package.json#L44-L44: add--passWithNoTeststo thetest:coveragescript.packages/ai-durable-stream/package.json#L39-L39: add--passWithNoTeststo thetest:coveragescript.packages/ai-vue/package.json#L36-L36: add--passWithNoTeststo thetest:coveragescript.packages/ai/package.json#L68-L69: add--passWithNoTeststo thetest:coveragescript.packages/openai-base/package.json#L39-L39: add--passWithNoTeststo thetest:coveragescript.
🔧 Example fix (apply the equivalent to each listed file)
- "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
+ "test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary", | |
| "test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary", |
📍 Affects 5 files
packages/ai-codex/package.json#L44-L44(this comment)packages/ai-durable-stream/package.json#L39-L39packages/ai-vue/package.json#L36-L36packages/ai/package.json#L68-L69packages/openai-base/package.json#L39-L39
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-codex/package.json` at line 44, Update the test:coverage script
in packages/ai-codex/package.json at lines 44-44,
packages/ai-durable-stream/package.json at lines 39-39,
packages/ai-vue/package.json at lines 36-36, packages/ai/package.json at lines
68-69, and packages/openai-base/package.json at lines 39-39 to include
--passWithNoTests in each Vitest coverage command.
| "test:oxlint": "oxlint src --type-aware", | ||
| "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary", |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Inconsistent test:coverage script: missing --passWithNoTests. Several packages copy the test:coverage template without the --passWithNoTests flag that other packages in this PR use (for example packages/ai-isolate-node/package.json, packages/ai-isolate-cloudflare/package.json, packages/ai-isolate-quickjs/package.json, packages/ai-memory/package.json, and packages/ai-ollama/package.json). If any of these packages has no test files, Vitest exits with a failure instead of passing, and the PR Coverage job (pnpm run test:coverage:check) fails for a reason unrelated to actual coverage regressions.
packages/ai-mcp/package.json#L59-L60: add--passWithNoTeststo thetest:coveragescript.packages/ai-mistral/package.json#L35-L35: add--passWithNoTeststo thetest:coveragescript.packages/ai-openai/package.json#L47-L47: add--passWithNoTeststo thetest:coveragescript.packages/ai-opencode/package.json#L43-L43: add--passWithNoTeststo thetest:coveragescript.packages/ai-openrouter/package.json#L43-L43: add--passWithNoTeststo thetest:coveragescript.packages/ai-persistence/package.json#L46-L46: add--passWithNoTeststo thetest:coveragescript.
🛠️ Proposed fix (example for packages/ai-mcp/package.json)
- "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
+ "test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "test:oxlint": "oxlint src --type-aware", | |
| "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary", | |
| "test:oxlint": "oxlint src --type-aware", | |
| "test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary", |
📍 Affects 6 files
packages/ai-mcp/package.json#L59-L60(this comment)packages/ai-mistral/package.json#L35-L35packages/ai-openai/package.json#L47-L47packages/ai-opencode/package.json#L43-L43packages/ai-openrouter/package.json#L43-L43packages/ai-persistence/package.json#L46-L46
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-mcp/package.json` around lines 59 - 60, Add the --passWithNoTests
flag to each test:coverage script in packages/ai-mcp/package.json,
packages/ai-mistral/package.json, packages/ai-openai/package.json,
packages/ai-opencode/package.json, packages/ai-openrouter/package.json, and
packages/ai-persistence/package.json, preserving the existing Vitest coverage
options.
| "@tanstack/ai": "workspace:*", | ||
| "@types/node": "^24.10.1", | ||
| "@vitest/coverage-v8": "4.0.14", | ||
| "@vitest/coverage-v8": "4.1.10", |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
node <<'NODE'
import { execFileSync } from 'node:child_process'
import { readFileSync } from 'node:fs'
const files = execFileSync(
'find',
['packages', '-name', 'package.json', '-type', 'f'],
{ encoding: 'utf8' },
).trim().split('\n').filter(Boolean)
let failed = false
for (const file of files) {
const pkg = JSON.parse(readFileSync(file, 'utf8'))
const sections = [
pkg.dependencies,
pkg.devDependencies,
pkg.peerDependencies,
]
const coverage = sections.find(
section => section?.['`@vitest/coverage-v8`'],
)?.['`@vitest/coverage-v8`']
const vitest = sections.find(
section => section?.vitest,
)?.vitest
if (coverage === '4.1.10' && vitest !== '4.1.10') {
console.error(`${file}: coverage=${coverage}, vitest=${vitest}`)
failed = true
}
}
process.exit(failed ? 1 : 0)
NODERepository: TanStack/ai
Length of output: 3575
🌐 Web query:
unpkg @vitest/coverage-v8@4.1.10 package.json peerDependencies vitest
💡 Result:
In the @vitest/coverage-v8 version 4.1.10 package.json file, the peer dependency configuration is defined as follows: "peerDependencies": { "vitest": "4.1.10" }, "peerDependenciesMeta": { "vitest": { "optional": false } } This indicates that @vitest/coverage-v8@4.1.10 requires vitest version 4.1.10 as a peer dependency [1][2]. Note that while some users have discussed adding coverage providers as optional peer dependencies to the core vitest package to improve compatibility with strict package managers like pnpm, the @vitest/coverage-v8 package itself strictly requires the corresponding version of vitest [3].
Citations:
- 1: https://app.unpkg.com/@vitest/coverage-v8@4.1.10
- 2: https://unpkg.com/@vitest/coverage-v8@4.1.10/
- 3: vitest should declare @vitest/coverage-v8 and @vitest/coverage-istanbul as optional peer dependencies vitest-dev/vitest#10015
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "Root package manifests with vitest declarations:"
rg -n --glob 'package.json' '"vitest"\s*:' .
echo
echo "Top-level lockfile summary for vitest and coverage-v8:"
if [ -f pnpm-lock.yaml ]; then
rg -n -A 4 -B 1 '(@@ )?/(vitest|`@vitest/coverage-v8`)(@[0-9.]+)?:?| vitest@| `@vitest/coverage-v8`@' pnpm-lock.yaml | head -200
fi
echo
echo "Package files matching original affected names with vitest and coverage declarations:"
for f in packages/ai-solid/package.json packages/ai-bedrock/package.json packages/ai-claude-code/package.json packages/ai-client/package.json packages/ai-code-mode-skills/package.json packages/ai-code-mode/package.json packages/ai-sandbox-vercel/package.json packages/ai-sandbox/package.json packages/ai-solid-ui/package.json packages/ai-svelte/package.json packages/ai-utils/package.json; do
if [ -f "$f" ]; then
echo "--- $f"
node - <<EOF
const fs = require('fs')
const p = JSON.parse(fs.readFileSync(process.argv[1], 'utf8'))
for (const section of ['dependencies','devDependencies','peerDependencies']) {
if (p[section]) {
if (p[section].vitest || p.section?.['`@vitest/coverage-v8`']) {
console.log(section + ':', {
vitest: p[section].vitest,
coverage: p[section]['`@vitest/coverage-v8`']
})
}
}
}
EOF
fi
doneRepository: TanStack/ai
Length of output: 3955
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "Manifest snippets for original affected packages:"
for f in packages/ai-solid/package.json packages/ai-bedrock/package.json packages/ai-claude-code/package.json packages/ai-client/package.json packages/ai-code-mode-skills/package.json packages/ai-code-mode/package.json packages/ai-sandbox-vercel/package.json packages/ai-sandbox/package.json packages/ai-solid-ui/package.json packages/ai-svelte/package.json packages/ai-utils/package.json; do
if [ -f "$f" ]; then
echo "--- $f"
python3 - <<PY
import json, sys
p = json.load(open(sys.argv[1], 'r'))
for section in ['dependencies','devDependencies','peerDependencies']:
s = p.get(section)
if s and ('vitest' in s or '`@vitest/coverage-v8`' in s):
print(section, {
'vitest': s.get('vitest'),
'`@vitest/coverage-v8`': s.get('`@vitest/coverage-v8`')
})
PY
else
echo "MISSING $f"
fi
done
echo
echo "pnpm coverage-v8 declarations:"
rg -n '`@vitest/coverage-v8`|vitest:' pnpm-lock.yaml | head -120Repository: TanStack/ai
Length of output: 343
🌐 Web query:
@vitest/coverage-v8@4.1.10 peerDependencies vitest exact 4.1.10 pnpm optional peer dependencies optional false
💡 Result:
In the Vitest ecosystem, @vitest/coverage-v8 is declared as an optional peer dependency of vitest [1][2]. This change was introduced to resolve issues with strict package managers like pnpm, which struggled to resolve the coverage provider dynamically at runtime when it was not explicitly listed in the dependency tree [3][1]. Specifically, Vitest versions including 4.1.10 list @vitest/coverage-v8 within their package.json under peerDependencies, with a corresponding entry in peerDependenciesMeta marking it as optional: true [3][1][2]. Regarding pnpm behavior: 1. When peerDependenciesMeta.optional is set to true, the package manager does not require the dependency to be present for the installation to succeed [4]. 2. By default, pnpm's autoInstallPeers setting (which is true) will attempt to automatically install all peer dependencies, including those marked as optional [5][6]. 3. If you wish to prevent pnpm from automatically installing these optional peers, you can set autoInstallPeers to false in your.npmrc or pnpm-workspace.yaml file [5][6]. 4. Note that there have been reported discrepancies where optional peers are still installed despite autoInstallPeers: false due to specific implementation details in certain pnpm versions, though this is generally considered a configuration or behavioral edge case rather than the standard intent [6][7].
Citations:
- 1: fix: add
@vitest/coverage-v8and@vitest/coverage-istanbulas optional dependency vitest-dev/vitest#10025 - 2: fix: add
@vitest/coverage-v8and@vitest/coverage-istanbulas optional dependency vitest-dev/vitest#10025 - 3: vitest should declare @vitest/coverage-v8 and @vitest/coverage-istanbul as optional peer dependencies vitest-dev/vitest#10015
- 4: https://pnpm.io/10.x/package_json
- 5: https://pnpm.io/settings
- 6: autoInstallPeers installs optional peer dependencies (peerDependenciesMeta optional: true) pnpm/pnpm#11155
- 7: pnpm 12 adds optional peers to a workspace importer with autoInstallPeers disabled pnpm/pnpm#13325
🌐 Web query:
vitest coverage provider peer dependencies coverage-v8 compatibility vitest 4.1.10
💡 Result:
In Vitest v4.1.10, the @vitest/coverage-v8 package serves as the primary coverage provider [1][2]. As of recent updates within the v4 release cycle, @vitest/coverage-v8 and @vitest/coverage-istanbul are explicitly configured as optional peer dependencies of vitest [3][4]. This change was implemented to resolve package resolution issues, particularly in environments with strict dependency isolation like pnpm, by allowing the coverage provider to be properly linked while remaining optional [4]. Key compatibility considerations for Vitest v4.1.10 include: 1. Version Synchronicity: While these packages are now optional peer dependencies, it remains critical to keep the version of the coverage provider (@vitest/coverage-v8) in sync with the core vitest package [5]. Historically, strict peer dependency requirements have caused installation failures (e.g., ERESOLVE errors) when versions mismatched [5]. 2. Configuration: The V8 provider is the default in Vitest [1][6]. To use it, ensure the package is installed in your project: npm install -D @vitest/coverage-v8 3. Migration Notes: If you are migrating to v4, note that several legacy coverage options (such as coverage.all) have been removed, and the default behavior for coverage report inclusion has changed to only include files loaded during the test run [7][8]. For consistent behavior, ensure that both vitest and @vitest/coverage-v8 are updated to v4.1.10 [9][10].
Citations:
- 1: https://vitest.dev/guide/coverage
- 2: https://www.npmjs.com/package/@vitest/coverage-v8
- 3: fix: add
@vitest/coverage-v8and@vitest/coverage-istanbulas optional dependency vitest-dev/vitest#10025 - 4: vitest should declare @vitest/coverage-v8 and @vitest/coverage-istanbul as optional peer dependencies vitest-dev/vitest#10015
- 5: [CI Failure Doctor] Dependabot PR #11024 broke CI: npm peer dependency conflict between vitest@4.0.17 and @vitest/coverage-v8@4.0.10 github/gh-aw#11030
- 6: https://vitest.dev/config/coverage
- 7: https://vitest.dev/guide/migration.html
- 8: https://github.com/vitest-dev/vitest/blob/1a4705da/docs/guide/migration.md
- 9: https://github.com/vitest-dev/vitest/releases/tag/v4.1.10
- 10: https://newreleases.io/project/npm/@vitest/coverage-v8/release/4.1.10
Align vitest with @vitest/coverage-v8 at the point where they are declared.
@vitest/coverage-v8@4.1.10 has an exact vitest peer of 4.1.10. Keep the matching local vitest declaration on the same exact version so updates do not drift the coverage provider from its supported Vitest core version.
📍 Affects 11 files
packages/ai-solid/package.json#L68-L68(this comment)packages/ai-bedrock/package.json#L49-L49packages/ai-claude-code/package.json#L58-L58packages/ai-client/package.json#L70-L70packages/ai-code-mode-skills/package.json#L87-L87packages/ai-code-mode/package.json#L72-L72packages/ai-sandbox-vercel/package.json#L53-L53packages/ai-sandbox/package.json#L77-L77packages/ai-solid-ui/package.json#L73-L73packages/ai-svelte/package.json#L72-L72packages/ai-utils/package.json#L55-L55
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-solid/package.json` at line 68, Align the local vitest
declaration with `@vitest/coverage-v8` at exact version 4.1.10 in
packages/ai-solid/package.json:68-68, packages/ai-bedrock/package.json:49-49,
packages/ai-claude-code/package.json:58-58,
packages/ai-client/package.json:70-70,
packages/ai-code-mode-skills/package.json:87-87,
packages/ai-code-mode/package.json:72-72,
packages/ai-sandbox-vercel/package.json:53-53,
packages/ai-sandbox/package.json:77-77, packages/ai-solid-ui/package.json:73-73,
packages/ai-svelte/package.json:72-72, and packages/ai-utils/package.json:55-55,
ensuring each package keeps both dependencies on the same exact version.
| if (!before) { | ||
| additions.push(name) | ||
| rows.push([name, ...METRICS.map((m) => `${after[m].toFixed(2)}%`), 'new']) | ||
| continue |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Fail when a measured package has no baseline entry.
Lines 99-102 report an addition but allow the command to succeed. A new or omitted package can therefore pass CI without coverage regression gating.
Treat additions as a failure unless the package is in an explicit documented exclusion list. Require pnpm test:coverage:update to add the baseline entry.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/coverage-check.mjs` around lines 99 - 102, Update the coverage
comparison flow around the !before branch to fail the command when a measured
package lacks a baseline entry, while preserving success only for packages in
the explicit documented exclusion list. Ensure the failure directs users to run
pnpm test:coverage:update to add the missing baseline entry, and keep the
existing additions reporting for permitted exclusions.
Coverage was already absent from `test`, `test:pr`, `test:ci` and the git hooks, but the docs still presented it as part of the local workflow — four coverage rows in CONTRIBUTING's day-to-day command table, and an instruction to re-baseline with `pnpm test:coverage:update` on your own machine. That last one was an active trap: the committed numbers are Linux numbers and a few packages measure differently per platform, so following the doc on a Mac or Windows box reintroduces exactly the ai-sandbox-local-process failure the previous commit fixed. The check now prints a paste-ready JSON block for every package the baseline needs to absorb, in both the log and the run summary, so re-baselining is a copy out of CI rather than a local run. `test:coverage:update` stays as the maintainer escape hatch but is no longer the documented path.
Review feedback: - quote `--coverage.include='src/**'` in all 50 test:coverage scripts so a POSIX shell can't expand the pattern before vitest parses it. Verified the quoted form behaves identically on Windows (pnpm normalises it) and yields byte-identical percentages, so it is safe cross-platform. - CONTRIBUTING referenced `pnpm test:eslint`, which does not exist — the repo moved to oxlint. Fixed that and the two other stale ESLint mentions, and added the Coverage gate to the list of what CI runs.
|
Thanks — went through all six. Two applied in 187f107, three declined on the evidence, one design call. Applied
Declined — Every one of these has tests, so the flag would be actively harmful: All eleven reported coverage on the green CI run. Declined — pinning The premise doesn't hold: 45 of the 50 packages don't declare Worth noting the underlying concern was real and is what this PR fixes: the provider was pinned to Design call — failing when a measured package has no baseline entry Keeping it non-fatal. Hard-failing creates a chicken-and-egg problem: a new package can't be baselined locally (the numbers are Linux-authoritative — see CONTRIBUTING), so the author would need a CI round-trip just to unblock their own PR. The suggested exclusion list is a second config surface that itself needs maintaining. It isn't silent as-is: unbaselined packages are listed in the run summary, marked |
…ing it The committed baseline had the maintenance problems it looked like it would: it needed manual syncing, it was platform-sensitive (ai-sandbox-local-process reads ~20pp higher on Windows, which broke the gate on this very PR), and adding or removing a package meant hand-editing a generated file. The Coverage job now measures each affected package twice in one job — on the PR head and on its merge-base with main — and compares those. Nothing to keep in sync, no platform skew (both sides run on the same runner), and package add/remove needs no action. - delete coverage-baseline.json, `test:coverage:check`, `test:coverage:update` and the paste-a-JSON-block flow; all of it existed only to tend the file - coverage-check.mjs gains `--collect <dir>` (snapshot summaries so they survive the checkout between runs) and `--base <dir> --head <dir>` - resolve the package list once, with `--with-target=test:coverage`, so both sides measure the same set and a docs-only PR resolves to an empty list rather than erroring in run-many; the base side intersects that list with the projects that exist there, so PR-added packages report as `new` - new coverage.yml runs test:coverage:all on pushes to main purely to populate the Nx Cloud cache: the base-side task hashes then already exist remotely and, because test:coverage declares its coverage dir as an output, restore without re-running tests Verified locally: collect/compare round-trip, a drop failing with only the out-of-tolerance metrics listed, PR-added packages reporting `new` without failing, removed packages ignored, missing base dir, empty head, bad args.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/pr.yml:
- Around line 59-63: The coverage-check.mjs script is new in this PR and does
not exist in the merge-base, causing the command at line 82 to fail when the
workflow checks out NX_BASE. Copy scripts/coverage-check.mjs to a temporary
location in $RUNNER_TEMP before checking out the base revision, then update both
the base-run coverage-check command (around line 82) and the PR-run
coverage-check command (line 63 in the diff) to use the temporary copy path
instead of the local script file. This preserves the head version of the script
for both the base and PR collection runs.
In `@scripts/coverage-check.mjs`:
- Around line 36-38: Update the coverage comparison flow around the zero-total
summary check to distinguish an unmeasured head package from a genuinely
non-comparable summary. When a package resolved from the base intersection has
no head summary or zero statements, fail the gate with the existing “Not
measured on this PR” behavior, unless the package is explicitly excluded;
preserve successful handling for excluded packages and normal comparisons.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: fa15c666-6789-4950-b3ff-b4f1e34a62ef
📒 Files selected for processing (6)
.github/workflows/coverage.yml.github/workflows/pr.ymlCLAUDE.mdCONTRIBUTING.mdpackage.jsonscripts/coverage-check.mjs
💤 Files with no reviewable changes (1)
- package.json
🚧 Files skipped from review as they are similar to previous changes (1)
- CONTRIBUTING.md
| // A package with no source loaded reports totals of 0/0 as 100%; that is not | ||
| // a number worth comparing. | ||
| if (summary.total.statements.total === 0) return undefined |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fail when a previously measured package has no head summary.
The PR workflow resolves package names from the head, then measures the base intersection. Therefore, a name present only in base means that the requested head measurement produced no comparable summary.
A change that disables collection or reduces statements to zero is reported as “Not measured on this PR” and exits successfully. Treat this state as a gate failure, unless the package is in an explicit exclusion list.
Also applies to: 149-175, 212-216
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/coverage-check.mjs` around lines 36 - 38, Update the coverage
comparison flow around the zero-total summary check to distinguish an unmeasured
head package from a genuinely non-comparable summary. When a package resolved
from the base intersection has no head summary or zero statements, fail the gate
with the existing “Not measured on this PR” behavior, unless the package is
explicitly excluded; preserve successful handling for excluded packages and
normal comparisons.
The merge-base step ran `node scripts/coverage-check.mjs --collect` after checking out the base commit, where that file does not exist — this PR is what adds it: Error: Cannot find module '/home/runner/work/ai/ai/scripts/coverage-check.mjs' Not specific to this PR either: any PR that adds or moves the script would hit the same thing. Copying it to $RUNNER_TEMP before the checkout fixes it and also guarantees both runs and the comparison use the same version, which the previous `git checkout $GITHUB_SHA -- scripts/...` in the compare step only approximated. The base coverage run itself was fine, at 3/6 Nx cache hits before coverage.yml has ever run on main.
The previous run passed for the wrong reason. `coverage/` is gitignored, so checking out the merge base left this PR's coverage files on disk, and the base-side `--collect` read them back: 47 packages were compared against themselves, which is why every delta was exactly 0.00. Both runs now clear `packages/*/coverage` first. Nx restores the directory for cached tasks, so genuine cache hits still work. On this PR the base side will now legitimately collect almost nothing — the merge base predates the json-summary reporter, so there is no base data to compare against and packages report as `new`. That resolves itself once this lands and main has the machinery.
zizmor flagged both `${{ steps.affected.outputs.list }}` uses inside run
blocks as code injection via template expansion (pr.yml:69, pr.yml:85), and
it's right: the list is built from package names in the PR's own files, so on
a fork PR a package named with shell metacharacters would be interpolated
straight into the script. Passing it through env keeps it out of the shell
source entirely.
Confirmed clean by running zizmor v1.28.0 locally over both workflows.
1 participant
🎯 Changes
Adds coverage reporting as a first-class script option and a
CoverageCI job that fails when coverage drops.Coverage did not actually work before this.
@vitest/coverage-v8was pinned to4.0.14across all 50 packages whilevitestresolves to4.1.10, so vitest could not load the provider at all:The two pre-existing
test:coveragescripts (ai,ai-client) could not have run. Aligning the provider on4.1.10fixes it.vitestitself is already on the latest stable — v5 exists only as5.0.0-beta.7, so it is untouched.What's added
test:coverageon every package —vitest run --coverage, measured oversrc/**withtext-summary+json-summaryreporters. Flags live in the script, so no per-package vitest config changes were needed.scripts/coverage-check.mjs— diffs the fresh summaries againstcoverage-baseline.jsonand exits non-zero when statements/branches/functions/lines drop more than 0.5pp. Packages not affected by a PR produce no summary and are skipped rather than counted as 0%.test:coverage:all,test:coverage:check,test:coverage:update.coverage-baseline.json— seeded from a full run. There are no hand-picked thresholds; each package's current number is its own floor, which avoids arguing about one global percentage across 50 very different packages.Coveragejob inpr.yml— runstest:coverage:checkon affected packages, parallel totest.$GITHUB_STEP_SUMMARY, so it renders on the Checks tab on every run, pass or fail. Nopull-requests: writeneeded, no comment spam.Failure output:
Current state
Highlights by line coverage — full table in
coverage-baseline.json:ai-openaiat 48% andai-ollamaat 28% stand out given how central they are. Nothing here is a gate on this PR — the ratchet just stops them getting worse.Known gaps
ai-acpandai-sandboxare absent from the baseline. Their suites fail on Windows for reasons unrelated to coverage (EBUSYon temp-dir unlink; a POSIX-vs-Windows path assertion) —test:libfails identically. They'll show up asnew(non-failing, with a nudge to re-baseline) on the first CI run; committing the baseline from a Linux run adds them..tsxfiles are dropped from the report with aFailed to parse ... Excluding it from coveragewarning — a rolldown limitation in the provider's uncovered-file remapping..tsxthat tests do load is measured normally, so the UI packages read higher than their real coverage. Documented in CONTRIBUTING.md.✅ Checklist
pnpm run test:pr.🚀 Release Impact
Summary by CodeRabbit
Tests
Chores
Documentation