ci(e2e): scope risk signals to selected tests

[jyaunches]

Summary

Risk-signal evidence now counts only tests selected by Vitest's testNamePattern. This prevents selector-excluded sibling tests from blocking a PR while preserving failures for selected skips and selectors that match no tests.

Changes

• Apply the resolved Vitest name pattern before the risk-signal reporter counts test results.
• Add regression coverage for an excluded sibling, a selected skip, and a selector that matches no tests.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This changes internal CI evidence accounting and conforms to the existing selected-shard contract.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: PR ci(e2e): scope risk signals to selected tests #7488; internal CI evidence only. test/e2e/README.md already defines counts for each selected shard.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable
• Station profile/scenario: Not applicable
• Result: Not applicable
• Supporting evidence: Not applicable

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npm exec -- vitest run test/e2e-risk-signal-reporter.test.ts --project integration: 14 passed. A real reporter selection recorded passed=1, skipped=0; a selected static skip recorded passed=0, skipped=1.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: npm test was attempted, but the restricted local sandbox blocked loopback-server tests and caused unrelated timeouts. PR CI remains authoritative.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Julie Yaunches jyaunches@nvidia.com

Summary by CodeRabbit

• New Features

  • Risk-signal reporting now respects the configured test name pattern.
  • Evidence counts include only matching tests, including passed, failed, skipped, and pending results.
  • Reports accurately show zero counts when no tests match the configured pattern.

• Tests

  • Added coverage for filtered test runs, skipped tests, and patterns with no matches.

[coderabbitai]

📝 Walkthrough

Walkthrough

The risk-signal reporter now applies Vitest’s configured testNamePattern when counting tests. E2E tests add named mocks and verify matching, skipped-only, and no-match outcomes.

Changes

Risk-signal name filtering

Layer / File(s)	Summary
Pattern-aware risk-signal counting test/e2e/risk-signal-reporter.ts	The reporter normalizes test names, filters counts using testNamePattern, captures the pattern during initialization, and applies it when writing risk-signal output.
Pattern filtering test coverage test/e2e-risk-signal-reporter.test.ts	Test mocks provide per-test names, and coverage verifies excluded tests, skipped matches, and patterns matching no tests.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant VitestConfig
  participant E2eRiskSignalReporter
  participant TestModules
  participant RiskSignalJSON
  VitestConfig->>E2eRiskSignalReporter: provide testNamePattern during onInit
  E2eRiskSignalReporter->>TestModules: read full test names and results
  TestModules-->>E2eRiskSignalReporter: return matching test results
  E2eRiskSignalReporter->>RiskSignalJSON: write filtered risk-signal counts

Loading

Suggested labels: area: ci, area: e2e, area: observability

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: scoping e2e risk signals to selected tests.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

test/e2e-risk-signal-reporter.test.ts (1)

149-223: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Exercise the reporter lifecycle.

These tests pass the pattern directly to writeRiskSignal, so they would still pass if onInit stopped capturing vitest.config.testNamePattern or onTestRunEnd stopped forwarding it. Add one lifecycle test that invokes both hooks and asserts an excluded sibling is absent from the written signal. Vitest applies this pattern to full test names. (vitest.dev)

As per path instructions, “Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-risk-signal-reporter.test.ts` around lines 149 - 223, Add a
lifecycle-focused test in test/e2e-risk-signal-reporter.test.ts that invokes
onInit with a Vitest config containing testNamePattern, then invokes
onTestRunEnd without passing the pattern directly. Assert the written signal
excludes an unmatched sibling test while retaining the selected test, using only
the public written-signal outcome.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e-risk-signal-reporter.test.ts`:
- Around line 149-223: Add a lifecycle-focused test in
test/e2e-risk-signal-reporter.test.ts that invokes onInit with a Vitest config
containing testNamePattern, then invokes onTestRunEnd without passing the
pattern directly. Assert the written signal excludes an unmatched sibling test
while retaining the selected test, using only the public written-signal outcome.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 76638290-e888-4970-b1fa-c1caf8be7027

📥 Commits

Reviewing files that changed from the base of the PR and between 5faed2c and 8eefe21.

📒 Files selected for processing (2)

• test/e2e-risk-signal-reporter.test.ts
• test/e2e/risk-signal-reporter.ts

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

1 optional E2E recommendation

• network-policy

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-risk-signal-reporter.test.ts`:
- Around line 154-156: Update the test around testedSha to use the EXPECTED_SHA
fixture or mock execFileSync instead of reading the current Git checkout,
keeping the test independent of Git availability and repository state. In the
test cleanup, add vi.unstubAllEnvs() so environment stubs do not leak into
subsequent tests.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3aecb626-ed89-41ef-84ca-9e0eb1283d0b

📥 Commits

Reviewing files that changed from the base of the PR and between 8eefe21 and e35476b.

📒 Files selected for processing (1)

• test/e2e-risk-signal-reporter.test.ts