ci(e2e): publish phase runtime summary

[cjagwani]

Summary

Follow-up to #7100: scheduled and ordinary manual E2E runs now publish a current-run semantic phase runtime table in the existing GitHub Actions scorecard. The table reuses the artifact audit instead of adding live-log noise, and the resume test no longer reports a bookkeeping-only phase that consistently completed in zero seconds.

Changes

• download this run's E2E artifacts in the read-only scorecard job and append the existing runtime audit table to the GitHub step summary
• report the slowest observed phase's duration and passed, failed, or skipped outcome
• keep missing or invalid progress evidence diagnostic-only so it cannot hide the main E2E result
• remove record the completed resume contract from onboard-resume; the final meaningful comparison phase now owns target completion
• protect the workflow boundary, native Node loading path, audit rendering, and phase plan with regression tests
• document the automatic current-run table and the existing multi-run audit command

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: no user-facing product behavior changed; the E2E maintainer guides are updated.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: the scorecard remains read-only (actions: read, contents: read), uses an immutable artifact-action pin, treats download/audit failure as diagnostic-only, and has workflow-boundary regression coverage.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed implementation
• Result: docs-updated
• Evidence: test/e2e/README.md and test/e2e/docs/README.md document the current-run GitHub Actions scorecard table and its phase-duration and outcome fields.
• Agent: Codex Desktop
• PR: ci(e2e): publish phase runtime summary #7388

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: not applicable
• Station profile/scenario: not applicable
• Result: not applicable
• Supporting evidence: not applicable

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npm exec vitest -- run --project e2e-support test/e2e/support/test-runtime-audit.test.ts test/e2e/support/e2e-scorecard.test.ts test/e2e/support/e2e-scorecard-coordinator.test.ts test/e2e/support/e2e-operations-workflow-boundary.test.ts (61 passed); npm run test:e2e-phases:check (121 tests across 80 files); npm run test:projects:check; npm run typecheck:cli.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: full e2e-support run reached 1,343 passes; five discovery timeouts passed on isolated rerun, and one unrelated lifecycle failure reproduced unchanged in the original checkout.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Summary by CodeRabbit

• New Features
  • GitHub Actions scorecards now add an E2E Test Phase Runtime section, including a semantic phase runtime table and slowest-phase details.
  • Runtime reporting now includes each phase’s duration and outcome, with the slowest observed phase’s outcome shown as well.
• Bug Fixes
  • If runtime progress artifacts are missing or invalid, scorecards render a fallback “runtime unavailable” message instead of failing.
• Documentation
  • Updated E2E documentation to reflect the expanded runtime tables and enriched scorecard summary behavior for scheduled/manual runs.
• Tests
  • Expanded E2E runtime and workflow boundary validations for the new section, artifact handling, and outcome reporting.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The E2E scorecard now downloads progress artifacts, audits phase runtimes with outcomes, handles unavailable artifacts with fallback Markdown, and appends the runtime section to GitHub Actions summaries. Workflow validators, tests, E2E phase metadata, and documentation are updated accordingly.

Changes

E2E runtime reporting

Layer / File(s)	Summary
Outcome-aware runtime audit scripts/audit-test-runtime.mts, test/e2e/support/test-runtime-audit.test.ts	Runtime rows and tables now include the slowest phase outcome, with validation for passed, failed, and skipped phases plus empty-artifact summary coverage.
Scorecard artifact and summary integration .github/workflows/e2e.yaml	The scorecard downloads current-run artifacts, invokes the runtime audit, uses fallback Markdown on failure, and appends the runtime summary.
Workflow boundary and module contract validation tools/e2e/operations-workflow-boundary.mts, test/e2e/support/e2e-operations-workflow-boundary.test.ts, test/e2e/support/e2e-scorecard.test.ts	Validation and tests enforce pinned artifact downloads, sparse checkout contents, runtime environment wiring, module exports, audit invocation, and combined output.
E2E phase metadata and reporting documentation test/e2e/live/onboard-resume.test.ts, test/e2e/README.md, test/e2e/docs/README.md	The resume test removes a completed-resume phase marker, while documentation describes runtime outcomes and scorecard reporting.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ScorecardJob
  participant ArtifactStore
  participant RuntimeAudit
  participant GitHubSummary
  ScorecardJob->>ArtifactStore: Download e2e-* progress artifacts
  ScorecardJob->>RuntimeAudit: Audit RUNTIME_ARTIFACTS
  RuntimeAudit-->>ScorecardJob: Return runtime Markdown or failure
  ScorecardJob->>GitHubSummary: Append scorecard and runtime sections

Loading

Possibly related PRs

• NVIDIA/NemoClaw#7100: Introduces the runtime-audit script and progress-artifact model extended by this change.

Suggested labels: feature, area: observability

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: publishing the E2E phase runtime summary in CI.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/publish-e2e-runtime-audit

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit c190bea in the codex/publish-e2e-ru... branch remains at 96%, unchanged from commit d13e34a in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit c190bea in the codex/publish-e2e-ru... branch remains at 80%, unchanged from commit d13e34a in the main branch.

Show a code coverage summary of the most impacted files.

File	main d13e34a	codex/publish-e2e-ru... c190bea	+/-
src/lib/state/m...-acquisition.ts	89%	84%	-5%
src/lib/platform.ts	89%	84%	-5%
src/lib/messagi...nnels/policy.ts	100%	100%	0%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/securit...ntial-filter.ts	93%	93%	0%
src/lib/state/m...ock-identity.ts	95%	95%	0%
src/lib/state/m...lock-storage.ts	97%	97%	0%
src/lib/trace.ts	94%	94%	0%
src/lib/tunnel/services.ts	73%	73%	0%
src/lib/domain/.../connect-env.ts	89%	97%	+8%

---

_{Updated July 23, 2026 00:47 UTC}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/e2e.yaml:
- Around line 5121-5133: Move the runtime audit block that calls
runtimeAudit.auditTestRuntime and formatRuntimeAuditSummary so it executes
before buildTraceTimingResult(). Preserve its existing warning and fallback
summary behavior, ensuring the current run’s semantic phase runtime table is
added before trusted cloud-onboard timing is compared.

In `@test/e2e/support/e2e-operations-workflow-boundary.test.ts`:
- Around line 72-73: Separate the artifact action pin validation from the
pattern validation in the relevant boundary tests around the download
configuration. Mutate only download.uses when testing SHA enforcement and only
download.with!.pattern when testing pattern validation, asserting each rejection
independently so one invalid field cannot satisfy the other check.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2c30bd41-52dc-478c-b408-a10b43211c15

📥 Commits

Reviewing files that changed from the base of the PR and between 1d418f5 and d26fb2c.

📒 Files selected for processing (9)

• .github/workflows/e2e.yaml
• scripts/audit-test-runtime.mts
• test/e2e/README.md
• test/e2e/docs/README.md
• test/e2e/live/onboard-resume.test.ts
• test/e2e/support/e2e-operations-workflow-boundary.test.ts
• test/e2e/support/e2e-scorecard.test.ts
• test/e2e/support/test-runtime-audit.test.ts
• tools/e2e/operations-workflow-boundary.mts

💤 Files with no reviewable changes (1)

• test/e2e/live/onboard-resume.test.ts

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions
Status: Canonical ledger: 0 blocker(s), 1 warning(s), 0 suggestion(s).

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 fewer warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, onboard-resume

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Preserve the invalid-progress artifact for diagnosis

• Location: .github/workflows/e2e.yaml:5136
• Category: correctness
• Problem: The scorecard treats a failed artifact download and an invalid progress artifact as advisory, but it does not identify where invalid progress state is created, why it cannot be fixed at that source, or when this fallback can be removed.
• Impact: A scheduled or manual scorecard can report unavailable runtime timing repeatedly while the malformed producer remains undiagnosed.
• Recommendation: Document the producer-side invalid-state boundary and removal condition, or make the artifact producer fail before upload when it creates an invalid progress summary.
• Verification: Inspect the progress-summary writer and upload action to confirm whether malformed test-progress.json can reach an e2e-* artifact.
• Test coverage: Add a producer-to-upload test that rejects an invalid test-progress.json before the artifact action receives it.
• Evidence: .github/workflows/e2e.yaml adds a continue-on-error artifact download and catches every audit error to emit advisory text. scripts/audit-test-runtime.mts rejects malformed summaries, including missing or unknown phase outcomes. The changed workflow-boundary test verifies scorecard continuation after an invalid progress artifact but does not exercise the producing or upload boundary.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[prekshivyas]

Reviewed exact head 25a1775 against base 1d418f5. Security PASS across all nine review categories. The scorecard remains read-only, actions are immutably pinned, artifact traversal skips links, and missing or invalid progress evidence stays diagnostic-only without hiding the E2E result. Focused validation: 61 tests passed. No actionable findings. Exact-SHA credentialed E2E still requires maintainer authorization.

[cv]

Addressed PRA-1 in signed commit 867f673: the runtime-audit support test now supplies otherwise valid summaries with a missing phase outcome and an unknown phase outcome, and confirms both are rejected. Fresh GitHub checks are running on the new head.