ci(e2e): add rolling nightly runtime history

[cjagwani]

Summary

Scheduled E2E scorecards now compare the current run with up to ten prior completed nightly runs, so runtime regressions and intermittent failures are visible without adding more live terminal noise. The report includes prior median and p95, outcome history, failure streaks, common failed phases, significant phase regressions, and a five-row flake watch.

Related Issue

Part of #7145.

Changes

• Build one bounded, timing-and-outcome-only summary from the current run's validated test-progress.json artifacts.
• Retain scheduled summaries for 14 days and exclude manual runs from the historical baseline.
• Compare the ten slowest current tests with matching prior nightly samples, including phase-level regression findings.
• Rank up to five current tests that both passed and failed in the available nightly window by pass/fail flips, then failure count.
• Add a shared exact-file ZIP reader because both trace timing and runtime history consume GitHub artifact archives; direct extraction would expose path and link handling. The reader is protected by artifact-zip.test.ts and the existing trace artifact integration tests.
• Validate the scorecard workflow boundary and document the nightly comparison contract in test/e2e/README.md.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Focused review of the GitHub artifact boundary confirmed exact root-file matching, no extraction, bounded entries and bytes, rejection of links/encryption/split/ZIP64/duplicates, CRC validation, strict summary schemas, and run-ID provenance.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: test/e2e/README.md
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — focused e2e-support suite: 112/112 passed; final review-focused suite: 20/20 passed; flake-watch suite: 7/7 passed; malformed ZIP cases: 6/6 passed; trace timing integration: 14/14 passed; CLI type-check passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Summary by CodeRabbit

• New Features
  • Scheduled E2E scorecards now include a nightly runtime trend section in the report.
  • Scheduled runs publish a bounded e2e-runtime-summary.json artifact for trend comparisons.
• Documentation
  • Updated scheduled operations guidance to describe the nightly trend tables, baseline sourcing, and regression/flake-watch rules.
• Bug Fixes
  • Improved validation and secure parsing for artifact ZIP inputs used by scorecards.
  • Strengthened runtime-history analysis and reporting for more reliable comparisons.
• Tests
  • Expanded E2E coverage for runtime-history formatting, artifact ZIP reading, and scorecard workflow boundary/contract enforcement.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds E2E runtime history sampling, validated ZIP extraction, nightly regression reporting, scheduled artifact publication, workflow boundary checks, tests, and operational documentation.

Changes

E2E runtime trend pipeline

Layer / File(s)	Summary
Runtime sample collection scripts/audit-test-runtime.mts, test/e2e/support/test-runtime-audit.test.ts	Progress artifacts are validated and grouped into normalized scenario and phase runtime history samples.
Validated artifact ZIP extraction scripts/scorecard/read-artifact-zip.mts, scripts/scorecard/analyze-trace-timing.mts, test/e2e/support/artifact-zip.test.ts, test/helpers/artifact-zip.ts, test/e2e/support/e2e-scorecard.test.ts	A shared ZIP reader validates structure, bounds, metadata, decompression, and CRC integrity; trace summary extraction uses it.
Nightly history analysis scripts/scorecard/analyze-runtime-history.mts, test/e2e/support/e2e-runtime-history.test.ts	Runtime summaries are normalized, persisted, loaded from completed scheduled runs, compared statistically, and formatted with regression findings.
Scorecard workflow publication .github/workflows/e2e.yaml, tools/e2e/*, test/e2e/support/e2e-operations-workflow-boundary.test.ts, test/e2e/support/upload-e2e-artifacts-workflow-boundary.test.ts, test/e2e/README.md	The scorecard includes the nightly trend, uploads the bounded scheduled summary artifact, validates workflow wiring, and documents comparison behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: feature, area: ci, area: e2e, area: observability

Suggested reviewers: prekshivyas, cv

Sequence Diagram(s)

sequenceDiagram
  participant E2EWorkflow
  participant RuntimeAudit
  participant HistoryAnalyzer
  participant GitHubArtifacts
  E2EWorkflow->>RuntimeAudit: read progress artifacts
  RuntimeAudit-->>E2EWorkflow: runtime history samples
  E2EWorkflow->>HistoryAnalyzer: build runtime history
  HistoryAnalyzer->>GitHubArtifacts: load prior scheduled summaries
  GitHubArtifacts-->>HistoryAnalyzer: validated prior summaries
  HistoryAnalyzer-->>E2EWorkflow: markdown and JSON summary
  E2EWorkflow->>GitHubArtifacts: upload scheduled runtime summary

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding rolling nightly runtime history to E2E scorecards.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/e2e-runtime-history

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit f9554ba in the codex/e2e-runtime-hi... branch remains at 96%, unchanged from commit 8a54612 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit f9554ba in the codex/e2e-runtime-hi... branch remains at 80%, unchanged from commit 361ceac in the main branch.

Show a code coverage summary of the most impacted files.

File	main 361ceac	codex/e2e-runtime-hi... f9554ba	+/-
src/lib/platform.ts	89%	84%	-5%
src/lib/state/m...-acquisition.ts	89%	86%	-3%
src/lib/onboard/dashboard.ts	72%	72%	0%
src/lib/inferen...pter-forward.ts	88%	88%	0%
src/lib/messagi...nnels/policy.ts	100%	100%	0%
src/lib/actions...eway-restart.ts	93%	95%	+2%
src/lib/onboard...shboard-port.ts	90%	93%	+3%
src/lib/domain/.../connect-env.ts	89%	97%	+8%
src/lib/onboard...orward-start.ts	89%	98%	+9%
src/lib/onboard...ixed-forward.ts	38%	77%	+39%

---

_{Updated July 24, 2026 20:34 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections match; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/upload-e2e-artifacts-workflow-boundary.test.ts`:
- Around line 141-148: Rename the test case around
validateUploadE2eArtifactsInvocations to describe the rejected behavior:
changing the scorecard upload path to an invalid location must be identified as
a rejection outside its scheduled runtime summary contract.

In `@tools/e2e/upload-e2e-artifacts-workflow-boundary.mts`:
- Around line 387-396: Update the scorecard branch around the scorecard contract
validation to avoid bypassing shared post-upload placement checks. Extract the
common validation for ensuring upload-e2e-artifacts is followed only by
Docker-auth cleanup into a helper, then invoke it for scorecard after validating
its step shape while preserving the existing behavior for other workflows.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e4a6e346-1f0c-4d8d-8d45-18aa925c0740

📥 Commits

Reviewing files that changed from the base of the PR and between a9eb648 and 49fc610.

📒 Files selected for processing (2)

• test/e2e/support/upload-e2e-artifacts-workflow-boundary.test.ts
• tools/e2e/upload-e2e-artifacts-workflow-boundary.mts

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e/upload-e2e-artifacts-workflow-boundary.mts`:
- Around line 262-277: Update validateUploadPlacement to verify that the upload
step occurs after all required artifact-producing steps, in addition to its
existing successor validation. Reuse the workflow step names or
producer-identifying symbols already used by the validator, and add a regression
test that moves the upload before runtime-summary generation and confirms
validation fails.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a1d546d6-6aa3-4e4f-a577-491a7d9e7398

📥 Commits

Reviewing files that changed from the base of the PR and between 49fc610 and 0a77539.

📒 Files selected for processing (2)

• test/e2e/support/upload-e2e-artifacts-workflow-boundary.test.ts
• tools/e2e/upload-e2e-artifacts-workflow-boundary.mts

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/helpers/artifact-zip.ts`:
- Around line 25-50: Mark the entry-name encoding as UTF-8 in both ZIP headers
generated by the artifact helper. Update the general-purpose bit flags written
by the local header construction and the central directory construction, while
preserving the existing compression, checksum, and filename handling.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2d19d93a-a320-494c-9254-91fa2e4ded9b

📥 Commits

Reviewing files that changed from the base of the PR and between 0a77539 and db88a45.

📒 Files selected for processing (3)

• test/e2e/support/artifact-zip.test.ts
• test/e2e/support/e2e-runtime-history.test.ts
• test/helpers/artifact-zip.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/e2e/support/e2e-runtime-history.test.ts

[cjagwani]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[cjagwani]

@apurvvkumaria The final head f9554bac409c1139f1b969b0bc4c950b9135c572 is mergeable and approved; all non-credentialed CI is green and the review ledger has 0 actionable findings. The credentialed E2E approval controller run 30124567776 failed before dispatch with No required-reviewer approval was recorded for approve-credentialed-e2e-for-internal-pr, so the coordination check is still waiting. Could you use the manual run-control-plane maintainer fallback, or restore the protected environment approval, for exact base fd0763437189fae0642fa367a7b546cc0d39d9a9 and targets cloud-onboard, credential-sanitization, and security-posture?

[apurvvkumaria]

Fix-forward dispatched via the manual run-control-plane maintainer fallback for exact head f9554bac409c1139f1b969b0bc4c950b9135c572 and base fd0763437189fae0642fa367a7b546cc0d39d9a9: https://github.com/NVIDIA/NemoClaw/actions/runs/30127477265. I also rechecked the review threads: all CodeRabbit findings are resolved or withdrawn, the advisor ledger has no actionable findings, and no additional code change is needed for the non-blocking docstring warning.

[apurvvkumaria]

The exact-head maintainer fallback completed successfully for f9554bac409c1139f1b969b0bc4c950b9135c572: controller run 30127477265. Evidence verification passed without retry, and the PR gate coordination check is green. I rechecked the current review state: the PR is approved, all actionable review threads are resolved, and there are no remaining questions for @apurvvkumaria. The docstring coverage warning is informational and does not justify another code churn cycle. I also moved the release target from v0.0.95 to the confirmed v0.0.96 queue.