ci(e2e): report runner wait and execution time

[cjagwani]

Summary

The E2E scorecard now separates each job's runner queue time from its execution time. The report identifies whether a slow run waited for capacity or spent time executing tests without exposing raw runner labels.

Related Issue

Part of #7145.

Changes

• Add a table for the ten jobs with the longest combined queue and execution duration.
• Report queue time, execution time, outcome, and normalized standard, larger, or unknown runner class.
• Treat any self-hosted label set as unknown and never expose raw runner labels.
• Preserve the existing needs fallback when GitHub's jobs API is unavailable.
• Document the scorecard output and add focused summary and coordinator coverage.
• Refresh from base SHA 623c87394fd461be3ebcac100031be312663b588 with signed merge commit 922d02022ab360436aa08f8a3ae5010daa8becdc. The refreshed patch hash cf1d06b1321eaa171765499ea1def712706bb9da2398c9b2ebbcfd68b69d5eea matches the pre-refresh patch.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop reviewed all nine security categories at PR SHA 922d02022ab360436aa08f8a3ae5010daa8becdc against base SHA 623c87394fd461be3ebcac100031be312663b588. All categories passed: the change emits normalized runner classes, does not emit raw labels, and does not change runner selection or test execution.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: Reviewed test/e2e/README.md and all changed scorecard text and test titles at PR SHA 922d02022ab360436aa08f8a3ae5010daa8becdc against base SHA 623c87394fd461be3ebcac100031be312663b588. The README documents queue and execution timing, normalized runner classes, and runner-label redaction. No additional user-facing docs are needed. Focused E2E-support tests passed 29/29. npm run docs passed with 0 errors and 2 warnings.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project e2e-support test/e2e/support/e2e-scorecard.test.ts test/e2e/support/e2e-scorecard-coordinator.test.ts: 2 files and 29 tests passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Scorecard job summaries now calculate queue and execution timing, classify runner sizes, and render the ten longest jobs in a markdown table. Tests cover formatting, truncation, fallback summaries, and runner-label redaction. E2E reporting documentation describes the updated output and runtime baseline.

Changes

Scorecard observability

Layer / File(s)	Summary
Job timing model and calculation scripts/scorecard/summarize-jobs.mts	JobSummary now includes timing rows; API jobs produce bounded queue/execution durations and normalized runner classes, while fallback summaries return an empty array.
Timing table rendering and coverage scripts/scorecard/coordinate-scorecard.mts, test/e2e/support/e2e-scorecard*.test.ts, test/e2e/README.md	Scorecard markdown renders formatted timing rows, and tests and documentation cover runner classification, truncation, fallback output, and reporting requirements.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant APIJobs
  participant summarizeJobs
  participant renderSummaryLines
  participant SlackReport
  APIJobs->>summarizeJobs: Provide job timestamps and labels
  summarizeJobs->>summarizeJobs: Calculate and classify timing rows
  summarizeJobs-->>renderSummaryLines: Return timingRows
  renderSummaryLines->>SlackReport: Render timing markdown table

Loading

Suggested labels: area: ci, area: e2e, area: docs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: reporting runner wait time and execution time in E2E scorecard output.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/e2e-job-timing

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 922d020 in the codex/e2e-job-timing branch remains at 96%, unchanged from commit 623c873 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 922d020 in the codex/e2e-job-timing branch remains at 80%, unchanged from commit 623c873 in the main branch.

Show a code coverage summary of the most impacted files.

File	main 623c873	codex/e2e-job-timing 922d020	+/-
src/lib/domain/.../connect-env.ts	97%	89%	-8%
src/lib/inferen...del-registry.ts	100%	97%	-3%
src/lib/adapter.../auth-config.ts	96%	96%	0%
src/lib/inferen...lama-version.ts	100%	100%	0%
src/lib/messagi...nnels/policy.ts	100%	100%	0%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/securit...ntial-filter.ts	93%	93%	0%
src/lib/state/m...ock-identity.ts	95%	95%	0%
src/lib/platform.ts	84%	89%	+5%
src/lib/inferen...time-context.ts	87%	93%	+6%

---

_{Updated July 25, 2026 00:32 UTC}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/scorecard/summarize-jobs.mts`:
- Around line 103-110: Update normalizeRunnerClass to check normalized labels
for "self-hosted" before checking "ubuntu-latest", so jobs with both labels
return "unknown"; preserve the existing standard, larger, and empty-label
classifications otherwise.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 02d24688-9a22-4453-a84b-f6412937961c

📥 Commits

Reviewing files that changed from the base of the PR and between 88b3e35 and 45a7353.

📒 Files selected for processing (5)

• scripts/scorecard/coordinate-scorecard.mts
• scripts/scorecard/summarize-jobs.mts
• test/e2e/README.md
• test/e2e/support/e2e-scorecard-coordinator.test.ts
• test/e2e/support/e2e-scorecard.test.ts

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

test/e2e/support/artifact-zip.test.ts (1)

102-116: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Cover a valid payload with a mismatched CRC.

This only corrupts compressed bytes; add a case that preserves valid deflate data but changes both local and central-directory CRC fields, then expects null. That exercises the checksum-validation contract directly.

Proposed test addition

+    const crcMismatchArchive = Buffer.from(archive);
+    const centralDirectoryOffset = crcMismatchArchive.readUInt32LE(
+      crcMismatchArchive.length - 6,
+    );
+    crcMismatchArchive.writeUInt32LE(
+      crcMismatchArchive.readUInt32LE(14) ^ 0xffffffff,
+      14,
+    );
+    crcMismatchArchive.writeUInt32LE(
+      crcMismatchArchive.readUInt32LE(centralDirectoryOffset + 16) ^ 0xffffffff,
+      centralDirectoryOffset + 16,
+    );
+    expect(
+      readValidatedArtifactZipEntry(crcMismatchArchive, "summary.json", { maxBytes: 1_024 }),
+    ).toBeNull();

As per path instructions, review tests for “behavioral confidence rather than implementation lock-in”; the supplied stack identifies CRC validation as part of this reader’s contract.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/support/artifact-zip.test.ts` around lines 102 - 116, Add a
CRC-mismatch case to the test for readValidatedArtifactZipEntry using the same
valid deflated archive, preserving its compressed payload while changing the CRC
fields in both the local header and central directory. Assert that reading
summary.json with the existing maxBytes option returns null, directly covering
checksum validation without depending on implementation details.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/scorecard/analyze-runtime-history.mts`:
- Around line 335-336: Guard the current-summary creation and file write in the
runtime-history flow around createRuntimeSummary and fs.writeFileSync so invalid
input or IO failures are caught without propagating out of buildRuntimeHistory
or aborting the scorecard. Preserve the upload step’s expected output-file
contract: keep the successful write on the happy path, and ensure the existing
fallback/error handling does not leave a required upload file unexpectedly
missing.

---

Nitpick comments:
In `@test/e2e/support/artifact-zip.test.ts`:
- Around line 102-116: Add a CRC-mismatch case to the test for
readValidatedArtifactZipEntry using the same valid deflated archive, preserving
its compressed payload while changing the CRC fields in both the local header
and central directory. Assert that reading summary.json with the existing
maxBytes option returns null, directly covering checksum validation without
depending on implementation details.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c5469a9b-02e0-4c3c-be7f-d77ee60dcb2d

📥 Commits

Reviewing files that changed from the base of the PR and between 048c802 and 436e5a3.

📒 Files selected for processing (12)

• .github/workflows/e2e.yaml
• scripts/audit-test-runtime.mts
• scripts/scorecard/analyze-runtime-history.mts
• scripts/scorecard/analyze-trace-timing.mts
• scripts/scorecard/read-artifact-zip.mts
• test/e2e/README.md
• test/e2e/support/artifact-zip.test.ts
• test/e2e/support/e2e-operations-workflow-boundary.test.ts
• test/e2e/support/e2e-runtime-history.test.ts
• test/e2e/support/test-runtime-audit.test.ts
• test/e2e/support/upload-e2e-artifacts-workflow-boundary.test.ts
• tools/e2e/upload-e2e-artifacts-workflow-boundary.mts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/e2e/README.md

[senthilr-nv]

Reviewed Files changed at PR SHA 922d020 against base SHA 623c873. I approve this exact revision for squash merge after all required checks and both exact-pair E2E gates pass.