fix(ci): retry confirmed hosted runner loss once

[apurvvkumaria]

Summary

The PR E2E gate now creates one fresh child run when the first child fails only because a standard GitHub-hosted runner disappeared. It recognizes both GitHub's canonical lost-communication annotation and the exact shutdown-log signature observed in the repeated Hermes cancellations. Assertion, timeout, OOM, policy, mixed, custom-runner, and incomplete-evidence failures remain terminal.

Related Issue

Part of #7146. Parent epic: #7140.

Changes

• Require a completed failed first child run, exact workflow identity, complete Jobs and Checks pagination, and either the canonical SHA-bound lost-runner annotation or one exact generic-cancellation annotation plus authenticated terminal shutdown evidence.
• Authenticate the fallback through GitHub's job-log API without forwarding the repository token to storage. Allowlist the signed HTTPS result host, require a strong ETag and exact bounded range, and accept only the terminal runner shutdown/cleanup block bound to the cancelled step and job timestamps. Cap annotation count, per-field bytes, and normalized aggregate evidence.
• Require every non-passing job to match the standard ubuntu-latest GitHub Actions runner, runner group, check, annotation, and terminal step-shape contract. Re-read and fingerprint all run, check, annotation, and optional log evidence before dispatch.
• Create a fresh child run and replacement required check for the one retry. Preserve the first run, check, state, plan, and evidence as linked diagnostic history.
• Revalidate the PR SHA, base SHA, workflow SHA, risk plan, state, and retry history before each privileged mutation.
• Keep coordination checks immutable across base identities so a controller that resumes after a rapid retarget cannot fail the newer base's gate.
• Terminalize the second attempt and clean partial replacement state after failure, cancellation, skipped setup, interruption, or response loss. The required observer follows the latest exact check.
• Serialize gate mutations by repository, PR number, PR SHA, and base SHA. Keep up to 100 pending exact-identity mutations in GitHub's FIFO queue without canceling their owner.
• Bound each child wait at 140 minutes, the controller at 330 minutes, the observer at 358 minutes, and the required job at 360 minutes.
• Document the strict retry criteria, lineage, cleanup behavior, and timeout envelope in test/e2e/README.md.
• Keep merged test(e2e): add semantic live progress phases #7100 progress telemetry and ci(e2e): record runner comparison telemetry #7399 resource telemetry unchanged. This PR does not add a second sampler or retry final-main runs.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: an independent 9-category audit passed at 697d78e5f; it verified annotation and log authentication, token isolation, bounded transport, spoof rejection, immutable retry lineage, two-pass evidence revalidation, cross-base check ownership, cleanup, timeout arithmetic, and FIFO mutation ownership with no unresolved finding.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: test/e2e/README.md
• Agent: Codex Desktop
• PR: fix(ci): retry confirmed hosted runner loss once #7423

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable
• Station profile/scenario: Not applicable
• Result: Not applicable
• Supporting evidence: Not applicable

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 14 PR-gate integration files passed 244 tests at 697d78e5f; the focused classifier/transport suites passed 72 tests; the test-only review fix passed all 26 controller tests at ac430dcf7; the documented fallback lifecycle passed docs validation and all 39 classifier tests at d1f824728; CLI and JavaScript type checking, changed-file hooks, source-shape, test-size, Biome, YAML, Markdown, gitleaks, and diff checks passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — targeted controller coverage is authoritative; GitHub CI will run the repository gate.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings
• Doc pages follow the style guide (doc changes only) — no docs/ page changed
• New doc pages include SPDX header and frontmatter (new pages only) — no new doc page

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• New Features
  • Added a one-time automated PR E2E retry for hosted-runner loss, including dedicated retry and abandon lifecycle steps with evidence download/verification.
  • Updated PR E2E gate coordination to run retry-sensitive paths only on the first workflow attempt using attempt-aware, per-PR/per-SHA concurrency isolation.
• Bug Fixes
  • Hardened gate/approval execution to prevent cross-run collisions and incorrect retry/authorization, with improved fail-closed behavior.
  • Increased relevant time budgets and tightened “superseded/cancel” handling.
• Documentation
  • Expanded PR E2E gate flow documentation for retries, evidence rules, and terminal outcomes.
• Tests
  • Expanded end-to-end coverage for runner-loss classification, retry lifecycle, authorization, and command parsing edge cases.

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit e06d546 in the codex/7146-runner-lo... branch remains at 96%, unchanged from commit e065f57 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit e06d546 in the codex/7146-runner-lo... branch remains at 80%, unchanged from commit 21e60ae in the main branch.

Show a code coverage summary of the most impacted files.

File	main 21e60ae	codex/7146-runner-lo... e06d546	+/-
src/lib/agent/r...ime-manifest.ts	100%	100%	0%
src/lib/messagi...nnels/policy.ts	100%	100%	0%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/securit...ntial-filter.ts	93%	93%	0%
src/lib/trace.ts	94%	94%	0%
src/lib/state/g...way-registry.ts	94%	95%	+1%
src/lib/platform.ts	84%	89%	+5%

---

_{Updated July 23, 2026 14:44 UTC}

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR adds attempt-aware PR E2E coordination, exact revision concurrency keys, expanded timeouts, trusted hosted-runner-loss classification, and a one-time retry flow with isolated state/evidence slots and terminal cleanup.

Changes

PR E2E coordination and retry flow

Layer / File(s)	Summary
Attempt-aware workflow coordination .github/workflows/pr-e2e-gate.yaml, test/pr-e2e-gate-workflow.test.ts, tools/e2e/pr-e2e-required.mts	Workflow jobs now use first-attempt guards, PR/head/base SHA concurrency keys, non-canceling queues, longer budgets, and coordinated runner-loss retry steps.
Controller retry protocol and state binding tools/e2e/pr-e2e-gate.mts, test/pr-e2e-gate-command.test.ts	The controller adds retry and abandonment commands, slot-scoped paths, deterministic check reservations, bound-state validation, retry dispatch, and cleanup handling.
Runner-loss evidence validation tools/e2e/pr-e2e-gate.mts, test/pr-e2e-gate-runner-loss-classifier.test.ts	Workflow jobs and annotations are validated against runner identity, attempt, workflow SHA, and trusted hosted-runner-loss markers.
Retry and lifecycle validation test/pr-e2e-gate-runner-loss-retry.test.ts, test/pr-e2e-gate-lifecycle.test.ts, test/pr-e2e-gate*.test.ts	Tests cover retry authorization, state isolation, failure finalization, lifecycle identity checks, terminal controller failures, and completed-check abandonment behavior.
Gate contract documentation test/e2e/README.md	Documentation describes exact-revision concurrency, approval authorization, trusted retry classification, and updated evidence budgets.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• NVIDIA/NemoClaw#6662: Extends the same PR E2E controller and workflow with runner-loss retry and attempt-aware coordination.
• NVIDIA/NemoClaw#7342: Updates the protected approval path further extended by this PR.
• NVIDIA/NemoClaw#7392: Adds related hosted-runner-loss evidence and retry plumbing.

Suggested labels: bug-fix

Suggested reviewers: cv, jyaunches

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 1.41% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: a one-time retry for confirmed hosted runner loss in CI.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7146-runner-loss-retry-refresh

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Review Advisor — Blocking findings reported

Advisor assessment: Blockers require maintainer review
Next action: Review the blockers below.
Findings: 1 blocker · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 1 blocker · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 3 warnings · 3 suggestions

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

Blockers

PRA-1 Blocker — Use supported GitHub Actions concurrency configuration

• Location: .github/workflows/pr-e2e-gate.yaml:41
• Category: architecture
• Problem: The workflow adds `queue: max` to concurrency mappings. GitHub Actions concurrency supports `group` and `cancel-in-progress`; it does not provide a `queue` setting.
• Impact: The workflow cannot guarantee the documented FIFO retention of pending jobs. If GitHub rejects the workflow schema or ignores the unknown key, E2E gate and approval lifecycle behavior differs from the documented contract.
• Fix: Remove `queue: max` and implement only behavior supported by GitHub Actions concurrency, or provide a checked-in mechanism that enforces the required queue policy.
• Verification: Inspect GitHub Actions workflow syntax documentation for permitted `concurrency` properties, then validate the workflow with the repository's checked-in workflow validation evidence.
• Test coverage: Add a workflow-contract test that rejects unsupported concurrency keys and verifies the supported concurrency behavior for exact PR/head/base identities.
• Evidence: `.github/workflows/pr-e2e-gate.yaml` adds `queue: max` to the `initialize` concurrency mapping at line 41 and to other changed concurrency mappings. `test/e2e/README.md` states that `queue: max` keeps pending jobs in FIFO order, but the workflow schema does not define this property.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/pr-e2e-gate-runner-loss-retry.test.ts`:
- Around line 924-937: Remove the test-body branch around finishPrGate in the
parameterized test and derive the expected rejection or resolution assertion
directly from the it.each data, preserving the success-specific error
expectation. Also replace the conditional branches in the helper paths guarded
by options.jobPages and options.createRetryStateDirectory with equivalent
expressions, without changing behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b07caa2c-3064-46ed-bec9-3770472e98a7

📥 Commits

Reviewing files that changed from the base of the PR and between 21e60ae and a419984.

📒 Files selected for processing (13)

• .github/workflows/pr-e2e-gate.yaml
• test/e2e/README.md
• test/pr-e2e-gate-command.test.ts
• test/pr-e2e-gate-fork-skip.test.ts
• test/pr-e2e-gate-internal-approval.test.ts
• test/pr-e2e-gate-lifecycle.test.ts
• test/pr-e2e-gate-runner-loss-classifier.test.ts
• test/pr-e2e-gate-runner-loss-retry.test.ts
• test/pr-e2e-gate-workflow.test.ts
• test/pr-e2e-gate.test.ts
• test/pr-e2e-required.test.ts
• tools/e2e/pr-e2e-gate.mts
• tools/e2e/pr-e2e-required.mts

[coderabbitai]

🧹 Nitpick comments (1)

test/pr-e2e-gate.test.ts (1)

773-779: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Scope the negative assertion to the newer-base gate.

Checking that no request uses PATCH does not directly prove that the check introduced at Line 755 was untouched; it can miss another mutation verb and reject unrelated PATCH requests. Assert that no mutating request targets that specific check, while retaining the separate POST assertion for the original base SHA.

As per path instructions, tests should prioritize behavioral confidence over implementation lock-in.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/pr-e2e-gate.test.ts` around lines 773 - 779, Update the negative
assertion in the newer-base gate test to verify that no mutating request targets
the check identified by its newer-base SHA, rather than globally rejecting PATCH
requests. Retain the separate POST assertion for the original base SHA and
preserve behavioral coverage without depending on a particular mutation verb.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/pr-e2e-gate.test.ts`:
- Around line 773-779: Update the negative assertion in the newer-base gate test
to verify that no mutating request targets the check identified by its
newer-base SHA, rather than globally rejecting PATCH requests. Retain the
separate POST assertion for the original base SHA and preserve behavioral
coverage without depending on a particular mutation verb.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f553129c-1158-4cd5-90cc-2372cdc4496f

📥 Commits

Reviewing files that changed from the base of the PR and between a419984 and 357e4b8.

📒 Files selected for processing (4)

• test/e2e/README.md
• test/pr-e2e-gate-runner-loss-retry.test.ts
• test/pr-e2e-gate.test.ts
• tools/e2e/pr-e2e-gate.mts

🚧 Files skipped from review as they are similar to previous changes (3)

• test/pr-e2e-gate-runner-loss-retry.test.ts
• test/e2e/README.md
• tools/e2e/pr-e2e-gate.mts

[apurvvkumaria]

Live runner-loss validation and trusted-base boundary

Exact PR head: d1f82472863175ed1fcde56d9a3445c135a078f6.

All ordinary required CI passed. In E2E child 29999433179, cloud onboarding, credential sanitization, and OpenClaw security posture passed. The sole non-passing job was Hermes security posture 89180965578, which lost a standard ubuntu-latest GitHub-hosted runner; no test assertion failed.

The live evidence matches this PR's strict fallback contract:

• exactly one non-passing job and one .github failure annotation: The operation was canceled.
• standard GitHub Actions runner/group/label identity
• one cancelled live-test step, with later cleanup skipped
• authenticated terminal log sequence: runner shutdown signal → operation cancelled → orphan cleanup
• timestamps bound to the cancelled step and job; exact single-LF EOF

No retry was expected in this PR run because the authoritative coordinator 29999405614 intentionally checked out trusted main at 21e60ae287e8c2a184f71406ac8b418f046330d1 via github.workflow_sha. The privileged pull_request_target / workflow_run controller must not execute this PR's unmerged code. Therefore the old main classifier terminalized the check; the new retry path can first run live only after merge.

This run validates the exact real-world evidence shape captured by the new tests. The remaining non-success E2E result is hosted-runner infrastructure and requires either maintainer acceptance for this PR or an explicit trusted-base rerun; it is not evidence to relax the classifier.