fix(ci): trust exact-head Hermes swap setup

[apurvvkumaria]

Summary

Stages exact-head Hermes swap provisioning in the trusted main workflow before checkout. This phase keeps the reviewed candidate helper only so the workflow change can validate under the older trusted workflow currently on main; after this lands, a focused follow-up will remove that compatibility helper and complete the #7391 trust-boundary fix.

Related Issue

Part of #7145. Security fix-forward for #7391.

Changes

• Provision at least 32 GiB of usable swap before checkout for the eight exact-head, GitHub-hosted Hermes lanes that rebuild or exercise the Hermes image.
• Require the canonical repository, workflow_dispatch, refs/heads/main, an exact trusted workflow SHA, the generated controller matrix, and explicit trusted Hermes target selection before privileged setup can run.
• Use a fixed trusted program with absolute executables, a clean shell environment, a root-owned randomized file, a 16 GiB disk reserve, bounded activation-visibility polling, and fail-closed cleanup.
• Temporarily retain the exact-head candidate helper for rollout compatibility, but allocate 32 GiB + 4,096 bytes so formatting still leaves the required 32 GiB usable. Once the trusted workflow is on main, the helper observes sufficient swap and exits before creating its fixed file.
• Pin the eligible job inventory, ordering, controller dependency, step shape, environment, and exact program bytes with a workflow-boundary validator.
• Add mutation and executable cleanup tests proving candidate changes cannot alter the root command and partial failures do not delete active swap.
• Document the trusted boundary and hosted-runner capacity behavior in test/e2e/README.md.

The larger-runner route cannot directly replace this fallback for exact-head validation because the trusted controller intentionally refuses candidate-selected runner labels. Pre-merge live validation executes the older trusted workflow from main, so this rollout phase retains the reviewed candidate helper while adding the trusted pre-checkout step. After this phase lands, a focused follow-up will delete the helper and its tests; that follow-up's exact-head E2E will execute the trusted setup from main.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: Internal E2E maintainer behavior is documented in test/e2e/README.md; no user-facing behavior or docs/ page changes.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent focused re-review found no blocker after trusted target selection, controller sequencing, exact step-shape validation, bounded sizing, and active-swap-safe failure cleanup were enforced at 947375403; 5874616a3 and 1ac224771 add the requested reuse and successful-provisioning regression tests without changing runtime behavior.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: test/e2e/README.md documents the two-phase rollout, trusted pre-checkout boundary, temporary exact-head compatibility helper, 32 GiB usable / 32 GiB + 4,096-byte allocation distinction, five bounded activation observations, fail-closed cleanup, protected lanes, and required follow-up removal; no canonical user-doc change is needed.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — after the rollout, delayed-visibility, and supported util-linux syntax fixes: trusted boundary 11/11, live helper 52/52, and E2E workflow contracts 42/42 passed independently; project membership, source-shape, repository checks, CLI build/typecheck, docs validation, diff validation, and push hooks passed.
• Applicable broad gate passed — npm run checks, npm run typecheck:cli, npm run source-shape:check, npm run test:titles:check, npm run test:projects:check, and npm run test-size:check passed.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• New Features
  • Added “trusted Hermes E2E swap” provisioning for eligible Hermes end-to-end jobs, gated to trusted main-branch dispatches and run before code checkout.
• Documentation
  • Updated the E2E README with trusted-swap routing, disk/swap planning, and cleanup fail-closed semantics.
• Bug Fixes
  • Enforced fail-closed workflow boundaries to prevent candidate code from influencing privileged swap provisioning or step ordering.
• Tests
  • Added a workflow-boundary suite for trusted Hermes swap stability/cleanup behavior and updated E2E watch-trigger coverage; adjusted live Vitest invocation tests.
• Refactor
  • Simplified the live Vitest invocation flow by removing Hermes swap provisioning logic from the CLI path.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR moves Hermes E2E swap provisioning from live Vitest execution into trusted workflow steps, adds strict workflow and helper-source validation, updates protected jobs and documentation, and adds cleanup, drift, and watch-trigger tests.

Changes

Trusted Hermes E2E swap

Layer / File(s)	Summary
Trusted swap script and validators tools/e2e/trusted-hermes-swap-workflow-boundary.mts	Adds fail-closed swap provisioning, cleanup handling, workflow validation, and candidate-helper source validation.
Protected workflow integration .github/workflows/e2e.yaml, tools/e2e/workflow-boundary.mts	Adds the trusted swap step to protected Hermes jobs and validates its placement, guards, environment, and script content.
Live Vitest provisioning removal and documentation tools/e2e/live-vitest-invocation.mts, test/e2e/support/live-vitest-invocation.test.ts, test/e2e/README.md	Removes Hermes swap provisioning from live Vitest execution and documents the trusted workflow contract.
Boundary tests and watch integration test/e2e/support/trusted-hermes-swap-workflow-boundary.test.ts, test/helpers/vitest-watch-triggers.ts, test/vitest-watch-triggers.test.ts, ci/source-shape-test-budget.json	Adds simulated failure and workflow-drift coverage and registers the new boundary test for workflow-triggered runs.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WorkflowDispatch
  participant TrustedHermesSwap
  participant Checkout
  participant HermesE2ETests
  WorkflowDispatch->>TrustedHermesSwap: validate provenance and runner
  TrustedHermesSwap->>TrustedHermesSwap: provision and activate swap
  TrustedHermesSwap->>Checkout: allow checkout after successful setup
  Checkout->>HermesE2ETests: run Hermes E2E jobs

Loading

Possibly related PRs

• NVIDIA/NemoClaw#7391: Refactors Hermes E2E swap provisioning from live Vitest helpers into trusted workflow steps and boundary validation.

Suggested labels: area: ci, area: e2e, integration: hermes, bug-fix, v0.0.93

Suggested reviewers: cv, prekshivyas, jyaunches

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately highlights the CI change to trusted exact-head Hermes swap setup.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/e2e-trusted-swap-boundary

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 763fc6a in the codex/e2e-trusted-sw... branch remains at 96%, unchanged from commit f1161a5 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 763fc6a in the codex/e2e-trusted-sw... branch remains at 80%, unchanged from commit f1161a5 in the main branch.

Show a code coverage summary of the most impacted files.

File	main f1161a5	codex/e2e-trusted-sw... 763fc6a	+/-
src/lib/domain/.../connect-env.ts	97%	89%	-8%
src/lib/state/g...way-registry.ts	95%	94%	-1%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/state/m...-acquisition.ts	87%	89%	+2%

---

_{Updated July 24, 2026 02:17 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / medium confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 3 warnings · 1 suggestion
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 3 more warnings, 1 more suggestion.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

2 optional E2E recommendations

• hermes-e2e
• mcp-bridge

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Addressed PRA-1 in 5874616. The executable fake-command harness now supplies exactly 32 GiB of existing active swap, verifies a successful exit, and pins the command log to the read-only /mnt ownership check plus active-capacity query—so no mkdir, mktemp, fallocate, mkswap, activation, swapoff, or cleanup command can run. The focused file passes all 10 tests.

[apurvvkumaria]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[apurvvkumaria]

Final exact-head status for f2a6290ac5e981beb24640f636eadb217df30749:

• All ordinary CI, self-hosted checks, CodeQL, CodeRabbit, documentation receipt, Terra, and Nemotron completed successfully. The canonical advisor has 0 blockers, 0 warnings, and 0 suggestions.
• Independent final security review is clear: 0 blockers, 0 warnings, 0 suggestions; 75/75 focused tests passed.
• The supported manual control-plane fallback 30046092989 dispatched exact-head child 30046124587 for head f2a6290a, base ac5579e9, plan 96ac3a08….
• Credential sanitization, cloud onboarding, and OpenClaw security posture passed. Hermes security posture was canceled only after the hosted runner reported a shutdown signal; there was no test assertion or swap-provisioning failure.
• The controller correctly classified child-cancelled and attempted its single permitted retry, but an immediate check read-after-write consistency race caused PR gate check does not authorize this runner-loss retry; no retry child was created.

No actionable review threads remain. The branch is clean and the head is GitHub Verified. The formal E2E coordination check remains red because of the hosted-runner cancellation/retry-control-plane failure, so this comment records evidence rather than claiming the gate passed.

[prekshivyas]

Security and correctness review of exact head d347bc2: PASS across secrets, input validation, authorization, dependencies, logging, data protection, workflow configuration, security tests, and system/TOCTOU controls. The privileged swap program is fixed in the trusted main workflow before candidate checkout, bound to the exact dispatch/workflow SHA and ephemeral GitHub-hosted Linux x64 runner, uses root-owned randomized paths with capacity bounds, and fails closed without deleting active swap on uncertain cleanup. Focused workflow-boundary and harness tests pass locally; no unresolved review threads or actionable automated findings. Merge remains gated on the refreshed exact-head CI/E2E result.

[prekshivyas]

Maintainer CI waiver for exact head d347bc2553cd4b786c63536338e510b8eadf1750:

I accept the pending/non-success E2E / PR Gate Coordination check for this dependency PR only. The changed behavior is the trusted pre-checkout workflow program itself, so it cannot execute until this revision is present on main. All ordinary exact-head CI, CodeQL, CodeRabbit, both review-advisor lanes, DCO, docs receipt, and self-hosted checks passed; the independent exact-head security review is approved; 80 focused E2E-support tests and 5 watch-trigger integration tests passed locally; all commits are GitHub Verified; and no actionable review threads remain.

This waiver does not waive downstream live validation. After merge, the affected hosted Hermes E2E will be run from the trusted main workflow before #7449 or #7450 is approved or merged. Follow-up/scope: #7145.

[prekshivyas]

Downstream exact-head evidence from #7447 confirms this dependency is active. Child run 30055699724 exercised two fresh hosted-runner attempts for head dd1f7a8f42edc5508fb981b7ad6c924c2d8f8cd5. In both attempts, both hermes-e2e and Hermes security-posture stopped before Vitest with the same host swap signature: 0 of 34359738368 bytes active, followed by Operation not permitted while cleaning /mnt/nemoclaw-hermes-e2e.swap. That is four identical pre-test failures across fresh runners. No Hermes product assertion ran or failed. Once #7444 is on main, I will rerun the #7447 exact-head Hermes lanes through the trusted workflow.

[prekshivyas]

Exact-head review complete for e9691df. The staged rollout keeps the new trusted pre-checkout boundary, retains the existing exact-head helper only until the trusted workflow reaches main, and corrects the allocation to 32 GiB + 4,096 bytes so at least 32 GiB remains usable. Focused boundary/helper/workflow tests, CLI build/typecheck, source-shape, project membership, docs validation, hooks, DCO, and commit verification pass. No code or security blocker found; final approval remains contingent on refreshed CI and exact-base E2E.

[prekshivyas]

Exact-head review complete for 780d772. The live failure showed delayed swapon --show visibility after successful activation. Both the temporary compatibility helper and final trusted workflow now make five bounded one-second observations, require the exact swap name and capacity, never reactivate, and treat successful activation as potentially active during fail-closed cleanup. Regression suites pass 52/52, 11/11, and 42/42; typecheck, source-shape, project membership, docs, DCO, hooks, and commit verification pass. No code or security blocker found; merge remains contingent on refreshed CI and exact-base E2E.

[prekshivyas]

Exact-head hosted result for 780d772a24bc92817cc38fc4dab1152dd5742c72:

• Supported manual control-plane child: 30060255442
• Hermes security-posture job: 89380356291
• Exact plan: f34ad6221606e469f01e26d9eef939d24de83beb8bedef5914f53d8bb9d05b1a

The new bounded observation path executed. swapon returned success, then all five one-second observations failed to see either the exact swap name/capacity:

Hermes E2E swap provisioning failed: required swap was not visible after 5 attempts

The job stopped before Vitest, so no Hermes product assertion ran or failed. This result does not support a short read-after-activation timing race on the current ubuntu-latest host; extending the wait is not evidence-based.

The repository variable E2E_LARGER_RUNNER_LABEL is currently unset. This account cannot enumerate repository or organization runner labels (actions/runners returns 403), so selecting or granting the required larger runner needs runner-group administration. Downstream #7447 has exact-head local arm64 PASS evidence for both the functional Hermes target and dedicated security-posture mode; the remaining hosted boundary is runner capacity/access.

[prekshivyas]

Re-reviewed exact head 763fc6a. The util-linux failure was caused by unsupported swapon --output syntax being parsed as --output-all. This head uses the supported --show=NAME/--show=SIZE forms in both rollout paths, and the tightened harnesses reject the old syntax. Focused tests pass (52 + 11 + 42), with type-check, repository guards, and docs validation green. Independent maintainer review remains requested because I pushed the fix.