Skip to content

ci(hermes): add swap for image exports#7380

Merged
ericksoa merged 1 commit into
mainfrom
codex/stabilize-hermes-image-export
Jul 22, 2026
Merged

ci(hermes): add swap for image exports#7380
ericksoa merged 1 commit into
mainfrom
codex/stabilize-hermes-image-export

Conversation

@ericksoa

@ericksoa ericksoa commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • provision the established bounded 32 GiB swap before both hosted Hermes image exports
  • retain memory, disk, swap, and Docker usage diagnostics around the resource boundary
  • enforce the swap size, commands, and build ordering in the reusable-workflow contract

Why

Current main completed all 63 Hermes Dockerfile steps, then received runner shutdown during final layer export twice: attempt 1 and failed-only retry. The repository already documents and uses this 32 GiB swap mitigation for Hermes rebuild exports. This applies the same bounded mitigation to the only two reusable-workflow jobs that export Hermes images. It does not change the Dockerfile, image contents, or runtime behavior.

Verification

  • CI=1 npx vitest run --project e2e-support test/e2e/support/sandbox-images-workflow-boundary.test.ts test/e2e/support/hermes-secret-boundary-workflow.test.ts --reporter=verbose (15 passed)
  • direct sandbox-image workflow validator returned no errors
  • Biome check passed
  • git diff --check passed

Signed-off-by: Aaron Erickson aerickson@nvidia.com

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Copilot AI review requested due to automatic review settings July 22, 2026 11:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The workflow adds a reusable 32G swap setup before both Hermes image exports. The workflow validator now checks its presence, commands, uniqueness, and ordering, with E2E coverage and updated documentation.

Changes

Hermes export swap

Layer / File(s) Summary
Provision swap before Hermes exports
.github/workflows/sandbox-images-and-e2e.yaml, test/e2e/README.md
Adds reusable 32G swap provisioning before Hermes image work and documents the updated behavior.
Validate swap workflow contract
tools/e2e/sandbox-images-workflow-boundary.mts, test/e2e/support/sandbox-images-workflow-boundary.test.ts
Validates swap-step uniqueness, required commands, ordering, and activation for both Hermes export jobs, with negative E2E coverage.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#7332: Extends the same workflow-boundary validator with additional contract checks.

Suggested labels: integration: hermes, area: ci, area: e2e

Suggested reviewers: copilot, cv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding swap setup for Hermes image exports.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/stabilize-hermes-image-export

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 36b15e1 in the codex/stabilize-herm... branch remains at 96%, unchanged from commit 091ca29 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 36b15e1 in the codex/stabilize-herm... branch remains at 80%, unchanged from commit 091ca29 in the main branch.

Show a code coverage summary of the most impacted files.
File main 091ca29 codex/stabilize-herm... 36b15e1 +/-
src/lib/state/m...-acquisition.ts 89% 84% -5%
src/lib/agent/r...ime-manifest.ts 100% 100% 0%
src/lib/sandbox...rce-identity.ts 87% 87% 0%
src/lib/state/m...ock-identity.ts 95% 95% 0%
src/lib/state/m...lock-storage.ts 97% 97% 0%
src/lib/tunnel/services.ts 73% 73% 0%
src/lib/state/g...way-registry.ts 94% 95% +1%

Updated July 22, 2026 12:05 UTC

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/README.md`:
- Around line 413-420: Update the rebuild fixture sentence to explicitly state
that it verifies a 32 GiB swap-file floor on GitHub Actions, rather than saying
it “verifies that floor.” Keep the existing trusted control-plane and
main-workflow conditions unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6cf2fe55-c2f0-4fa8-8c03-7dc2e004c303

📥 Commits

Reviewing files that changed from the base of the PR and between 091ca29 and 36b15e1.

📒 Files selected for processing (4)
  • .github/workflows/sandbox-images-and-e2e.yaml
  • test/e2e/README.md
  • test/e2e/support/sandbox-images-workflow-boundary.test.ts
  • tools/e2e/sandbox-images-workflow-boundary.mts

Comment thread test/e2e/README.md
Comment on lines +413 to +420
The two Hermes rebuild jobs and both reusable-workflow Hermes image exporters
add a bounded 32 GiB swap file on their ephemeral hosted runners before the
memory-heavy image build. The rebuild fixture verifies that floor and
provisions the same swap file on GitHub Actions when a trusted control-plane
run uses the workflow definition from `main`. Those paths build large Hermes
image layers and can otherwise exhaust the runner's default memory and swap
during Docker layer export. Other E2E jobs keep the standard runner memory
configuration.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Clarify the swap-floor sentence.

“Verifies that floor” is ambiguous. Name the contract explicitly so the documentation matches the validator:

Proposed wording
-The rebuild fixture verifies that floor and provisions the same swap file
+The rebuild fixture verifies the 32 GiB swap floor and provisions the same swap file
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
The two Hermes rebuild jobs and both reusable-workflow Hermes image exporters
add a bounded 32 GiB swap file on their ephemeral hosted runners before the
memory-heavy image build. The rebuild fixture verifies that floor and
provisions the same swap file on GitHub Actions when a trusted control-plane
run uses the workflow definition from `main`. Those paths build large Hermes
image layers and can otherwise exhaust the runner's default memory and swap
during Docker layer export. Other E2E jobs keep the standard runner memory
configuration.
The two Hermes rebuild jobs and both reusable-workflow Hermes image exporters
add a bounded 32 GiB swap file on their ephemeral hosted runners before the
memory-heavy image build. The rebuild fixture verifies the 32 GiB swap floor and provisions the same swap file on GitHub Actions when a trusted control-plane
run uses the workflow definition from `main`. Those paths build large Hermes
image layers and can otherwise exhaust the runner's default memory and swap
during Docker layer export. Other E2E jobs keep the standard runner memory
configuration.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/README.md` around lines 413 - 420, Update the rebuild fixture
sentence to explicitly state that it verifies a 32 GiB swap-file floor on GitHub
Actions, rather than saying it “verifies that floor.” Keep the existing trusted
control-plane and main-workflow conditions unchanged.

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions
Status: Canonical ledger: 0 blocker(s), 1 warning(s), 0 suggestion(s).

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Test the swap size and build-order invariants

  • Location: test/e2e/support/sandbox-images-workflow-boundary.test.ts:164
  • Category: tests
  • Problem: The new test removes `swapon` from both jobs, but it does not mutate the required `32G` allocation or move the swap step after either Hermes build.
  • Impact: A future workflow edit can leave swap enabled but reduce its capacity or activate it after the memory-heavy build. The boundary test suite would not detect either violation.
  • Recommendation: Add fixture mutations for an allocation size other than `32G` and for a swap step after each Hermes build. Assert the validator reports the size and ordering errors.
  • Verification: Inspect `validateHermesExportSwap` and run the focused workflow-boundary test after each fixture mutation.
  • Test coverage: A workflow-boundary test that rejects an undersized swap allocation and a swap step placed after the Hermes build in both guarded jobs.
  • Evidence: `validateHermesExportSwap` requires `sudo fallocate -l 32G "$swap_file"` and requires the swap step before each Hermes build. The added test mutates only `sudo swapon "$swap_file"` and asserts only the two missing-activation errors.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@ericksoa

Copy link
Copy Markdown
Contributor Author

Maintainer exact-head disposition for 36b15e158513983497b7f3beed351c2dc2f2594d:

  • Current main reproduced the defect in CI run 29919417008 attempt 2, job 88927259359: all 63 Hermes Dockerfile steps passed, then the no-swap runner shut down with exit 143 during BuildKit exporting layers.
  • The exact-head real-workflow proof run 29917549629 passed both patched exporters: the canonical Hermes export completed in 178.8s and the messaging-plan Hermes export completed in 538.1s, followed by their image/boundary checks.
  • Selected E2E run 29918942438 passed credential sanitization, cloud onboard, and OpenClaw security posture. Hermes security posture lost two distinct hosted runners after Vitest started, with no test assertion or product error on either attempt.

I accept the failed gate as a hosted-runner-loss exception for this workflow-only resource fix. I will use the maintainer exact-head admin merge after the already-running current-main platform proof reaches a terminal state, so that evidence is preserved rather than canceled.

@ericksoa
ericksoa merged commit f992494 into main Jul 22, 2026
107 of 112 checks passed
@ericksoa
ericksoa deleted the codex/stabilize-hermes-image-export branch July 22, 2026 13:20
@ericksoa

Copy link
Copy Markdown
Contributor Author

Post-merge maintainer proof on main SHA f9924949922f8e554f94aeefdd23a993a801e7b4:

  • CI / Main 29923425375 is terminal success.
  • Canonical sandbox image build passed.
  • Dedicated Hermes production image job 88937025996 passed build, node-tar scan, readability, secret-boundary, root-entrypoint, image export, and isolation upload. The runner enabled the 32 GiB export swap and completed the formerly failing layer export in 114.4 seconds.
  • The OpenClaw and Hermes messaging-plan image boundaries passed, including the Hermes export path.
  • The downstream installed state-directory guard metadata contract passed against the produced OpenClaw and Hermes images.
  • Code Quality and Security / Code Scanning are terminal success on the same main SHA.

This closes the hosted-runner-loss exception with exact post-merge main evidence; no product or sandbox-contract waiver remains for the image-build path.

@cv cv mentioned this pull request Jul 22, 2026
22 tasks
cv added a commit that referenced this pull request Jul 22, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical `## v0.0.92` release entry to
`docs/changelog/2026-07-22.mdx` before the release plan is generated.
The entry summarizes all ten pull requests merged after v0.0.91,
including the OpenClaw security update and Jaeger runtime regression
coverage.

## Changes

- Added the canonical v0.0.92 changelog entry.
- Recorded the user-visible, security, documentation, CI, and test
changes in the release range.
- #7280 -> `docs/changelog/2026-07-22.mdx`: OpenClaw 2026.7.1 and
Node.js 22.23.1 security/runtime update.
- #7378 -> `docs/changelog/2026-07-22.mdx`: canonical macOS watcher path
validation.
- #7379 -> `docs/changelog/2026-07-22.mdx`: stabilized full WSL platform
validation.
- #7380 -> `docs/changelog/2026-07-22.mdx`: bounded swap for hosted
Hermes image exports.
- #7100 -> `docs/changelog/2026-07-22.mdx`: semantic progress phases for
live E2E tests.
- #7376 -> `docs/changelog/2026-07-22.mdx`: restored v0.0.91 changelog
history and corrected tagged guidance.
- #7374 -> `docs/changelog/2026-07-22.mdx`: reviewed Homebrew formula
transition for installer integrity checks.
- #7346 -> `docs/changelog/2026-07-22.mdx`: provider-neutral headless
server deployment guidance.
- #7381 -> `docs/changelog/2026-07-22.mdx`: stabilized Hermes guard
timing and WSL ownership fixtures.
- #7339 -> `docs/changelog/2026-07-22.mdx`: real-artifact Jaeger header
remediation regression coverage.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates the canonical dated changelog
and release heading contract.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts` passed 6/6 tests.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to a
changelog-only change.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — passed
with 0 errors and 2 pre-existing Fern warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added `v0.0.92` release notes covering sandboxing updates
(OpenClaw/Node.js bumps, integrity pinning remediation, mcporter
handling, and upgrade validation).
* Updated deployment guidance for provider-neutral headless installs,
and improved live E2E test reporting plus phase-plan validation.
* Tightened installer integrity-check messaging during an OpenShell
Homebrew transition and expanded platform/image validation (including
macOS/WSL timing) and hosted image export behavior.
* Restored the previously missed `v0.0.91` changelog entry and release
validation guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants