Skip to content

fix(e2e): seed historical Hermes API key#7364

Merged
cv merged 2 commits into
mainfrom
codex/fix-rebuild-hermes-api-key-fixture
Jul 22, 2026
Merged

fix(e2e): seed historical Hermes API key#7364
cv merged 2 commits into
mainfrom
codex/fix-rebuild-hermes-api-key-fixture

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Repair the rebuild-hermes and rebuild-hermes-stale-base live fixtures after the historical sandbox began failing the API bearer-token lifecycle check before rebuild. The synthetic old sandbox now contains a valid per-run API server key, so the test can verify that rebuild rotates it and then keeps the new token stable.

Related Issue

Follow-up to #7144.

Changes

  • Seed API_SERVER_KEY in the synthetic historical Hermes .env file.
  • Generate a valid 64-character per-run fixture key and include it in command redaction and backup leak scanning.
  • Assert the historical Dockerfile contains the key required by the token-lifecycle probe.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: This only repairs a synthetic live-E2E fixture and does not change product behavior or user workflows.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Maintainer review is pending.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: not applicable
  • Station profile/scenario: not applicable
  • Result: not applicable
  • Supporting evidence: not applicable

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — focused E2E-support fixture test: 9/9; CLI build and typecheck passed.
  • Applicable broad gate passed — npm run check:diff passed.
  • Quality Gates section completed with required justifications or waivers — sensitive-path review is pending.
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved protection against sensitive API server keys appearing in rebuild artifacts, command output, and diagnostic scans.
    • Ensured rebuilt sandbox environments receive the required API server key configuration.
  • Tests
    • Added coverage verifying API server key injection and credential redaction during rebuild workflows.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria self-assigned this Jul 22, 2026
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 25781078-5290-436e-a294-58772e113f92

📥 Commits

Reviewing files that changed from the base of the PR and between 0dc241c and 925f158.

📒 Files selected for processing (3)
  • test/e2e/live/rebuild-hermes-old-sandbox.ts
  • test/e2e/live/rebuild-hermes.test.ts
  • test/e2e/support/rebuild-hermes-old-base-fixture.test.ts

📝 Walkthrough

Walkthrough

The sandbox Dockerfile builder now accepts an API server key and writes it to the generated environment. The live rebuild test derives a key digest, propagates it through sandbox setup and cleanup commands, and includes it in credential leak checks.

Changes

Hermes API server key handling

Layer / File(s) Summary
Dockerfile API key contract
test/e2e/live/rebuild-hermes-old-sandbox.ts, test/e2e/support/rebuild-hermes-old-base-fixture.test.ts
The Dockerfile builder requires apiServerKey, emits API_SERVER_KEY in the generated environment, and verifies the generated assignment.
Live rebuild key propagation
test/e2e/live/rebuild-hermes.test.ts
The live rebuild test derives PRE_REBUILD_API_SERVER_KEY, passes it to Dockerfile generation, redacts it during cleanup and readiness operations, and scans for it in credential leak paths.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: seeding a historical Hermes API key for E2E rebuild fixtures.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-rebuild-hermes-api-key-fixture

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 925f158 in the codex/fix-rebuild-he... branch remains at 96%, unchanged from commit 0dc241c in the main branch.

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Blocking findings reported

Advisor assessment: Blockers require maintainer review
Next action: Review the blockers below.
Findings: 1 blocker · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 1 blocker · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported 1 fewer blocker, the same number of warnings, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, rebuild-hermes, rebuild-hermes-stale-base

2 optional E2E recommendations
  • hermes-e2e
  • hermes-gpu-startup

Blockers

PRA-1 Blocker — Do not bake the Hermes API key into the fixture image

  • Location: test/e2e/live/rebuild-hermes-old-sandbox.ts:51
  • Category: correctness
  • Problem: The historical sandbox Dockerfile writes apiServerKey to /sandbox/.hermes/.env during image build. The Hermes sandbox contract requires API_SERVER_KEY to be minted at sandbox startup, not stored in an image layer.
  • Impact: The predictable pre-rebuild bearer key persists in the synthetic image and can be recovered by an image-layer reader before rebuild. This bypasses the runtime key-minting boundary that the Hermes image contract protects.
  • Fix: Keep API_SERVER_KEY out of the generated Dockerfile. Seed the required historical state through the same runtime startup path that mints the key, or change the fixture setup so it does not require a pre-baked key.
  • Verification: Inspect the generated Dockerfile and the pre-rebuild image filesystem; neither must contain an API_SERVER_KEY assignment. Compare with test/e2e/live/hermes-sandbox-secret-boundary.test.ts.
  • Test coverage: Add a fixture test that rejects a generated Dockerfile containing API_SERVER_KEY, and a live assertion that the pre-rebuild sandbox receives a runtime-minted key rather than an image-baked value.
  • Evidence: test/e2e/live/rebuild-hermes-old-sandbox.ts:51 renders API_SERVER_KEY=${options.apiServerKey} in the Dockerfile. test/e2e/live/hermes-sandbox-secret-boundary.test.ts:132 fails when the image .env contains API_SERVER_KEY. test/e2e/live/hermes-sandbox-secret-boundary.test.ts:761-765 records the contract that the image contains no baked API_SERVER_KEY and startup mints a unique key.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@cv
cv merged commit fdad045 into main Jul 22, 2026
74 of 76 checks passed
@cv
cv deleted the codex/fix-rebuild-hermes-api-key-fixture branch July 22, 2026 05:24
@github-actions github-actions Bot added the v0.0.91 Release target label Jul 22, 2026
ericksoa added a commit that referenced this pull request Jul 22, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Updates the reviewed OpenClaw distribution from 2026.6.10 to 2026.7.1
and the sandbox Node image to 22.23.1. The image build fails closed
while applying integrity-pinned remediations for vulnerable Axios copies
in the Slack and Teams archives and the affected OpenTelemetry Jaeger
pair in diagnostics. The separately locked `mcporter@0.7.3` graph now
resolves exact `@hono/node-server@2.0.11` and `fast-uri@3.1.4` releases
and passes the reviewed audit, signature, install, and CLI checks.

The sandbox credential contract remains unchanged: generated
`openclaw.json` contains the non-secret `apiKey: "unused"` sentinel,
while `COMPATIBLE_API_KEY` remains an OpenShell provider-environment
placeholder whose real value is resolved at the OpenShell boundary. No
literal provider credential is persisted in OpenClaw configuration or
state.

OpenClaw 2026.7.1 changed loopback shared-token CLI calls to omit the
signed CLI device identity. That prevented the paired-device scope
comparison from reaching the canonical `scope-upgrade` pairing gate. The
compatibility patch now retains the signed CLI identity only after a
stored operator device credential exists. The shared token still
authenticates the loopback call; the paired-device record remains
authoritative for scopes, and an `operator.admin` request must still
create and receive canonical pairing approval.

The installed-base contract is covered explicitly. A pinned E2E row
installs NemoClaw v0.0.89 with OpenClaw 2026.6.10 from a
registry-SRI-bound frozen source archive, materializes the legacy
per-agent SQLite, Memory Core, and update-check state, then upgrades in
place to 2026.7.1. The current installer is deliberately not given the
provider credential, so the post-upgrade turn proves reuse of the
credential already held by OpenShell. The shared frozen-installer
adapter also keeps the historical v0.0.36, v0.0.55, v0.0.74, and v0.0.89
matrix deterministic without weakening the current candidate.

The merge from current `main` preserves its Node `tar@7.5.20` package
fix and bundled-npm tar remediation, rebinding the affected-base guard
to the upgraded Node 22.23.1 image digest. For the E2E-only OpenClaw
2026.3.11 stale-upgrade fixture, the exact reviewed source shape is
verified, the replacement `tar@7.5.19` archive is registry-SRI-bound,
and that reviewed package is bundled into the remediated OpenClaw
archive before installation. There is no completed-image scanner
exception: any remaining affected copy fails the strict scan, and both
stale-upgrade E2Es require the rebuilt current image inventory to be
completely clean.

## Changes

- Pin OpenClaw and its first-party plugin archives to reviewed 2026.7.1
artifacts and update the Node 22.23.1 image digest.
- Remediate only the reviewed Axios and diagnostics OTEL subtrees, with
original-package identity checks, integrity pins, canonical tree
digests, and archive-shape drift rejection.
- Lock the mcporter runtime to `@hono/node-server@2.0.11` and
`fast-uri@3.1.4`; verify the script-disabled production install,
registry signatures, ESM transport construction/close path, and
installed CLI version.
- Retain stored CLI device identity on loopback shared-token calls so
OpenClaw's paired-device scope gate remains authoritative without moving
provider secrets into OpenClaw state.
- Preserve private device, credential, and SQLite state across startup
and installed-base migrations, including bounded repair of legacy
update-check state.
- Add the frozen v0.0.89/OpenClaw 2026.6.10 to 2026.7.1 installed-base
E2E row and retain the broader historical migration matrix.
- Preserve current-main node-tar remediation, run it only after curl is
installed in the DeepAgents and Hermes bases, and verify both patch
helpers in the sandbox build context.
- Exact-shape remediate the E2E-only 2026.3.11 source archive from
`tar@7.5.11` to a registry-SRI-bound, bundled `tar@7.5.19`, with the
completed-image scan remaining strict and exception-free.
- Set the authoritative local base-build allowance to 90 seconds for
upgrade-head matrix runs that must build the exact local base.
- Incorporate current main's historical Hermes fixture repair so rebuild
coverage seeds, rotates, redacts, and leak-scans the required per-run
API server key.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — exact head reviewed during maintainer follow-through; the
implementation fails closed on package identity, archive shape,
dependency graph, SRI drift, and device-scope enforcement.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: not applicable
- Station profile/scenario: not applicable
- Result: not applicable
- Supporting evidence: not applicable

## Verification

- [x] PR description includes `Signed-off-by:` lines and all new commits
are SSH-signed and DCO-clean.
- [x] `npm run checks`, `npm run typecheck`, `npm run typecheck:cli`,
`npm run source-shape:check`, and `npm run test-size:check` passed on
the current-main reconciliation.
- [x] Focused combined suites passed: 258 tests with one expected skip;
an independent combined run passed 61/61; the fresh real OpenClaw
2026.7.1 tarball harness passed 5/5.
- [x] The real upstream 2026.7.1 compiled distribution audit passed all
five required compatibility-shape checks.
- [x] After merging current `main` through #7360, the focused
remediation/integrity/scanner suites passed 34/34 and the installed-base
migration/workflow suites passed 19/19.
- [x] After reconciling current `main` through #7364, the exact affected
Hermes historical fixture passed 9/9, the focused OpenClaw contract set
passed 81 tests with one expected skip, and repository/source-shape
checks passed.
- [ ] Exact-head required GitHub CI and E2E gate are in flight.
- [x] No secrets, API keys, or credentials committed.

Evidence:

- Prior upgrade-head historical evidence: [five migration rows
passed](https://github.com/NVIDIA/NemoClaw/actions/runs/29862234216),
including the dedicated v0.0.89/OpenClaw 2026.6.10 installed-base
upgrade. Current-head coverage is also selected in the full matrix
below.
- Scope-upgrade proof on parent head 54b59af: [focused run
29888373497](https://github.com/NVIDIA/NemoClaw/actions/runs/29888373497)
and the independent full-matrix cell both passed the formerly
deterministic `operator.admin` cron path.
- Prior-head focused rebuild coverage: [run
29889345896](https://github.com/NVIDIA/NemoClaw/actions/runs/29889345896),
covering both stale OpenClaw upgrade fixtures, DeepAgents routing, and
both Hermes rebuild paths.
- Preserved prior-head full E2E evidence: [run
29889347151](https://github.com/NVIDIA/NemoClaw/actions/runs/29889347151)
improved the original 66-failure/22-pass signal to 19 failures/68
passes. Every remaining red was a pre-existing DeepAgents trust-fixture
failure, hosted inference capacity error, runner loss/cancellation, or
transient load/SSH failure; every OpenClaw upgrade, rebuild, pairing,
and installed-base migration path passed. The intermediate-head [replay
29888849758](https://github.com/NVIDIA/NemoClaw/actions/runs/29888849758)
remains active append-only evidence.
- Current exact-head append-only full matrix: [run
29893928454](https://github.com/NVIDIA/NemoClaw/actions/runs/29893928454),
queued without canceling or replacing the active replay.
- Current exact-head required CI: [run
29893824224](https://github.com/NVIDIA/NemoClaw/actions/runs/29893824224),
[security CodeQL
29893824206](https://github.com/NVIDIA/NemoClaw/actions/runs/29893824206),
and [E2E gate
29893823011](https://github.com/NVIDIA/NemoClaw/actions/runs/29893823011).

---
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>

---------

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Co-authored-by: Aaron Erickson <aerickson@nvidia.com>
cv pushed a commit that referenced this pull request Jul 22, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the canonical `v0.0.91` changelog entry that was missed before the
release tag was cut. Correct the custom-image compatibility guidance
because the tagged code still accepts the legacy inference route
selector instead of removing it in v0.0.91.

## Changes

- Add `docs/changelog/2026-07-22.mdx` with the release summary and
detailed security, rebuild, Hermes, DGX Station, and
historical-validation changes from the published v0.0.91 announcement.
- Link shipped behavior to the most specific published OpenClaw and
Hermes documentation routes.
- Correct the v0.0.90 changelog and command reference so they match the
compatibility fallback present in the tagged v0.0.91 code without
inventing a new removal version.
- Keep the immutable v0.0.91 release tag unchanged; this is the
documented post-release recovery path.

Source summary:

- [#7332](#7332),
[#7289](#7289), and
[#7294](#7294) ->
`docs/changelog/2026-07-22.mdx`: Summarize completed-image `node-tar`
remediation and current and historical container verification.
- [#7213](#7213),
[#7363](#7363),
[#7366](#7366), and
[#7369](#7369) ->
`docs/changelog/2026-07-22.mdx`: Summarize trusted base preparation,
backup reuse, deletion convergence, and rebuild confidence.
- [#7212](#7212) ->
`docs/changelog/2026-07-22.mdx`: Summarize the Hermes API bearer-token
lifecycle and supported retrieval command.
- [#7327](#7327) and
[#7328](#7328) ->
`docs/changelog/2026-07-22.mdx`: Summarize qualified DGX Station
guidance and reproducible coding-agent installation instructions.
- [#7355](#7355),
[#7360](#7360),
[#7362](#7362), and
[#7364](#7364) ->
`docs/changelog/2026-07-22.mdx`: Summarize restored historical OpenClaw
upgrade and Hermes rebuild validation.
- [#7189](#7189) ->
`docs/changelog/2026-07-20.mdx`, `docs/reference/commands.mdx`: Correct
its forward-looking removal deadline after v0.0.91 shipped with the
documented legacy fallback still present.
- [#7282](#7282),
[#7306](#7306), and
[#7341](#7341) need no additional
user-guide update because they already update their owned contributor or
user-facing text directly.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates dated changelog structure, SPDX
syntax, version ordering, and published routes.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable; this documentation-only PR does not
change Station preparation or runtime behavior.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts` (6 passed).
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — not run; this is a focused
documentation-only change.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 existing site-wide warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the native changelog entry uses the required parser-safe MDX SPDX
comment and intentionally has no frontmatter.

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated v0.0.91 guidance for custom Dockerfiles, including continued
legacy compatibility and recommended migration to
`NEMOCLAW_INFERENCE_PROVIDER_ID`.
* Added release notes covering image security scanning, safer rebuild
behavior, token management, DGX Station guidance, and deterministic
release validation.
* Clarified that existing custom images may continue using the legacy
selector temporarily, with fallback removal planned for a future
release.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Co-authored-by: cjagwani <cjagwani@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.91 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants