fix(hermes): repair v0.19 runtime migrations by jyaunches · Pull Request #7849 · NVIDIA/NemoClaw

jyaunches · 2026-07-29T20:13:47Z

Summary

Hermes 0.19 introduced deterministic runtime regressions after v0.0.97: dashboard seed-marker drift, an unsupported blanket reasoning-effort request, and a mutable cron execution ledger inside the sealed cron job-definition directory. This change aligns the reviewed configuration contracts and relocates only mutable cron execution history into the writable runtime boundary while keeping cron job definitions sealed.

The trusted exact-head E2E also exposed a stale Hermes MCP fixture: Hermes 0.19 names native MCP tools as mcp__server__tool, but the fixture still queried the Hermes 0.18 single-underscore name and accepted an echoed search query as a match. The fixture now uses the v0.19 name and requires an exact structural match before progressing.

Changes

Require the current Hermes dashboard seed marker and omit the blanket reasoning-effort default from generated requests.
Patch the hash-pinned Hermes 0.19 execution ledger and quick-snapshot sources to use runtime/cron-executions.db.
Repair the writable gateway and runtime boundaries during restart while preserving cron as root:sandbox with mode 0755.
Emit fixed, allowlisted startup diagnostics without exposing untrusted output.
Align the Hermes MCP fixture with the v0.19 mcp__server__tool name and fail closed unless discovery and schema responses structurally identify the exact deferred tool.
Add focused unit, integration, E2E-support, and live-E2E regression coverage for the post-tag failure paths.
Document the sealed cron definition boundary, runtime ledger, restore behavior, and exact-head Hermes dependency evidence.

Type of Change

Code change (feature, bug fix, or refactor)
Code change with doc updates
Doc only (prose changes, no code sample modifications)
Doc only (includes code sample changes)

Quality Gates

Tests added or updated for changed behavior
Existing tests cover changed behavior — justification:
Tests not applicable — justification:
Docs updated for user-facing behavior changes
Docs not applicable — justification:
Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop completed the repository nine-category review for exact head 8e0bbe24a against base and merge base d52d4599a; all categories passed with no findings. The final delta is test-only and strengthens fail-closed MCP fixture validation.
Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

Documentation writer subagent reviewed the completed changes
Result: docs-updated
Evidence: docs/manage-sandboxes/backup-restore.mdx documents the relocated cron ledger and named-profile snapshot limitation. docs/security/hermes-0.19.0-dependency-review.md documents the restart regression, protected cron directory, writable runtime boundary, and Hermes v0.19 mcp__server__tool naming. The final test-only delta aligns the fixture with that documented naming and requires an exact structural discovery match; no additional user-facing documentation is required.
Agent: Codex Desktop

DGX Station Hardware Evidence

Tested on DGX Station
Tested commit: Not applicable
Station profile/scenario: Not applicable
Result: Not applicable
Supporting evidence: Not applicable

Trusted E2E Failure and Fix

The approved trusted run for prior exact head cdd62f906 passed Bedrock-compatible Anthropic with Hermes and both Hermes inference-switch scenarios, but failed MCP bridge (Hermes). This is a deterministic fixture regression, not a flake or production MCP failure. PR #7771 upgraded Hermes to 0.19; upstream commit e01f58ff1 changed native MCP names to mcp__server__tool. The fixture retained mcp_fake_fake_echo, and its substring assertion mistook the echoed query for a discovery match before calling tool_describe with the nonexistent legacy name.

Exact head 8e0bbe24a corrects the tool name, requires matches[].name to equal the deferred name, validates the exact described schema, and adds an echoed-query-with-empty-matches regression test. A new exact-head trusted E2E verdict is required before merge.

Verification

PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Focused Hermes suites passed 111 tests with 3 skips; the post-writing affected set passed 86 tests with 1 skip; startup passed 39 tests; MCP/E2E-support passed 57 tests; PR-risk/gate coverage passed 97 tests. On exact head 8e0bbe24a, npm exec -- vitest run test/mcp-bridge-servers.test.ts passed all 10 tests after normal hooks; npm run typecheck, targeted Biome checks, git diff --check, and npm run checks:repository passed.
Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Exact-head GitHub CI is in progress.
Quality Gates section completed with required justifications or waivers
No secrets, API keys, or credentials committed
npm run docs builds without warnings (doc changes only) — result: exact-head build completed with 0 errors and 2 existing Fern warnings.
Doc pages follow the style guide (doc changes only)
New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Julie Yaunches jyaunches@nvidia.com

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

copy-pr-bot · 2026-07-29T20:13:50Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-07-29T20:13:58Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Hermes configuration removes reasoning_effort, dashboard seeding requires a reviewed-policy marker, and cron execution history moves from cron to runtime. Docker patching, state manifests, startup permissions, snapshot behavior, diagnostics, and lifecycle tests are updated accordingly.

Changes

Hermes runtime alignment

Layer / File(s)	Summary
Configuration and dashboard contracts `agents/hermes/config/hermes-config.ts`, `src/lib/sandbox/config.ts`, `src/lib/sandbox/hermes-dashboard-reseed.test.ts`, `test/e2e/live/*`, `test/generate-hermes-config.test.ts`	Hermes no longer emits `reasoning_effort`; dashboard reseeding now requires the reviewed-policy marker, with updated unit and E2E assertions.
Runtime ledger relocation and image patching `agents/hermes/patch-cron-execution-runtime.py`, `agents/hermes/Dockerfile`, `agents/hermes/manifest.yaml`, `test/hermes-cron-execution-runtime-patch.test.ts`, `test/hermes-final-image-layout.test.ts`	The cron execution database and quick snapshot inventory move to `runtime/cron-executions.db`, using guarded, hash-verified build-time patching and updated image contracts.
Startup state layout and permissions `agents/hermes/start.sh`, `scripts/managed-gateway-control.py`, `test/hermes-start.test.ts`, `test/hermes-discord-recovery-permissions.test.ts`, `test/managed-gateway-control.test.ts`	Startup repair targets `gateway` and `runtime` with explicit failure diagnostics, while locked cron definitions remain separately sealed and diagnostic expectations use runtime wording.
Snapshot and lifecycle validation `docs/manage-sandboxes/backup-restore.mdx`, `docs/security/hermes-0.19.0-dependency-review.md`, `test/hermes-state-ledger-snapshot.test.ts`, `test/state-dir-guard.test.ts`, `test/e2e/live/mcp-bridge-hermes-lifecycle.ts`, `test/e2e/support/mcp-bridge-hermes-lifecycle.test.ts`, `test/e2e/live/mcp-bridge-servers.ts`, `test/mcp-bridge-servers.test.ts`	Documentation and tests cover runtime-ledger online backup/restore, sealed cron state, locked gateway restarts, structured MCP tool responses, and rollback inspection requiring `matched` state.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DockerBuild
  participant HermesPatcher
  participant HermesImage
  participant SnapshotRestore
  DockerBuild->>HermesPatcher: patch cron ledger and snapshot inventory
  HermesPatcher->>HermesImage: write runtime/cron-executions.db paths
  HermesImage->>SnapshotRestore: expose declared runtime ledger
  SnapshotRestore-->>HermesImage: backup and restore SQLite ledger online

Suggested labels: area: security, platform: container

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 3.13% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately summarizes the main Hermes v0.19 runtime migration fixes in this changeset.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch codex/fix-hermes-upgrade-e2e-regressions

_{Comment @coderabbitai help to get the list of available commands.}

github-code-quality · 2026-07-29T20:14:38Z

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 8e0bbe2 in the codex/fix-hermes-upg... branch remains at 96%, unchanged from commit d52d459 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 8e0bbe2 in the codex/fix-hermes-upg... branch remains at 81%, unchanged from commit d52d459 in the main branch.

Show a code coverage summary of the most impacted files.

File	`main` d52d459	`codex/fix-hermes-upg...` 8e0bbe2	+/-
`src/lib/policy/...ne-exclusion.ts`	97%	91%	-6%
`src/lib/sandbox...rce-identity.ts`	88%	88%	0%
`src/lib/tunnel/services.ts`	73%	73%	0%

_{Updated July 30, 2026 01:22 UTC}

github-actions · 2026-07-29T20:18:44Z

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions

Model lanes

GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
Nemotron 3 Ultra (second opinion): Failed

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, full-e2e, hermes-e2e, security-posture, bedrock-runtime-compatible-anthropic, hermes-inference-switch, inference-routing, mcp-bridge, mcp-bridge-dev, network-policy

1 warning · 0 suggestions

Warnings

Warnings do not block.

`PRA-1` Warning — Cover cron ledger writes after a locked gateway restart

Location: test/e2e/live/mcp-bridge-hermes-lifecycle.ts:43
Category: tests
Problem: The locked-restart helper checks directory modes and MCP integrity but does not exercise Hermes cron execution history after the restart.
Impact: A final image can preserve the expected directory modes while the patched gateway cannot create or reopen runtime/cron-executions.db under Shields up.
Recommendation: Extend the locked-restart live scenario to create and reopen a cron execution through Hermes after restart, while asserting cron remains root:sandbox mode 0755.
Verification: Inspect the locked-restart helper and confirm the new scenario invokes Hermes cron execution after restart and checks the relocated ledger path.
Test coverage: In the locked Hermes gateway restart E2E, create a cron execution and verify a subsequent Hermes operation reopens or writes runtime/cron-executions.db after Shields up, with cron still root:sandbox 0755.
Simplification (shrink): Remove Mode-only locked-restart confidence for the relocated cron ledger.; use One direct live create-and-reopen assertion in the existing locked-restart helper.. Net: 0 lines.
Keep: Keep the existing permission and integrity assertions; add the runtime operation rather than replacing them.
Evidence: test/e2e/live/mcp-bridge-hermes-lifecycle.ts adds locked-restart checks for gateway, runtime, and cron modes but no cron execution API operation. agents/hermes/Dockerfile relocates the gateway-owned cron execution ledger to runtime/cron-executions.db.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

ericksoa

Approve pending e2e green

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> # Conflicts: # agents/hermes/start.sh # scripts/managed-gateway-control.py # test/e2e/live/mcp-bridge-hermes-lifecycle.ts # test/managed-gateway-control.test.ts

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

github-actions · 2026-07-29T23:03:40Z

🌿 Preview your docs: https://nvidia-preview-pr-7849.docs.buildwithfern.com/nemoclaw

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/deployment/sandbox-hardening.mdx`:
- Line 11: Update the agent-variants metadata in sandbox-hardening.mdx to
include the applicable supported variants hermes and deepagents alongside
openclaw, ensuring generic NemoClaw sandbox hardening guidance is discoverable
for all covered products.

In `@docs/monitoring/monitor-sandbox-activity.mdx`:
- Line 13: Update the shared page declaration and session-state section in
monitor-sandbox-activity.mdx so Hermes does not receive OpenClaw-only
instructions. Wrap the session-state content around lines 60-75 with the
existing AgentOnly component for the openclaw variant, or replace it with valid
Hermes-specific instructions before retaining both variants in agent-variants.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f232a9a8-92b8-45b7-a09f-8c97638b19b4

📥 Commits

Reviewing files that changed from the base of the PR and between 117ddf2 and cea7192.

📒 Files selected for processing (81)

.agents/skills/nemoclaw-contributor-update-docs/SKILL.md
.agents/skills/nemoclaw-contributor-update-hermes/references/hermes-contract-map.md
agents/hermes/Dockerfile
agents/hermes/manifest.yaml
agents/hermes/patch-cron-execution-runtime.py
agents/hermes/start.sh
docs/AGENTS.md
docs/CONTRIBUTING.md
docs/about/ecosystem-deepagents.mdx
docs/about/ecosystem-hermes.mdx
docs/about/ecosystem.mdx
docs/configure-agents/configure-agent-heartbeats.mdx
docs/configure-agents/understand-context-compaction.mdx
docs/deployment/gateway-lifecycle-authority.mdx
docs/deployment/install-openclaw-plugins.mdx
docs/deployment/sandbox-hardening.mdx
docs/get-started/quickstart-hermes.mdx
docs/get-started/quickstart-langchain-deepagents-code.mdx
docs/get-started/quickstart.mdx
docs/index.yml
docs/inference/configure-model-capabilities.mdx
docs/inference/declarative-agents-manifest.mdx
docs/inference/set-up-ollama.mdx
docs/inference/set-up-sub-agent.mdx
docs/inference/use-hermes-provider.mdx
docs/manage-sandboxes/add-channels-after-onboarding.mdx
docs/manage-sandboxes/backup-restore.mdx
docs/manage-sandboxes/enable-channels-during-onboarding.mdx
docs/manage-sandboxes/gateway-lifecycle-control.mdx
docs/manage-sandboxes/install-plugins-hermes.mdx
docs/manage-sandboxes/manage-messaging-channels.mdx
docs/manage-sandboxes/messaging-channels.mdx
docs/manage-sandboxes/runtime-controls.mdx
docs/manage-sandboxes/set-up-discord.mdx
docs/manage-sandboxes/set-up-microsoft-teams.mdx
docs/manage-sandboxes/set-up-slack.mdx
docs/manage-sandboxes/set-up-telegram.mdx
docs/manage-sandboxes/set-up-wechat.mdx
docs/manage-sandboxes/set-up-whatsapp.mdx
docs/monitoring/manage-deepagents-trace-export.mdx
docs/monitoring/monitor-sandbox-activity.mdx
docs/monitoring/set-up-deepagents-trace-export.mdx
docs/monitoring/understand-deepagents-trace-export.mdx
docs/monitoring/verify-deepagents-trace-export.mdx
docs/network-policy/configure-raw-tls-passthrough.mdx
docs/network-policy/explain-network-policy-to-agents.mdx
docs/network-policy/integration-policy-examples.mdx
docs/network-policy/set-up-gmail-with-an-app-password.mdx
docs/reference/commands.mdx
docs/reference/enterprise-readiness.mdx
docs/resources/engineer-agentic-documentation.mdx
docs/security/configure-corporate-ca-trust.mdx
docs/security/credential-rotation.mdx
docs/security/hermes-0.19.0-dependency-review.md
docs/security/openclaw-controls.mdx
docs/security/openshell-0.0.71-gateway-auth-review.mdx
docs/security/tcb-boundary.mdx
scripts/managed-gateway-control.py
scripts/sync-agent-variant-docs.mts
src/lib/actions/sandbox/process-recovery.test.ts
src/lib/actions/sandbox/process-recovery.ts
src/lib/actions/sandbox/status-flow.test.ts
src/lib/actions/sandbox/status-lookup-rendering.ts
src/lib/actions/sandbox/status-snapshot-recovery.test.ts
src/lib/actions/sandbox/status-snapshot.ts
test/check-docs-published-routes.test.ts
test/cli/sandbox-status-json.test.ts
test/e2e/live/dashboard-remote-bind-env.ts
test/e2e/live/dashboard-remote-bind.test.ts
test/e2e/live/mcp-bridge-hermes-lifecycle.ts
test/e2e/support/dashboard-remote-bind-env.test.ts
test/hermes-cron-execution-runtime-patch.test.ts
test/hermes-dependency-review.test.ts
test/hermes-discord-recovery-permissions.test.ts
test/hermes-final-image-layout.test.ts
test/hermes-start.test.ts
test/hermes-state-ledger-snapshot.test.ts
test/managed-gateway-control.test.ts
test/snapshot-stale-directory-restore.test.ts
test/state-dir-guard.test.ts
test/sync-agent-variant-docs.test.ts

coderabbitai · 2026-07-29T23:11:43Z

 keywords: ["nemoclaw sandbox hardening", "container security", "docker capabilities", "process limits"]
 content:
  type: "reference"
+agent-variants: ["openclaw"]


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Publish sandbox hardening guidance for every applicable variant.

This page describes generic NemoClaw sandbox security controls rather than OpenClaw-specific behavior. Unless there is an explicit scope decision, include hermes and deepagents in agent-variants so those users can access the hardening guidance.

As per path instructions, documentation variant scope must reflect the actual supported product surface.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/deployment/sandbox-hardening.mdx` at line 11, Update the agent-variants metadata in sandbox-hardening.mdx to include the applicable supported variants hermes and deepagents alongside openclaw, ensuring generic NemoClaw sandbox hardening guidance is discoverable for all covered products.

Source: Path instructions

coderabbitai · 2026-07-29T23:11:43Z

  type: "how_to"
 skill:
  priority: 10
+agent-variants: ["openclaw", "hermes"]


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Do not expose OpenClaw-only session instructions on the Hermes page.

This declaration publishes the page for Hermes, but the unguarded session-state section at Lines 60-75 documents OpenClaw-only paths and behavior. Wrap that section in <AgentOnly variant="openclaw"> or provide Hermes-specific instructions before declaring both variants.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/monitoring/monitor-sandbox-activity.mdx` at line 13, Update the shared page declaration and session-state section in monitor-sandbox-activity.mdx so Hermes does not receive OpenClaw-only instructions. Wrap the session-state content around lines 60-75 with the existing AgentOnly component for the openclaw variant, or replace it with valid Hermes-specific instructions before retaining both variants in agent-variants.

Source: Path instructions

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/mcp-bridge-hermes-lifecycle.test.ts`:
- Around line 38-45: Update the test case around assertHermesReloadRollback to
assert the specific rejection message or error pattern expected when
sandboxWithInspectionState("current") is rejected, rather than accepting any
thrown error. Preserve the existing hermes-e2e and MCP endpoint setup while
matching the observable contract for the current state.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3169e926-16a3-4165-a766-1645d91fc8bd

📥 Commits

Reviewing files that changed from the base of the PR and between cea7192 and ae3cbc8.

📒 Files selected for processing (2)

test/e2e/live/mcp-bridge-hermes-lifecycle.ts
test/e2e/support/mcp-bridge-hermes-lifecycle.test.ts

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

cjagwani · 2026-07-30T00:50:41Z

Exact-head protected E2E blocker at cdd62f90698ffdd83319d73c4d78e7743784b0bb: mcp-bridge (hermes) failed in phase 5, “exercise lifecycle and confirm Hermes bridge removal.” The configured bridge and rollback phases passed, but the post-lifecycle request did not contain the expected MCP_AUTH_REWRITE_OK::… result token (test/e2e/live/mcp-bridge.test.ts:1261). This directly exercises the Hermes MCP rollback/lifecycle files changed by this head, so I cannot classify it as unrelated infrastructure. The other completed selected jobs are green; mcp-bridge (deepagents) is still running.

I am not submitting a Changes Requested review and I am not modifying the contributor branch. Please diagnose this exact failure (or attach evidence that it is unrelated), then rerun the protected gate on the resulting quiet exact head.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

cjagwani · 2026-07-30T01:51:55Z

The exact-head protected rerun still blocks approval at 8e0bbe24ab2013b5d9c6e25cc351d6d28845d27f.

The selected mcp-bridge (hermes) job failed in the real adapter path: the mock returned Hermes tool_search did not return the deferred target, so the authenticated MCP proof token was never produced. OpenClaw MCP, Hermes inference-switch, and Hermes security-posture passed, which narrows this to Hermes deferred tool discovery rather than runner setup or general Hermes startup.

Evidence: https://github.com/NVIDIA/NemoClaw/actions/runs/30506251272/job/90756659597

The latest tool-name/parsing adjustment is therefore not sufficient for the protected Hermes lifecycle. I will re-review the next quiet revision; I am leaving this as a plain review comment, not Changes Requested.

prekshivyas

Approved. The remaining observations are non-blocking test-hardening opportunities. Please wait for the exact-head E2E / PR Gate to pass before merging.

## Summary Add the canonical pre-tag release entry for NemoClaw v0.0.98. The dated entry records the user-visible changes merged after v0.0.97 and links each release theme to its published documentation. ## Changes - Add `docs/changelog/2026-07-29.mdx` with the exact `## v0.0.98` release heading. - Summarize Hermes 0.19, Deep Agents Code automation and skill safety, readiness diagnostics, lifecycle recovery, uninstall behavior, messaging conflicts, dependency hardening, and bounded diagnostics. - Use the parser-safe MDX SPDX comment and root-absolute routes for published OpenClaw, Hermes, and Deep Agents documentation. ### Source summary - [#7849](#7849) -> `docs/changelog/2026-07-29.mdx`: Record the Hermes 0.19 runtime migration repairs for cron state, dashboard seeding, and MCP naming. - [#7662](#7662) -> `docs/changelog/2026-07-29.mdx`: Record bounded gateway and Docker subprocess diagnostics. - [#7850](#7850) -> `docs/changelog/2026-07-29.mdx`: Record verified no-clobber Deep Agents Code skill installation. - [#7848](#7848) -> `docs/changelog/2026-07-29.mdx`: Record post-reboot delivery-chain recovery for visible OpenClaw sandboxes. - [#7831](#7831) -> `docs/changelog/2026-07-29.mdx`: Record OpenShell gateway-state preservation during uninstall. - [#7827](#7827) -> `docs/changelog/2026-07-29.mdx`: Record the removal of upstream test sources from published Hermes images. - [#7775](#7775) -> `docs/changelog/2026-07-29.mdx`: Record the blocking diagnostic for unsupported `DOCKER_HOST` values. - [#7833](#7833) -> `docs/changelog/2026-07-29.mdx`: Record reviewed Python dependency baselines for Hermes and Deep Agents Code images. - [#7771](#7771) -> `docs/changelog/2026-07-29.mdx`: Record the managed Hermes Agent 0.19.0 upgrade. - [#7811](#7811) -> `docs/changelog/2026-07-29.mdx`: Record fail-closed messaging channel conflict handling. - [#7797](#7797) -> `docs/changelog/2026-07-29.mdx`: Record the managed non-interactive Deep Agents Code JSON envelope. - [#7782](#7782) -> `docs/changelog/2026-07-29.mdx`: Record the storage-remediation readiness capability. - [#7784](#7784) -> `docs/changelog/2026-07-29.mdx`: Record the 120-second OpenShell readiness budget for sandbox recreation. - [#7810](#7810) -> `docs/changelog/2026-07-29.mdx`: Record rejection of stale Deep Agents Code security inventories. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: `test/changelog-docs.test.ts` validates the native changelog contract, including the version heading, MDX SPDX comment, and published routes. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `docs/changelog/2026-07-29.mdx` was reviewed against `docs/CONTRIBUTING.md` and `WRITING.md` for release meaning, terminology, structure, voice, sentence form, MDX structure, published routes, and code-sample presentation. The changelog contract passed 6 tests. The docs build completed with 0 errors and 2 existing Fern warnings. - Agent: Codex CLI   ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable. `scripts/prepare-dgx-station-host.sh` is unchanged. - Station profile/scenario: Not applicable. - Result: Not applicable. - Supporting evidence: Not applicable. ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run validate:pr` passed after refreshing `origin/main` when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run test/changelog-docs.test.ts` passed 6 tests. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Not run for this documentation-only change. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — The build completed with 0 errors and 2 existing Fern warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) — Native changelog entries use the required parser-safe MDX SPDX comment and do not use frontmatter. --- Signed-off-by: San Dang <sdang@nvidia.com>  ## Summary by CodeRabbit - **New Features** - Added managed Hermes upgrades with verified releases, version reporting, and preserved configuration contracts. - Improved Deep Agents Code JSON output and skill installation behavior. - Added clearer Docker host and system readiness reporting. - Improved post-reboot delivery recovery and sandbox readiness timing. - **Bug Fixes** - Preserved gateway state when uninstalling with `--keep-openshell`. - Prevented conflicting messaging credentials from blocking onboarding and rebuilds. - Improved gateway diagnostics, dependency security, runtime filesystem protection, and evidence handling. - **Documentation** - Published the v0.0.98 release notes.

## Summary `nemoclaw shields down` replaced the complete live OpenShell policy and dropped generated policy entries for registered Model Context Protocol (MCP) servers. This change reconciles only exact NemoClaw-managed MCP entries during Shields transitions, so a surviving server remains reachable while removed servers stay removed. Stacked on prerequisite #8130, which makes Shields deadline recovery serialize with lifecycle mutations without signaling the lock owner, this focused fix supersedes the MCP portion of #7980. ## Related Issue Fixes #7952 ## Changes - Prove managed MCP policy ownership from exact agreement between the sandbox registry, committed generated-policy record, and live gateway policy. - Save the owned MCP key manifest with the Shields snapshot, remove snapshot-time managed entries during restoration, and overlay only current exact entries. - Fail closed on ambiguous, stale, incomplete, malformed, or legacy ownership during manual transitions. At an expired deadline, omit unproven managed MCP entries and audit the omission instead of extending the Shields-down window. - Preserve current managed MCP entries when building the permissive runtime policy, while rejecting an unreadable or ambiguous live policy. - Clean staged runtime policy files across early failure paths. - Restore the Hermes live regression assertions at the actual failure boundary and around the unrelated server lifecycle. - Document MCP policy reconciliation for manual and automatic restoration. ## Failure Timing and Hermes Upgrade Context The original journey had a hidden Shields lifecycle between the first successful call to server A and the later lifecycle for server B: 1. Run `shields up`. 2. Restart the Hermes gateway. 3. Run `shields down`. 4. Exercise the configuration rollback path. 5. Add and remove B. 6. Call A. Boundary instrumentation recorded in #7952 showed that A remained healthy through Shields up and the gateway restart. It became unusable immediately after Shields down, which dropped A's generated MCP policy. The later failure after B was removed was only where the test noticed the already-broken route; B removal was a misleading correlation. This surfaced during the Hermes upgrade work because new coverage and upgrade repairs landed nearly back-to-back: - #7761 added the Hermes MCP helper containing Shields up, gateway restart, Shields down, and rollback. Its verification collected and imported the live target but did not run the complete live E2E. - #7771 upgraded Hermes the next day, but its selected E2Es skipped the `mcp-bridge` target. - #7849 repaired Hermes 0.19 migrations and updated MCP tool naming, allowing the live test to progress far enough to expose the later failure. - #7866 moved the explicit `mcp restart A` before the first post-removal call. Restart reapplied A's generated policy and masked the missing-policy state. The corrected regression order is: 1. Run `shields up`. 2. Restart the Hermes gateway. 3. Run `shields down`. 4. Call A immediately. 5. Exercise the configuration rollback path. 6. Add B, prove the DNS-rebinding connection is denied, remove B, and verify that A's managed policy is unchanged while B's policy is gone. 7. Call A before the later explicit restart. 8. Capture the authenticated rediscovery offset. 9. Run `mcp restart A` without resupplying the secret. 10. Call A and verify authenticated rediscovery. Whole-policy Shields replacement and the filesystem-only runtime merge predate the Hermes upgrade. This is a latent NemoClaw Shields policy-composition defect detected by expanded Hermes regression coverage, not a Hermes upgrade regression. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent exact-head Codex security review passed all nine categories at `18039569796d6ac7604de032edb7abf84f2c73c4`; no findings. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: Reviewed `docs/manage-sandboxes/runtime-controls.mdx` and `docs/reference/commands.mdx`, all rendered guide variants, changed operator-facing text, comments, test titles, and the Hermes E2E chronology. Verified claims against source, issue #7952, and PRs #7761, #7771, #7849, and #7866. `npm run docs` completed with 0 errors and 2 existing Fern warnings. - Agent: Codex Desktop   ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable; `scripts/prepare-dgx-station-host.sh` is unchanged. - Station profile/scenario: Not applicable. - Result: Not applicable. - Supporting evidence: Not applicable. ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run validate:pr` passed after refreshing `origin/main` when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Focused CLI 123/123, integration 11/11, E2E support 13/13, `npm run typecheck:cli`, `npm run checks:repository`, test-size guardrail, E2E semantic phase plans, and serial `npm run test:changed` 674/674 passed. - [x] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: [Standard PR CI run 30824992396](https://github.com/NVIDIA/NemoClaw/actions/runs/30824992396) passed. One inherited 50 ms lifecycle-lock assertion timing flake passed on the failed-job rerun without a code change. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) `npm run docs` passed with 0 errors and 2 existing Fern warnings, so the warning-free checkbox remains unchecked. No new documentation pages were added. Trusted E2E [run 30826792180](https://github.com/NVIDIA/NemoClaw/actions/runs/30826792180) passed all 10 selected checks: cloud inference, cloud onboard, security posture, inference routing, MCP bridge, MCP bridge dev, network policy, onboard repair, onboard resume, and OpenShell credential-generation window. The primary review advisor reported no findings. Nemotron completed after retrying a protocol-only failure; its one test warning requested the exact transition/state ownership-mismatch deadline regression already present in `src/lib/shields/policy-transition.test.ts`, which passed. --- Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

fix(hermes): repair v0.19 runtime migrations

16b422a

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

jyaunches self-assigned this Jul 29, 2026

jyaunches added 2 commits July 29, 2026 16:37

test(e2e): target Hermes upgrade regressions

4c5e3f1

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

test(e2e): keep Bedrock assertion linear

117ddf2

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

jyaunches added the v0.0.98 label Jul 29, 2026

jyaunches marked this pull request as ready for review July 29, 2026 21:48

jyaunches requested a review from ericksoa July 29, 2026 21:49

cv approved these changes Jul 29, 2026

View reviewed changes

ericksoa approved these changes Jul 29, 2026

View reviewed changes

cv approved these changes Jul 29, 2026

View reviewed changes

jyaunches added 6 commits July 29, 2026 18:49

fix(hermes): repair v0.19 runtime migrations

c9db737

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

test(e2e): target Hermes upgrade regressions

57e8fe2

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

test(e2e): keep Bedrock assertion linear

4d597a1

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

fix(hermes): keep cron sealed during restart

71ad76b

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

merge: reconcile protected PR branch

751fe11

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com> # Conflicts: # agents/hermes/start.sh # scripts/managed-gateway-control.py # test/e2e/live/mcp-bridge-hermes-lifecycle.ts # test/managed-gateway-control.test.ts

docs(hermes): qualify pre-relocation build evidence

cea7192

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

coderabbitai Bot reviewed Jul 29, 2026

View reviewed changes

wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior labels Jul 29, 2026

jyaunches added 2 commits July 29, 2026 19:41

test(e2e): align Hermes MCP rollback state

3092d55

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

merge: reconcile current main

ae3cbc8

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

coderabbitai Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread test/e2e/support/mcp-bridge-hermes-lifecycle.test.ts Outdated

test(e2e): assert Hermes rollback mismatch

cdd62f9

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

test(e2e): align Hermes MCP tool names

8e0bbe2

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

prekshivyas approved these changes Jul 30, 2026

View reviewed changes

jyaunches merged commit 376beb5 into main Jul 30, 2026
127 of 139 checks passed

jyaunches deleted the codex/fix-hermes-upgrade-e2e-regressions branch July 30, 2026 02:13

coderabbitai Bot mentioned this pull request Jul 30, 2026

fix(security): extend Perl remediation to sibling images #7864

Merged

23 tasks

sandl99 mentioned this pull request Jul 30, 2026

docs: add v0.0.98 release entry #7873

Merged

23 tasks

This was referenced Jul 30, 2026

fix(hermes): store dashboard home as a profile #7892

Open

feat(onboard): add managed startup profile schema #7946

Merged

perf(e2e): move recovery repetition into integration tests #7942

Merged

jyaunches mentioned this pull request Jul 31, 2026

fix(shields): preserve managed MCP policies #7980

Closed

23 tasks

mercl-lau mentioned this pull request Jul 31, 2026

Hermes onboard fails at sandbox image build Step 137 — cron-executions.db ownership validation broken in v0.0.98 #7981

Closed

coderabbitai Bot mentioned this pull request Jul 31, 2026

fix(hermes): support legacy gateway image builds #7989

Merged

23 tasks

jyaunches mentioned this pull request Jul 31, 2026

epic(hermes): replace upgrade-sensitive coupling with explicit managed contracts #8004

Open

6 tasks

This was referenced Jul 31, 2026

refactor(hermes): add versioned CLI adapter #8025

Merged

refactor(hermes): generate configurations from one policy #8026

Open

jyaunches mentioned this pull request Aug 3, 2026

fix(shields): preserve managed MCP policies #8141

Merged

23 tasks

Uh oh!

Conversation

jyaunches commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Type of Change

Quality Gates

Documentation Writer Review

DGX Station Hardware Evidence

Trusted E2E Failure and Fix

Verification

Uh oh!

copy-pr-bot Bot commented Jul 29, 2026

Uh oh!

coderabbitai Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-code-quality Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Overview

TypeScript / code-coverage/plugin

TypeScript / code-coverage/cli

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review Advisor — No blocking findings reported

Model lanes

E2E guidance

Warnings

PRA-1 Warning — Cover cron ledger writes after a locked gateway restart

Uh oh!

ericksoa left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cjagwani commented Jul 30, 2026

Uh oh!

cjagwani commented Jul 30, 2026

Uh oh!

prekshivyas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jyaunches commented Jul 29, 2026 •

edited

Loading

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading

github-code-quality Bot commented Jul 29, 2026 •

edited

Loading

github-actions Bot commented Jul 29, 2026 •

edited

Loading

`PRA-1` Warning — Cover cron ledger writes after a locked gateway restart