Skip to content

fix(snapshot): never capture or restore OpenClaw runtime auth state#6882

Merged
cv merged 2 commits into
NVIDIA:mainfrom
harjothkhara:fix/6852-skip-runtime-auth-state-restore
Jul 15, 2026
Merged

fix(snapshot): never capture or restore OpenClaw runtime auth state#6882
cv merged 2 commits into
NVIDIA:mainfrom
harjothkhara:fix/6852-skip-runtime-auth-state-restore

Conversation

@harjothkhara

@harjothkhara harjothkhara commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

nemoclaw snapshot create runs every backed-up .json file through the credential sanitizer, which scrubs the key/token fields of OpenClaw's device identity (identity/device.json Ed25519 keypair) and paired-device token store (devices/). snapshot restore then wiped the sandbox's live pairing state and installed those scrubbed copies, so after a clone or restore the gateway destabilized and ordinary CLI clients (openclaw agent, openclaw tui) failed with GatewayCredentialsRequiredError. This surfaced at v0.0.75 once OpenClaw 2026.6.10 and the local-loopback pairing path (#6291) stopped providing an ambient gateway token that had been masking the corruption. After this change, backup no longer captures device identity or pairing state and restore never touches it; OpenClaw regenerates the device identity on demand and NemoClaw auto-pair re-pairs on connect.

Related Issue

Fixes #6852

Changes

  • Add a runtime_auth_state_dirs agent-manifest field (src/lib/agent/definition-types.ts, src/lib/agent/defs.ts). Its consumer is the snapshot backup/restore path in src/lib/state/sandbox.ts, which needs to distinguish machine-local auth state (device keys, pairing tokens) from durable user state. A direct exclusion in code would hard-code OpenClaw-specific dir names into the shared backup engine; the manifest keeps the agent contract declarative and per-agent. The reader validates that every entry is also a state_dirs member. Protected by src/lib/agent/runtime-auth-state-dirs.test.ts.
  • Declare identity and devices as runtime_auth_state_dirs in agents/openclaw/manifest.yaml. They stay in state_dirs so destroy still wipes them from the durable volume.
  • backupSandboxState filters runtime_auth_state_dirs out of the captured set; restoreSandboxStateInternal filters them out of the pre-restore cleanup and tar extract, keyed on the current target manifest so legacy snapshots that still contain those dirs are also skipped. Protected by test/snapshot-runtime-auth-state.test.ts (a round-trip integration test that fails on unpatched main).
  • Document the exclusion in docs/manage-sandboxes/backup-restore.mdx (OpenClaw variant).

AI-assisted contribution.

Type of Change

  • Code change with doc updates
  • Code change (feature, bug fix, or refactor)
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: change removes credential-scrubbed device-auth state from the backup/restore path; net effect is that machine-local auth secrets are never captured into snapshots (strictly reduces credential surface). Requesting maintainer sensitive-path review.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result: npx vitest run test/snapshot-runtime-auth-state.test.ts src/lib/agent/runtime-auth-state-dirs.test.ts → 5 passed; the integration test is red on unpatched main (expected [ 'agents', 'identity', 'devices' ] to not include 'identity'), green on this branch.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Note: dynamic Linux/Brev end-to-end verification was not run, so the gateway-crash symptom is not confirmed live. The credential-loss root cause and fix are static-traced (through NemoClaw's backup sanitizer and OpenClaw's GatewayCredentialsRequiredError throw site) and covered by the red-green integration test above.


Signed-off-by: harjoth harjoth.khara@gmail.com

Summary by CodeRabbit

  • New Features
    • Added agent manifest support for declaring runtime authentication state directories.
    • OpenClaw runtime authentication state is excluded from backups and handled safely during restores.
  • Bug Fixes
    • Backups no longer include OpenClaw identity keys and paired-device tokens.
    • Restores no longer overwrite the sandbox’s live gateway authentication state, even from legacy snapshots.
  • Documentation
    • Clarified what is excluded from snapshots and how identity/device state is regenerated.
  • Tests
    • Added validation and regression coverage for backup exclusion and restore safety.

Snapshot backups run every .json file through the credential sanitizer,
which scrubs the key/token fields of OpenClaw's device identity
(identity/device.json) and paired-device token store (devices/). Restoring
a snapshot then wiped the sandbox's working pairing state and installed
those corrupt copies, so the gateway destabilized and every ordinary CLI
client failed with GatewayCredentialsRequiredError once OpenClaw 2026.6.10
and the local-loopback pairing path (NVIDIA#6291) stopped providing an ambient
gateway token.

Declare identity/ and devices/ as runtime_auth_state_dirs in the OpenClaw
manifest: they stay in state_dirs so destroy still wipes them from the
durable volume, but backup never captures them and restore never touches
them — including for legacy snapshots whose embedded manifests still list
them, since the current target manifest is authoritative. OpenClaw
regenerates the device identity on demand and NemoClaw auto-pair re-pairs
on connect, matching fresh-onboard behavior.

Fixes NVIDIA#6852

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: harjoth <harjoth.khara@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 22b63dc4-5769-4e6c-9381-99593c0cdfde

📥 Commits

Reviewing files that changed from the base of the PR and between 5f2fe26 and 10637bc.

📒 Files selected for processing (1)
  • test/snapshot-runtime-auth-state.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/snapshot-runtime-auth-state.test.ts

📝 Walkthrough

Walkthrough

OpenClaw declares identity and devices as runtime authentication directories. Agent loading validates and exposes them, while sandbox backup and restore exclude them. Tests cover parsing, legacy snapshots, and preservation of live authentication state.

Changes

Runtime authentication state

Layer / File(s) Summary
Runtime authentication contract and agent loading
agents/openclaw/manifest.yaml, src/lib/agent/definition-types.ts, src/lib/agent/defs.ts, src/lib/agent/*test*, test/helpers/base-image-test-harness.ts
Adds runtime_auth_state_dirs, validates entries against state_dirs, exposes runtimeAuthStateDirs, and updates fixtures and parsing tests.
Snapshot backup and restore filtering
src/lib/state/sandbox.ts, test/snapshot-runtime-auth-state.test.ts, docs/manage-sandboxes/backup-restore.mdx
Excludes runtime authentication directories from backup manifests and restore operations, with regression coverage for legacy snapshots and documentation of the restore behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested labels: area: sandbox, bug-fix

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: preventing OpenClaw runtime auth state from being captured or restored.
Linked Issues check ✅ Passed [#6852] The PR excludes OpenClaw identity/devices from snapshot backup/restore and adds tests/docs covering the reported credential-loss regression.
Out of Scope Changes check ✅ Passed All changes relate to snapshot safety for OpenClaw runtime auth state, with supporting manifest, code, tests, and docs updates.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: full-e2e, hermes-e2e, onboard-repair, onboard-resume

3 optional E2E recommendations
  • state-backup-restore
  • snapshot-commands
  • ubuntu-repo-cloud-openclaw

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

The codebase-growth-guardrails check forbids newly added test files from
introducing if statements. Replace the manual HOME/PATH/OPENSHELL_BIN
save-and-restore conditionals with vi.stubEnv / vi.unstubAllEnvs, which
records and restores prior env values (including unset) without branching.
No behavior change; the embedded fake-ssh script's conditionals live inside
a template string and were never counted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: harjoth <harjoth.khara@gmail.com>
@wscurran wscurran added bug-fix PR fixes a bug or regression integration: openclaw OpenClaw integration behavior labels Jul 14, 2026
@wscurran

Copy link
Copy Markdown
Contributor

✨ Thanks for the fix. Preventing snapshot from capturing and restoring OpenClaw runtime auth state avoids gateway destabilization after clone or restore. Ready for maintainer review.


Related open issues:


Related open PRs:


Related open issues:

@prekshivyas prekshivyas added the v0.0.84 Release target label Jul 14, 2026
@prekshivyas prekshivyas self-assigned this Jul 14, 2026
@prekshivyas

Copy link
Copy Markdown
Collaborator

@cv Exact head 10637bc861c636d4019ad51b72a7c6884c0aefb9 has been maintainer-reviewed and all non-credentialed CI is green. The E2E gate selected full-e2e, hermes-e2e, onboard-repair, and onboard-resume and is waiting on the protected approve-credentialed-e2e-skip-for-fork-pr deployment. Please approve Review deployments on run 29373914686. No fork code received repository secrets.

@cv

cv commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

@prekshivyas approved, please review

@prekshivyas prekshivyas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 10637bc. Snapshot filtering and legacy-restore behavior are correct and covered; CI, PR Advisor, CodeRabbit, and the policy-approved fork E2E gate are clear.

@cv
cv merged commit 8eebef3 into NVIDIA:main Jul 15, 2026
69 of 71 checks passed
cv pushed a commit that referenced this pull request Jul 16, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical `docs/changelog/2026-07-15.mdx` entry with the exact
`## v0.0.84` heading for the release candidate range from `v0.0.83`
through `710d2b36b9eebcb6bca3c2b2f796a1bdb69c3a31`.
Fills two owner-page gaps for model-aware local inference health and
pre-write OpenClaw candidate validation.

## Changes

- Add the complete shared Fern changelog entry for `v0.0.84`, with
literal CLI names and root-absolute OpenClaw and Hermes routes.
- Document that sandbox status and doctor compare the configured Ollama
or vLLM model with provider inventory without issuing a completion.
- Document that host-side OpenClaw `config set` validates the complete
candidate before replacing live config or reaching gateway restart.
- Reconcile the `v0.0.84` release label with the commit range. PR #6773
is already contained in `v0.0.83` and remains documented there; CI,
test-harness, docs-infrastructure, and `.js` to `.mts` migration-only
changes require no additional user guidance.

### Source summary

- [#6882](#6882) ->
`docs/manage-sandboxes/backup-restore.mdx`,
`docs/changelog/2026-07-15.mdx`: Explain that OpenClaw runtime identity
and pairing state are excluded from snapshots and ignored during
restore.
- [#6873](#6873) ->
`docs/inference/set-up-ollama.mdx`, `docs/changelog/2026-07-15.mdx`:
Record the Ollama requested-model environment fallback and interactive
default.
- [#6835](#6835) ->
`docs/changelog/2026-07-15.mdx`: Include the sandbox name in the
documented rebuild resume-recovery behavior.
- [#6886](#6886) ->
`docs/inference/custom-endpoint-security.mdx`,
`docs/inference/set-up-openai-compatible-endpoint.mdx`,
`docs/changelog/2026-07-15.mdx`: Explain the exact-host trusted-private
endpoint opt-in and retained SSRF boundaries.
- [#6887](#6887) ->
`docs/reference/commands.mdx`, `docs/changelog/2026-07-15.mdx`: Document
Telegram channel health verdicts, summary behavior, and exit status.
- [#6863](#6863) ->
`docs/manage-sandboxes/lifecycle.mdx`, `docs/changelog/2026-07-15.mdx`:
Add the missing model-inventory behavior for local status and doctor
checks.
- [#6902](#6902) ->
`docs/manage-sandboxes/runtime-controls.mdx`,
`docs/changelog/2026-07-15.mdx`: Add the missing pre-write OpenClaw
candidate-validation contract.
- [#6916](#6916) ->
`docs/changelog/2026-07-15.mdx`: Preserve the failed-session
fresh-install recovery correction in the release entry.
- [#6934](#6934) ->
`docs/reference/commands.mdx`, `docs/reference/troubleshooting.mdx`,
`docs/security/credential-storage.mdx`, `docs/changelog/2026-07-15.mdx`:
Summarize completed-prompt checkpointing and validated credential reuse
during OpenClaw resume.
- [#6898](#6898) ->
`docs/inference/switch-models.mdx`,
`docs/inference/switch-providers.mdx`,
`docs/reference/troubleshooting.mdx`, `docs/changelog/2026-07-15.mdx`:
Explain Hermes dashboard convergence after in-place inference changes.
- [#6711](#6711) ->
`docs/manage-sandboxes/run-sandboxes.mdx`,
`docs/manage-sandboxes/uninstall-nemoclaw.mdx`,
`docs/reference/architecture.mdx`, `docs/reference/commands.mdx`,
`docs/changelog/2026-07-15.mdx`: Summarize port-scoped host state and
uninstall preservation.
- [#6767](#6767) ->
`docs/inference/configure-model-limits.mdx`,
`docs/inference/set-up-ollama.mdx`,
`docs/reference/troubleshooting.mdx`, `docs/changelog/2026-07-15.mdx`:
Record the Hermes `64000`-token Ollama floor and unchanged OpenClaw
floor.
- [#6862](#6862) ->
`docs/get-started/quickstart.mdx`,
`docs/inference/verify-inference-route.mdx`,
`docs/changelog/2026-07-15.mdx`: Explain retryable not-ready
finalization for unhealthy inference routes.
- [#6766](#6766) ->
`docs/security/tcb-boundary.mdx`, `docs/changelog/2026-07-15.mdx`:
Document definitive stale transition-lock recovery and fail-closed
ambiguous cases.
- [#6948](#6948) ->
`docs/manage-sandboxes/manage-mcp-servers.mdx`,
`docs/changelog/2026-07-15.mdx`: Include Hermes MCP apply-state race
recovery in the release entry without changing the established user
workflow.
- [#6964](#6964) ->
`docs/reference/troubleshooting.mdx`, `docs/changelog/2026-07-15.mdx`:
Record complete agent-specific fresh-install and resume recovery
commands.
- [#6883](#6883) ->
`docs/get-started/quickstart.mdx`, `docs/inference/set-up-vllm.mdx`,
`docs/reference/platform-support.mdx`, `docs/changelog/2026-07-15.mdx`:
Summarize the DGX Station Nemotron Ultra express path and pinned
managed-vLLM recipe.
- [#6985](#6985) ->
`docs/inference/set-up-vllm.mdx`, `docs/reference/commands.mdx`,
`docs/changelog/2026-07-15.mdx`: Capture the final automated and
interactive storage-warning behavior.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior —
`test/changelog-docs.test.ts` validates the dated-entry structure, exact
version heading, and preserved history.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts` (6 passed)
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — not run for this doc-only change.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors; Fern reported the unchanged unauthenticated
redirect-check and light-theme contrast warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the native changelog entry uses the required parser-safe MDX SPDX
comment and intentionally has no frontmatter.

---
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added the v0.0.84 changelog entry covering setup, endpoint onboarding,
model handling, sandbox readiness, recovery, channel status, and
configuration safeguards.
* Clarified that sandbox health checks validate configured models
against local Ollama and vLLM provider inventories without generating
completions or consuming tokens.
* Documented that invalid runtime configuration changes are rejected
while preserving the existing working configuration.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PR fixes a bug or regression integration: openclaw OpenClaw integration behavior v0.0.84 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gateway crashes and loses auth credentials in cloned/restored sandboxes on v0.0.75

4 participants