fix(state): restore SQLite state files via staged atomic replace (#7312)

[nvshaxie]

Summary

nemoclaw <sandbox> snapshot restore failed on every Hermes sandbox with Failed files: runtime/state.db, kanban.db, which breaks all QA tests that use useClonedSandboxLifecycle (~40 sanity tasks). The restore now validates each SQLite backup into a staged database owned by the restoring user and replaces the live database atomically, so the restore succeeds and the restored databases keep working for the gateway user. Failed validation leaves the live database and its sidecars unchanged.

Related Issue

Fixes #7312

Root cause

The Hermes gateway runs as the gateway user with umask 0007 and creates runtime/state.db and kanban.db at mode 0640 (SQLite creates database files at 0644 & ~umask), owner gateway, group sandbox. Snapshot restore runs over SSH as the sandbox user: group read lets snapshot create succeed, but the restore script opened the live database for an in-place sqlite3 online backup write, which fails deterministically with sqlite3.OperationalError: attempt to write a readonly database. The trailing os.chmod(dst, 0o660) would also fail for a non-owner. The copy strategy never hit this because it stages a temp file and renames over the target, which needs only directory write permission.

Reproduced mechanically in a container with the production ownership layout (gateway:sandbox 0640 database, setgid 2770 state directory, restore as sandbox): the old command fails with the exact error; the staged replace succeeds, restores the expected content, and the gateway user can still open the restored database read-write afterward.

Changes

• src/lib/state/state-file-restore.ts: the sqlite_backup restore command validates the backup into a mktemp staged database in the destination's parent directory (keeping the PRAGMA quick_check gate and both symlink guards), sets 0660 on the staged copy this user owns, renames it over the destination, and drops the replaced database's stale -wal/-shm sidecars. The Python step no longer chmods or writes the gateway-owned live file.
• src/lib/actions/sandbox/snapshot-hermes-gateway-hint.ts (+ co-located test): after a successful Hermes restore that actually restored a manifest-declared sqlite_backup file, print run \nemoclaw gateway restart`— the running gateway holds the replaced inode open and serves pre-restore state until it reopens the database. Consumer:runSnapshotRestoresuccess path insnapshot.ts`.
• docs/manage-sandboxes/backup-restore.mdx: Hermes-variant note that a running gateway serves pre-restore state databases until it is restarted.
• test/state-file-restore-command.test.ts: executable Linux coverage proves successful replacement of a read-only database, mode 0660, stale-sidecar removal, failure preservation of the live database and sidecars, and refusal of symlinked parents and targets.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes — docs/manage-sandboxes/backup-restore.mdx gains a Hermes-variant note about the post-restore gateway restart
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex maintainer review at exact head ea9a66de2 passed all nine security categories; independent review remains requested from @cv and @ericksoa
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: docs/manage-sandboxes/backup-restore.mdx accurately limits the Hermes restart instruction to restores containing state databases. The CLI hint matches the manifest-declared sqlite_backup condition. npm run docs passed.
• Agent: Codex Desktop

DGX Station Hardware Evidence

Not applicable — scripts/prepare-dgx-station-host.sh is unchanged.

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result: npm run test:changed passed 89/89; focused snapshot tests passed 7/7; local restore-contract tests passed 3 with 4 Linux-only executions selected for CI; npm run typecheck:cli passed
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: not applicable to this narrow restore fix; npm run check:diff passed the complete path-scoped gate
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only) — not applicable; no new page

---

Signed-off-by: Shawn Xie shaxie@nvidia.com

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • After restoring Hermes state, the CLI now displays a “gateway restart” reminder for the restored sandbox.

• Bug Fixes

  • SQLite-based state restores now use a staged copy with an atomic swap into place.
  • Restores now clean up stale -wal/-shm sidecars and keep the database file permissions as intended.

• Documentation

  • Updated the snapshot restore guide with a Hermes-specific restart note.

• Tests

  • Added coverage for Hermes restore hint output and SQLite restore command behavior, including WAL/SHM cleanup and permission handling.

[coderabbitai]

📝 Walkthrough

Walkthrough

Snapshot restore now stages SQLite databases for permissioned atomic replacement and removes stale WAL/SHM files. Successful Hermes restores conditionally print gateway restart guidance, with unit, integration, and documentation updates.

Changes

Snapshot restore behavior

Layer / File(s)	Summary
Atomic SQLite restore flow src/lib/state/state-file-restore.ts, test/state-file-restore-command.test.ts	SQLite restores use staged files, apply permissions before atomic replacement, remove stale sidecars, preserve symlink guards, and include command-generation and Linux integration tests.
Hermes restore guidance src/lib/actions/sandbox/snapshot-hermes-gateway-hint.ts, src/lib/actions/sandbox/snapshot.ts, src/lib/actions/sandbox/snapshot-hermes-gateway-hint.test.ts, docs/manage-sandboxes/backup-restore.mdx	Successful restores invoke a helper that prints a gateway restart instruction only for Hermes sandboxes with restored SQLite state files, with positive and no-output tests plus matching restore documentation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SnapshotRestore
  participant SQLiteRestore
  participant HermesHint
  participant TargetDirectory
  SnapshotRestore->>SQLiteRestore: restore SQLite state file
  SQLiteRestore->>TargetDirectory: stage, chmod, and atomically replace database
  SQLiteRestore->>TargetDirectory: remove stale WAL/SHM sidecars
  SnapshotRestore->>HermesHint: provide restored files and agent
  HermesHint-->>SnapshotRestore: print gateway restart hint for Hermes SQLite restore

Loading

Suggested labels: integration: hermes

Suggested reviewers: cv, ericksoa

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The restore flow now handles sqlite_backup restores safely and the tests/docs reflect the Hermes state-file fix for #7312.
Out of Scope Changes check	✅ Passed	The added hint, restore logic, tests, and docs are all directly tied to the SQLite restore fix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: restoring SQLite state files with a staged atomic replace.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/7312-snapshot-restore-sqlite-staged-replace

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

test/state-file-restore-command.test.ts (1)

67-85: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Prefer behavioral assertions over generated-shell text.

These substring and ordering checks lock the test to the current shell implementation. Keep coverage through command execution—like the integration test below—and assert readonly-target replacement, permissions, sidecar cleanup, and symlink refusal as outcomes instead.

As per path instructions, “Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/state-file-restore-command.test.ts` around lines 67 - 85, Replace the
generated-shell substring and ordering assertions in the state-file restore test
with command-execution coverage through the public restore boundary. Assert the
observable outcomes: readonly-target replacement succeeds with the expected
permissions, stale WAL/SHM sidecars are removed, and symlinked state parents or
SQLite targets are refused; avoid inspecting shell text or
implementation-specific command ordering.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/state-file-restore-command.test.ts`:
- Around line 67-85: Replace the generated-shell substring and ordering
assertions in the state-file restore test with command-execution coverage
through the public restore boundary. Assert the observable outcomes:
readonly-target replacement succeeds with the expected permissions, stale
WAL/SHM sidecars are removed, and symlinked state parents or SQLite targets are
refused; avoid inspecting shell text or implementation-specific command
ordering.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 029e1fdf-6191-4cda-a122-40e72d380951

📥 Commits

Reviewing files that changed from the base of the PR and between 426af01 and dcb973b.

📒 Files selected for processing (5)

• src/lib/actions/sandbox/snapshot-hermes-gateway-hint.test.ts
• src/lib/actions/sandbox/snapshot-hermes-gateway-hint.ts
• src/lib/actions/sandbox/snapshot.ts
• src/lib/state/state-file-restore.ts
• test/state-file-restore-command.test.ts

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-7344.docs.buildwithfern.com/nemoclaw

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / medium confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

Since last review: 0 prior items resolved · 0 still apply · 0 new items found

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume, state-backup-restore, upgrade-stale-sandbox

2 optional E2E recommendations

• rebuild-hermes
• snapshot-commands

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit ea9a66d in the fix/7312-snapshot-re... branch remains at 96%, unchanged from commit 5faed2c in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit ea9a66d in the fix/7312-snapshot-re... branch remains at 80%, unchanged from commit 5faed2c in the main branch.

Show a code coverage summary of the most impacted files.

File	main 5faed2c	fix/7312-snapshot-re... ea9a66d	+/-
src/lib/actions...box/snapshot.ts	80%	80%	0%
src/lib/messagi...nnels/policy.ts	100%	100%	0%
src/lib/onboard...der-metadata.ts	100%	100%	0%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/securit...ntial-filter.ts	93%	93%	0%
src/lib/trace.ts	94%	94%	0%
src/lib/platform.ts	84%	89%	+5%
src/lib/actions...gateway-hint.ts	0%	100%	+100%

---

_{Updated July 24, 2026 16:59 UTC}

[wscurran]

✨ Thanks for the fix. This restores snapshot restore for Hermes sandboxes and unblocks the cloned sandbox lifecycle tests.

---

Related open issues:

• #7312 [All Platforms][Sandbox] Snapshot restore fails on runtime/state.db and kanban.db — breaks cloned sandbox lifecycle

[prekshivyas]

Reviewed exact head ea9a66de2. The staged SQLite restore now fails closed, preserves the live database and sidecars on validation failure, atomically replaces valid restores, and limits the Hermes restart hint to restored manifest-declared SQLite state. Focused tests, broad CI, CodeRabbit, both advisors, and the authenticated exact-head E2E retry are green; no unresolved review threads remain.