fix(snapshot): use gateway metadata for VM-driver health checks

[se7en-agent]

Summary

Fix nemoclaw <sandbox> snapshot create on macOS Apple Silicon VM-driver sandboxes by checking live OpenShell gateway health through gateway metadata instead of the legacy cluster container. This prevents healthy VM-driver sandboxes from being rejected before snapshot creation starts.

Related Issue

Fixes #3567

Changes

• Treat openshellDriver: "vm" sandboxes like Docker-driver gateway paths for snapshot gateway health checks.
• Use OpenShell status / gateway info metadata via the shared gateway health classifier instead of inspecting openshell-cluster-nemoclaw for VM-driver sandboxes.
• Preserve the existing legacy gateway guard for non-Docker/VM paths so stopped legacy cluster gateways still fail closed.
• Add a CLI-level regression test for VM-driver snapshot creation with healthy gateway metadata and no legacy cluster container.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Verification

• npx prek run --all-files passes
• npm test passes
• Tests added or updated for new or changed behavior
• No secrets, API keys, or credentials committed
• Docs updated for user-facing behavior changes
• make docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Verify the snapshot creation on my MacOS apple Silicon device:

 node bin/nemoclaw.js my-assistant snapshot create --name my-snapshot
  Creating snapshot of 'my-assistant' (--name my-snapshot)...
  ✓ Snapshot v1 name=my-snapshot created (12 directories, 0 files)
    /Users/sevenc/.nemoclaw/rebuild-backups/my-assistant/2026-05-19T02-42-04-301Z

---

Signed-off-by: Se7en-Agent se7en.agent.ai@gmail.com

Summary by CodeRabbit

• Bug Fixes

  • Improved gateway running-state detection for snapshot commands, making snapshots more reliable for Docker- and VM-driven sandboxes by using metadata-based gateway health probing.

• Tests

  • Added test helpers and suites that validate snapshot creation against healthy VM-driven gateways and guard against false "failed to query" errors.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7caa0d9f-429a-40a1-abb5-aad1414d7468

📥 Commits

Reviewing files that changed from the base of the PR and between 6be0028 and b54cf60.

📒 Files selected for processing (1)

• src/lib/actions/sandbox/snapshot.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/actions/sandbox/snapshot.ts

---

📝 Walkthrough

Walkthrough

Snapshot runtime now detects gateway readiness for Docker/VM-driven sandboxes via OpenShell metadata (status + gateway info) evaluated with isGatewayHealthy; other drivers still use container State.Running. Tests add helpers and a healthy VM-driver environment to validate snapshot creation.

Changes

Gateway metadata probing and snapshot tests

Layer / File(s)	Summary
Gateway metadata probing logic src/lib/actions/sandbox/snapshot.ts	Removes stripAnsi, imports isGatewayHealthy, and changes gateway-running detection: queries OpenShell status and gateway info for named and active gateways and evaluates readiness via isGatewayHealthy for docker/vm; falls back to Docker container State.Running for other drivers.
Test helpers and stopped-gateway refactor test/snapshot-gateway-guard.test.ts	Adds writeExecutable() and writeSandboxRegistry() helpers and refactors the stopped-gateway environment builder to use them while preserving stale openshell sandbox list and a non-running docker inspect.
Healthy VM-driver gateway environment test/snapshot-gateway-guard.test.ts	Introduces makeHealthyVmGatewayEnv() that writes a sandbox registry with openshellDriver: "vm" and stubs openshell (gateway info, sandbox list, ssh-config, status), ssh, and docker inspect to simulate a healthy VM-driver gateway.
VM gateway snapshot creation test test/snapshot-gateway-guard.test.ts	New test suite verifies alpha snapshot create --name baseline succeeds against the healthy VM-driver environment and does not output the "Failed to query live sandbox state from OpenShell" error.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• NVIDIA/NemoClaw#3454: Changes onboarding to record sandbox openshellDriver as "docker", which affects driver-based probing logic used here.

Suggested labels

Platform: ARM64, Docker

Suggested reviewers

• ericksoa

Poem

🐰 I hop where registries gleam in the night,

I whisper to OpenShell, "Are you all right?"
Metadata answers, health lights the track,
VM and Docker nod — snapshots no longer lack.
A rabbit cheers softly: tests pass, whee!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: using gateway metadata for VM-driver health checks in snapshot operations, which directly addresses the core objective of fixing snapshot creation on VM-driver sandboxes.
Linked Issues check	✅ Passed	The code changes directly address issue #3567 by switching snapshot gateway health checks from legacy cluster container probing to OpenShell gateway metadata for VM-driver sandboxes, enabling snapshot creation to succeed on Apple Silicon with live Docker-driver/OpenShell gateways.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to fixing snapshot gateway health checks for VM-driver sandboxes; modifications to snapshot.ts update health detection logic and test file adds VM-driver gateway environment builder and regression test aligned with issue #3567 requirements.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

test/snapshot-gateway-guard.test.ts (1)

84-141: ⚡ Quick win

Consider refactoring to use the new test helpers.

The makeStoppedGatewayEnv() function manually creates the sandbox registry and executable stubs, duplicating logic now available in writeSandboxRegistry() and writeExecutable(). Refactoring to use these helpers would improve consistency and maintainability.

♻️ Example refactor

 function makeStoppedGatewayEnv(prefix: string): Record<string, string> {
   const home = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
   const localBin = path.join(home, "bin");
   fs.mkdirSync(localBin, { recursive: true });
+  writeSandboxRegistry(home, "alpha");

-  const registryDir = path.join(home, ".nemoclaw");
-  fs.mkdirSync(registryDir, { recursive: true });
-  fs.writeFileSync(
-    path.join(registryDir, "sandboxes.json"),
-    JSON.stringify({
-      sandboxes: {
-        alpha: {
-          name: "alpha",
-          model: "test-model",
-          provider: "nvidia-prod",
-          gpuEnabled: false,
-          policies: [],
-        },
-      },
-      defaultSandbox: "alpha",
-    }),
-    { mode: 0o600 },
-  );

   // openshell lies: sandbox list exits 0 and lists alpha as Ready even though
   // the gateway container is down (reads stale local registry/cache).
-  fs.writeFileSync(
-    path.join(localBin, "openshell"),
+  writeExecutable(path.join(localBin, "openshell"), [
-    [
-      "#!/bin/sh",
       'if [ "$1" = "sandbox" ] && [ "$2" = "list" ]; then',
       '  printf "NAME STATUS\\nalpha Ready\\n"',
       "  exit 0",
       "fi",
       "exit 0",
-    ].join("\n"),
-    { mode: 0o755 },
-  );
+  ]);

   // docker inspect: returns "false" for State.Running (gateway stopped).
-  fs.writeFileSync(
-    path.join(localBin, "docker"),
+  writeExecutable(path.join(localBin, "docker"), [
-    [
-      "#!/bin/sh",
       'if [ "$1" = "inspect" ]; then',
       '  echo "false"',
       "  exit 0",
       "fi",
       "exit 0",
-    ].join("\n"),
-    { mode: 0o755 },
-  );
+  ]);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/snapshot-gateway-guard.test.ts` around lines 84 - 141, Replace the
manual file/dir setup in makeStoppedGatewayEnv with the shared test helpers:
call writeSandboxRegistry to create the .nemoclaw/sandboxes.json (passing the
same registry object with alpha/defaultSandbox and file mode 0o600) and use
writeExecutable to create the two executables ("openshell" and "docker") in the
temp bin directory with the same script bodies and mode 0o755; preserve
returning the same env object (HOME and PATH including the created local bin)
and keep the script behavior (openshell returns "alpha Ready" on `sandbox list`,
docker returns "false" for inspect) so tests remain functionally identical while
removing duplicated fs logic in makeStoppedGatewayEnv.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/actions/sandbox/snapshot.ts`:
- Around line 204-221: The function probeDockerDriverGatewayRunning() is
misnamed because it handles both Docker and VM drivers; rename it to
probeGatewayMetadataHealth (or probeGatewayRunningViaMetadata) and update all
call sites (including the usage that currently invokes
probeDockerDriverGatewayRunning at the sandbox snapshot logic) to the new name;
ensure you update any exports, imports, and tests that reference
probeDockerDriverGatewayRunning and keep the function body (calls to
captureOpenshell and isGatewayHealthy) unchanged.

In `@test/snapshot-gateway-guard.test.ts`:
- Line 189: The test suite name string in the describe block is
misleading—rename the describe title from "snapshot Docker-driver gateway guard"
to match the actual VM-driver scenario (e.g., "snapshot VM-driver gateway guard"
or "snapshot Docker/VM-driver gateway guard") so it reflects the test using
makeHealthyVmGatewayEnv() (and any other VM-specific setup); update the
describe() call's first argument accordingly to keep test semantics unchanged.

---

Nitpick comments:
In `@test/snapshot-gateway-guard.test.ts`:
- Around line 84-141: Replace the manual file/dir setup in makeStoppedGatewayEnv
with the shared test helpers: call writeSandboxRegistry to create the
.nemoclaw/sandboxes.json (passing the same registry object with
alpha/defaultSandbox and file mode 0o600) and use writeExecutable to create the
two executables ("openshell" and "docker") in the temp bin directory with the
same script bodies and mode 0o755; preserve returning the same env object (HOME
and PATH including the created local bin) and keep the script behavior
(openshell returns "alpha Ready" on `sandbox list`, docker returns "false" for
inspect) so tests remain functionally identical while removing duplicated fs
logic in makeStoppedGatewayEnv.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4c2f0d44-e21a-43cf-bbfc-c4f883227b42

📥 Commits

Reviewing files that changed from the base of the PR and between 5a03166 and 4523318.

📒 Files selected for processing (2)

• src/lib/actions/sandbox/snapshot.ts
• test/snapshot-gateway-guard.test.ts

[wscurran]

✨ Thanks for submitting this detailed PR about fixing the snapshot creation issue on macOS Apple Silicon VM-driver sandboxes. This proposes a code change to use OpenShell gateway metadata for health checks instead of the legacy cluster container, which should resolve the issue reported in #3567.

---

Related open issues:

• #3567 [macOS][Sandbox] nemoclaw snapshot create fails on Apple Silicon: "Failed to query live sandbox state from OpenShell"

[wscurran]

✨ Thanks for submitting this detailed PR about fixing the snapshot creation issue on macOS Apple Silicon VM-driver sandboxes. This proposes a code change to use OpenShell gateway metadata for health checks instead of the legacy cluster container, which should resolve the issue reported in #3567.

---

Related open issues:

• #3567 [macOS][Sandbox] nemoclaw snapshot create fails on Apple Silicon: "Failed to query live sandbox state from OpenShell"