feat(sandbox): add a durable stop hold that prevents restart until explicit release

[ericksoa]

User Story

As an operator responding to suspicious or otherwise unwanted agent activity, I want to stop a sandbox and place a durable activation hold at the OpenShell gateway, so that every OpenShell client is prevented from restarting it until an explicit release while the sandbox identity and workspace remain available.

Problem Statement

OpenShell now provides storage-preserving, driver-neutral sandbox stop and sandbox start operations. Stop makes exec, SSH, forwarding, and exposed services unavailable and persists Stopped across gateway restart, but StartSandbox remains a valid operation for the same retained sandbox.

A product layered on OpenShell can record its own local “do not restart” flag, but that flag is not authoritative at the OpenShell lifecycle boundary. A direct CLI, SDK, API client, reconciler, or another controller can still request StartSandbox. Implementing the hold independently in every consuming product would duplicate lifecycle state and produce inconsistent guarantees.

NVIDIA/NemoClaw#10140 is a concrete downstream request for a quarantine-style operator command. NemoClaw can provide the convenience UX, but a restart prohibition that applies to all OpenShell clients needs a gateway-owned primitive.

Impact / Why This Matters

Today an operator can stop a sandbox while retaining its workspace, but preventing reactivation depends on coordination outside OpenShell. That is sufficient for ordinary idle resource management, not for an explicit operator hold where accidental or competing-client restart must be rejected.

The downstream alternatives are either incomplete (guard only NemoClaw-managed activation paths) or duplicative (add a second sandbox lifecycle authority above OpenShell). A small gateway-enforced hold would give NemoClaw and other orchestrators one portable contract across Docker, Podman, Kubernetes, VM, and future compute drivers.

Proposed Design

Add a public sandbox lifecycle hold operation and a separate explicit release operation. The final command names can be chosen during triage; one possible workflow is:

openshell sandbox hold my-sandbox --reason "unexpected outbound activity"
openshell sandbox release-hold my-sandbox --hold-id <id>

Externally observable behavior:

• Applying a hold durably records bounded, non-secret hold metadata before requesting the existing storage-preserving stop operation.
• The gateway rejects StartSandbox and any gateway reconciliation path that would reactivate the sandbox while the hold is active, regardless of which CLI, SDK, API client, or consuming product makes the request.
• If the stop request fails or its outcome is inconclusive, the hold remains active and status reports that the stop was not confirmed. The operation must not claim complete containment.
• Sandbox status exposes the active hold ID and creation time, plus bounded operator reason metadata if supported.
• Release requires an explicit release operation bound to the current hold. Release clears the hold but does not start the sandbox.
• Delete remains a separate operation with its existing state-removal semantics.
• The sandbox record, policy, provider attachments, settings, service definitions, and persistent workspace remain retained.
• The gateway composes this with the existing driver-neutral stop/start lifecycle; compute drivers do not need a separate quarantine API unless a concrete backend requirement later demonstrates one.

This is a lifecycle authorization and durable-intent feature, not a claim of forensic preservation or complete incident response. Credential revocation, evidence bundles, operation journals, independent postcondition observers, and cross-system atomicity are out of scope.

Acceptance Criteria

• A held sandbox has durable, bounded, non-secret hold metadata visible through the public API and CLI status.
• The hold survives gateway restart.
• StartSandbox and gateway-owned automatic reactivation reject a held sandbox before compute mutation.
• Applying a hold uses the existing per-sandbox lifecycle serialization and requests the existing driver-neutral stop behavior.
• A stop failure or ambiguous result leaves the hold active and is reported without claiming the sandbox is fully contained.
• Release is an explicit authorized operation bound to the active hold and does not start the sandbox.
• Repeating hold or release requests has deterministic, non-destructive behavior.
• Delete remains available according to its existing authorization and cleanup contract.
• Public CLI, gRPC, and supported SDK surfaces expose the capability consistently.
• Tests cover concurrent start/hold, gateway restart, stop failure, stale release identity, release-without-start, retained state, and at least one real compute-driver E2E path.

Alternatives Considered

• Use sandbox stop only: retains state and shuts down access, but any authorized client can immediately call sandbox start; it does not express an operator hold.
• Keep the hold only in NemoClaw: can guard NemoClaw commands but cannot guard direct OpenShell clients or other controllers, and makes NemoClaw a second lifecycle authority.
• Add a driver-specific quarantine RPC: places policy and durable intent below the gateway even though the gateway already owns lifecycle state, authorization, reconciliation, and the cross-driver stop contract.
• Delete the sandbox: prevents restart but removes the sandbox identity and may remove the workspace/evidence the operator intends to retain.
• Use “quarantine” as the public term: understandable for operators, but OpenShell already uses quarantine terminology for fail-closed policy generations. A neutral “hold” name may avoid conflating the two; naming does not change the required behavior.

Agent Investigation

Verified against OpenShell main at e64b0352e8965bec7e5683cb231ca38f9d4760ce.

• feat(sandbox): add storage-preserving suspend and resume lifecycle #2652 and merged PR feat(sandbox): add stop and start operations #2653 established public, durable, storage-preserving stop/start across the CLI, gRPC API, SDKs, gateway reconciliation, and supported compute drivers.
• The gateway documentation states that stopped sandboxes cannot be connected to, executed in, transferred to, forwarded from, or reached through exposed services, and that stable Stopped intent survives gateway restart.
• The gateway already owns per-sandbox lifecycle serialization and persists Stopping -> Stopped -> Starting intent before driver mutation.
• Both StopSandbox and StartSandbox currently use the public sandbox:write authorization surface; the sandbox resource has no distinct hold state or start guard.
• No existing OpenShell issue found in searches for quarantine, lifecycle hold, freeze, or guarded restart defines this capability.

[elezar]

Question: Does this reduce to identifying sanboxes that are "held" so that starting them again requires an extra step (or this info can be queried before starting them again)? Or should Held sandboxed not be restartable at all?

I would prefer extending the existing stop-start command pair instead of introducing a second pair which would then suffer from the same shorcomings as the original pair. (keep and un-keep, for example would also allow a kept sandbox to be restarted).