sandboxd: cross-node snapshot placement — operator-token lifecycle, gossip redirect, peer heal

docs/cluster.md

@@ -123,6 +123,55 @@ handle `Sandbox.Promote` returns is still owner-bound — `template.New` and
 `template.Delete` dial the owner directly, no gossip involved — and is the
 race-free choice immediately after a promote.
 
+## Checkpoints on a cluster
+
+A checkpoint lives on whichever node captured it (the default
+[`checkpoint_dir`](deploy.md#configuration)) unless the store is shared (a
+FUSE mount or the s3 backend), in which case every node resolves every
+checkpoint directly and nothing below applies.
+
+On the default per-node store, a branch claim (`Checkpoint.New` /
+[`POST /v1/checkpoints/{id}/claim`](sandboxd-api.md#post-v1checkpointsidclaim))
+at a node that does not hold the record runs a tier order:
+
+1. **Local claim** — the fast path when the claiming node already holds it.
+2. **Probe + redirect** — a live `HEAD` fan-out (HMAC-signed when the mesh
+   carries a `cluster_key`) to every mesh peer in parallel, redirecting to
+   the first owners that answer — the same claim-redirect contract a warm
+   miss already uses. The answer is capped at 3 addresses: a hint, not an
+   exhaustive list of every owner. The probe and
+   the follow-up claim are not atomic: a peer can answer the probe, then
+   lose the record — a delete's broadcast lands, or its own TTL sweep runs
+   (below) — before the retry reaches it, so a redirect can go stale
+   between the two calls.
+3. **Heal** — when `checkpoint_peer_heal` is on and neither of the above
+   answered, the node pulls the record from a probed peer, validates its
+   shape and id before trusting it, and publishes it locally so later
+   branches are local — paid once per node. The pull is bounded to one
+   overall time budget across however many peers it tries (`healBudget` in
+   `store/peer`), and a node accepts only so many heals at once
+   (`maxConcurrentHeals` in `pool`) — past that cap it answers `503` rather
+   than queuing indefinitely.
+
+### Delete is eventual, not a fleet-wide revocation
+
+`Checkpoint.Delete` (`DELETE /v1/checkpoints/{id}`) removes the local
+record, then best-effort broadcasts the delete to every peer this node
+currently sees, so a replica a heal pulled earlier is cleaned up too, not
+just the original. A peer that is offline or partitioned during that
+broadcast keeps its copy until the checkpoint TTL ages it out. A healed
+replica carries the source checkpoint's original `CreatedAt`, so it becomes
+eligible for expiry at the same instant everywhere; each node then removes
+it on its own hourly sweep, which is independently phased and retries on
+failure. The window in which a deleted checkpoint stays branchable by an
+id-holder is therefore normally `checkpoint_ttl_hours` plus the wait for the
+next hourly sweep; a sweep that fails retries on a later one, so persistent
+sweep failure extends retention until one succeeds — the TTL is the eligibility
+point, not a hard ceiling. This is why heal *requires* a nonzero, fleet-matching
+TTL (see [cluster-invariant config](#cluster-invariant-config)).
+With TTL disabled it could never close at all, so that combination is
+rejected at config load.
+
 ## State ownership
 
 Each kind of state has exactly one source of truth, and a restart rebuilds
@@ -134,6 +183,7 @@ fully from it:
 | API-applied pool targets (`PUT /v1/pools`) | `<data_dir>/pools.json` (machine owned) | yes |
 | claims | the claims journal + `Reconcile` | yes |
 | placement hints (warm counts, template sets) | gossip | rebuilt |
+| checkpoint ownership | a live per-request probe (no gossip); a healed replica is this node's own persisted copy, aged out by `checkpoint_ttl_hours` | yes |
 
 Pools are managed API-first. The first time a node takes `PUT /v1/pools`, it
 writes the applied set to `<data_dir>/pools.json` and from then on **that file
@@ -162,7 +212,7 @@ Some config must match on every node or the cluster fails in confusing ways:
 
 | config | what breaks on mismatch |
 |---|---|
-| `api_token`, `tenants` | the SDK replays the authorizing token across a redirect; a peer missing that tenant/token answers 401 |
+| `api_token`, `tenants` | the SDK replays the authorizing token across a redirect; a mid-rotation peer missing that tenant/token answers 401, which the SDK treats as transient — the claim falls back to the origin node (which already authorized it) and resolves there |
 | `preview_secret` | a preview URL signed on one node fails verification on another |
 | `mesh.cluster_key` | nodes cannot join / decrypt gossip at all |
 | `egress_ca` cluster root | a guest checkpointed/redirected across nodes trusts the root; a divergent root fails interception |

docs/deploy.md

@@ -71,15 +71,16 @@ sandboxd reads one JSON file (`-config`, default
 | `bridge` / `network` | unset | egress-lane attachment: a host bridge device, or a CNI conflist name. Mutually exclusive; with neither set the node serves only the no-network lane. [Guarded egress](egress.md) needs the bridge form and rejects a CNI network at load |
 | `egress_ca` | unset | [HTTPS-interception](egress.md#https-interception) PKI: `root_cert` (the cluster root baked into intercepted guests; may bundle old+new roots during rotation) plus this node's `intermediate_cert`/`intermediate_key` from `sandboxd ca issue-intermediate`. Required when any pool rule sets `intercept` |
 | `api_token` | unset | the operator (root) credential: when set, guards the node-level endpoints (Bearer) with full access, including release-by-id cleanup. Per-sandbox tokens guard ordinary sandbox-scoped calls |
-| `tenants` | unset | multi-tenant tokens next to `api_token`: `[{"name": "acme", "token": "…", "max_claims": 50}]`. A tenant token reaches the resource-creating verbs (claim, fork, promote, checkpoint, preview) and everything it creates is stamped with the tenant name; operator surfaces (`GET /v1/sandboxes`, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `/metrics`) answer it 403. `max_claims` (0 = unlimited) caps that tenant's live claims next to the node-wide cap. Requires `api_token` set (operator surfaces need it). Names and tokens must be unique, tokens distinct from `api_token`. On a cluster all nodes must carry the same tenants set (the SDK replays a tenant token across a redirect; a peer missing that tenant answers 401), and per-node caps mean a tenant's effective cluster limit is `max_claims` × nodes. Empty = exactly the single-token behavior |
+| `tenants` | unset | multi-tenant tokens next to `api_token`: `[{"name": "acme", "token": "…", "max_claims": 50}]`. A tenant token reaches the resource-creating verbs (claim, fork, promote, checkpoint, preview) and everything it creates is stamped with the tenant name; operator surfaces (`GET /v1/sandboxes` and the per-id reads under it, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `/metrics`) answer it 403. `max_claims` (0 = unlimited) caps that tenant's live claims next to the node-wide cap. Requires `api_token` set (operator surfaces need it). Names and tokens must be unique, tokens distinct from `api_token`. On a cluster all nodes must carry the same tenants set (the SDK replays a tenant token across a redirect; a peer missing that tenant answers 401), and per-node caps mean a tenant's effective cluster limit is `max_claims` × nodes. Empty = exactly the single-token behavior |
 | `max_fork_count` | 16 | children a single `fork` may create; each is a full-RAM VM, so this bounds one request's memory blast radius to the node's capacity |
 | `refill_concurrency` | 0 (auto) | concurrent VM provisioning budget, shared by warm-pool refills, fork clones, and the reap/hibernate/reconcile engine batches. 0 sizes it from the node: `NumCPU*2/3` clamped to [4, 256] — a 384-core node gets 256; small nodes keep a floor of 4 |
 | `preview_listen` | (off) | address for a preview HTTP server that serves guest ports under signed URLs; needs `preview_secret` |
 | `preview_secret` | — | cluster-shared HMAC secret signing preview tokens (all nodes share one) |
 | `preview_advertise` | = `preview_listen` | the base URL a browser/proxy reaches this node's preview server at |
 | `checkpoint_dir` | `<data_dir>/checkpoints` | where checkpoints and promoted templates live. Point it at a shared FUSE mount (JuiceFS over object storage, NFS) and every node sharing the mount can branch every checkpoint — the path's filesystem is the operator's choice. One contract on any shared root (mount or bucket): a template key has a single writer — promotes go to the sandbox's owner node, and operators must not race promotes of one name from different nodes (checkpoint ids are node-generated and never collide). A checkpoint deleted on one node while another is mid-branch from it fails that branch visibly |
 | `checkpoint_store` | dir | checkpoint AND promoted-template backend (both live in one store root, id-namespaced ck_/tp_): `{"kind": "s3", "s3": {"bucket": "…", "prefix": "ck/", "endpoint": "…", "region": "…", "force_path_style": true}}` stores checkpoints in object storage (any node claims any checkpoint, no shared mount needed). Credentials come from the standard AWS chain (env/IAM role), never this file. A crash between upload and the meta.json commit marker leaves orphan objects invisible to listings — add an S3 lifecycle rule to reclaim them. Absent = the dir backend at `checkpoint_dir` |
-| `checkpoint_ttl_hours` | 0 (keep forever) | ages out checkpoints older than this; the sweep runs hourly and at startup. Explicit deletes never wait for it |
+| `checkpoint_ttl_hours` | 0 (keep forever) | ages out checkpoints older than this; the sweep runs hourly and at startup. Explicit deletes never wait for it. Must be nonzero and match fleet-wide when `checkpoint_peer_heal` is on — it is the expiry eligibility point for a healed replica a delete broadcast missed, after which its next successful hourly sweep removes it; persistent sweep failure extends retention until one succeeds, so it is not a hard ceiling |
+| `checkpoint_peer_heal` | false | on a cluster, lets a node pull a checkpoint it lacks from a peer — found via a live probe, not gossip — rather than failing the branch; see [placement lifecycle](cluster.md#checkpoints-on-a-cluster). Three requirements, all enforced at config load: a nonempty `api_token` (the blob transfer between peers authenticates with it; without one the raw record stream would be open), `mesh.cluster_key` set (the pull presents the fleet `api_token` to an address learned from the peer probe, so the gossip layer carrying that address must itself be authenticated), and `checkpoint_ttl_hours` nonzero (a replica a delete broadcast missed becomes eligible for expiry after it, and its next successful hourly sweep removes it — so it is the finite eligibility point, not an exact ceiling). A shared checkpoint store (`checkpoint_store` kind `s3`) ignores this setting — every node already resolves every checkpoint directly, so there is nothing to heal |
 | `warm_max` (pool entry) | 0 (static) | turns on the demand-adaptive watermark for that pool: the warm target rises from `warm` toward `warm_max` while claims arrive faster than the measured provision lead covers, and decays back over ~a minute of silence |
 | `max_claims` | 0 (unlimited) | node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster, a claim is first redirected to a warm peer) |
 | `audit_log` | false | append every relayed request frame's op + addressing fields (never payloads) to `<data_dir>/audit.jsonl`, size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`); preview accesses record as op `preview_dial`. A request frame whose first line exceeds 4 KiB is skipped, never truncated |

docs/sandboxd-api.md

@@ -7,7 +7,9 @@ All bodies are JSON. Three token kinds:
   resource-creating verbs (claim, fork, promote, checkpoint create/claim,
   preview mint) and the tenant-scoped listings/deletes below; everything a
   tenant creates is stamped with its name. Operator surfaces
-  (`GET /v1/sandboxes`, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `GET /metrics`)
+  (`GET /v1/sandboxes` and the per-id reads under it, `GET /v1/info`,
+  `PUT /v1/pools`, `POST/DELETE /v1/drain`, `GET /metrics`,
+  `GET /v1/checkpoints/{id}/blob`)
   answer a tenant token `403` — authenticated but not authorized; an unknown
   token stays `401`.
 - **sandbox** — every claimed sandbox carries its own bearer token guarding
@@ -73,7 +75,8 @@ Releasing an already-gone sandbox is 404 — the SDK treats it as success.
 
 ## POST /v1/sandboxes/{id}/hibernate
 
-Auth: the sandbox's own token. Atomically snapshots the VM and stops it,
+Auth: the sandbox's own token, or the root token by id (operator, like
+release). Atomically snapshots the VM and stops it,
 freeing its memory; the next agent access restores it transparently
 (sessions, processes, and memory state intact — cocoon's hibernate keeps the
 snapshot point and the stop coincident). Idempotent on an already-hibernated
@@ -83,6 +86,14 @@ node only provides the transition. 204 on success, 404 unknown id or wrong
 token, 409 on the egress lane (egress-lane sandboxes never hibernate; see
 [egress](egress.md)).
 
+## POST /v1/sandboxes/{id}/wake
+
+Auth: the sandbox's own token, or the root token by id (operator). Restores
+a hibernated (or archived) sandbox and leaves it running — waking is
+otherwise only a side effect of the next agent access, so this is the
+explicit form for warming a sandbox ahead of use. Idempotent on one already
+running. 204 on success, 404 unknown id or wrong token.
+
 ## POST /v1/sandboxes/{id}/fork
 
 Auth: the node `api_token` (Bearer) — forking creates node resources, like a
@@ -205,12 +216,56 @@ token, 409 egress-lane sandbox (see [egress](egress.md)).
 
 ## POST /v1/checkpoints/{id}/claim
 
-Auth: node API token; body `{"ttl_seconds": 0}`. Claims a fresh sandbox
-branched from the checkpoint (a normal claim response, attributed to the
-caller); the checkpoint's recorded key applies — the unguessable id is the
-capability to branch. 404 for an unknown checkpoint, 409 for an egress-lane
-checkpoint (see [egress](egress.md)), 429 node or calling tenant at
-`max_claims`, or the node draining.
+Auth: node API token; body `{"ttl_seconds": 0, "no_redirect": false}`.
+Claims a fresh sandbox branched from the checkpoint (a normal claim
+response, attributed to the caller); the checkpoint's recorded key applies
+— the unguessable id is the capability to branch.
+
+Checkpoints are node-local (unless the store is shared — see
+[Configuration](deploy.md#configuration)), so a miss here runs a tier order:
+
+1. This node checks its own store first.
+2. On a miss, it probes up to 3 peers directly — a parallel `HEAD` to each
+   (authenticated on an encrypted mesh, see below) — and answers exactly like
+   a warm-miss
+   [`POST /v1/claim`](#post-v1claim): `200 {"redirect": ["10.0.0.6:7777",
+   "10.0.0.7:7777"]}`, retry the same body (+`no_redirect: true`) at each
+   candidate until one answers. The probe and the follow-up claim are not
+   atomic: a peer can answer the probe, then lose the record — a delete's
+   broadcast lands, or its own TTL sweep runs (below) — before the retry
+   reaches it, so a redirect can go stale between the two calls.
+3. If nothing answers the probe (or `no_redirect` is set), and the node has
+   `checkpoint_peer_heal` enabled, it pulls the record from a probed peer
+   itself, validates it, publishes it locally, and serves the claim from
+   there — paid once per node. See the full
+   [placement lifecycle](cluster.md#checkpoints-on-a-cluster) for how the
+   three tiers fit together.
+
+404 for an unknown checkpoint (locally, and after redirect and heal both
+miss), 409 for an egress-lane checkpoint (see [egress](egress.md)), 429 node
+or calling tenant at `max_claims` or the node draining, 503 when the node's
+concurrent-heal cap is already full — retryable, and the response carries a
+`Retry-After` hint.
+
+## GET/HEAD /v1/checkpoints/{id}/blob
+
+The peer-transfer route behind the probe and heal above — internal, not
+part of the public API; an SDK caller has no reason to call it directly.
+
+- `GET` streams the whole record — guest memory, disk, and meta — as a tar.
+  Operator-token only: the stream carries no tenant scoping, and its only
+  real caller is a peer's heal pull, which presents the fleet `api_token`.
+  401 missing or unrecognized token, 403 a valid tenant token (authenticated
+  but not the operator), 404 unknown checkpoint.
+- `HEAD` is the ownership probe: 200 when this node holds a branchable
+  (non-archive) copy, 404 otherwise. On a mesh with `cluster_key` set the
+  request must carry `X-Cocoon-Probe`, an HMAC over the id and a coarse time
+  bucket keyed off a probe-specific derivation of the cluster key — verified
+  before any disk is touched, replayable for roughly a minute at most. On a
+  keyless mesh (redirect-only fleets have no shared secret to sign with) the
+  id itself remains the only capability, matching the mesh's own posture.
+  Repeat probes for one id are answered from a short positive cache on the
+  asking node, which a local delete evicts immediately.
 
 ## GET /v1/checkpoints
 
@@ -223,12 +278,57 @@ Auth: node API token. A tenant may delete only its own records — anything
 else is 404, never a hint the id exists; root deletes anything. 204 on
 success, 404 unknown.
 
+**Delete removes the local record and then best-effort broadcasts to peers
+so a healed replica does not outlive it — this is eventual best-effort
+cleanup, not a fleet-wide revocation.** A peer that is offline or
+partitioned during the broadcast keeps its copy until the checkpoint TTL
+ages it out. A healed replica carries the source's original `CreatedAt`, so
+it becomes eligible for expiry at the same instant on every node; the actual
+removal is each node's own hourly sweep, which is independently phased and
+retries on a later sweep if one fails. So a deleted checkpoint normally stops
+being branchable within `checkpoint_ttl_hours` plus a sweep interval, but a
+node whose sweeps keep failing holds its replica until one succeeds — the TTL
+is the eligibility point, not a hard ceiling. The TTL must also match
+fleet-wide, which the
+[cluster-invariant config](cluster.md#cluster-invariant-config) digest
+checks. A window always exists because `checkpoint_peer_heal` cannot be
+enabled with `checkpoint_ttl_hours: 0` — a replica that can outlive a delete
+must have a finite eligibility point. A shared
+checkpoint store skips the broadcast: every node already resolves every
+record directly, so there is no replica to chase. `?no_forward=1` marks a
+delete already arriving from another node's own broadcast, so it is not
+itself re-broadcast (loop prevention); it is an internal parameter, not one
+an SDK caller should set.
+
 ## GET /v1/sandboxes
 
 Auth: root only (tenant tokens get 403). The operator index: `{"sandboxes":
 [{id, key, deadline, hibernated, archived?, from_checkpoint?, claim_ref?}]}` —
 never tokens.
 
+## GET /v1/sandboxes/{id}
+
+Auth: root only. One live claim in the index-row shape above, so a
+reconcile loop can read a single sandbox without scanning the whole node
+listing. 404 unknown id.
+
+## GET /v1/sandboxes/{id}/stats
+
+Auth: root only. One sandbox's resource usage — the per-sandbox counterpart
+to the node-scoped `/metrics`:
+
+```json
+{"id": "sb_…", "cpu_count": 2, "mem_total_bytes": 1073741824,
+ "mem_used_bytes": 187654144, "mem_used_measured": true,
+ "hibernated": false, "measured_at": "…"}
+```
+
+`cpu_count`/`mem_total_bytes` come from the size tier. `mem_used_bytes` is
+the host VMM process's resident set — the only usage signal available
+without a guest agent; `mem_used_measured` is false when there is no VMM
+process to read (hibernated, or the PID is not yet known), so a zero is
+never mistaken for idle. 404 unknown id.
+
 ## GET /metrics
 
 Auth: root only (tenant tokens get 403). Prometheus text format,