volumes: writable catalog disks need a guest filesystem shutdown lifecycle

[CMGS]

Summary

Volumes v1 (#66) shipped read-only only. Write-enabled catalog images were cut
because correct semantics need a clean guest filesystem shutdown on every path
that ends a claim, and sandboxd has no such path today. This issue carries the
decided design and the execution plan for that lifecycle feature.

Why v1 refused it

Every claim-ending path converges on the same forced VM removal:

• Manager.Release (sandboxd/pool/claim.go)
• Manager.reapOnce TTL reap (same file)
• Manager.rollbackClaim — persist or egress-arm failure, after the VM is live
  and the volumes are already attached and mounted
• Manager.quarantineClaim — restart egress resync (sandboxd/pool/reconcile.go)
• startup reconciliation collecting unowned sbx-* survivors, plus the
  cancellation-immune queueRemoval/retryRemoval queue in
  sandboxd/pool/remove.go

All of them destroy the VM without touching the guest. With mount -o ro that
costs nothing. With a writable mount the guest page cache is discarded
mid-write. sandboxd can also be killed outright, and there is no steady-state
VMM liveness sweep, so the design must assume the unmount never ran.

Verified hardware facts to build on

From the #66 round on bare metal (Cloud Hypervisor, rt:24.04 guest):

• Cloud Hypervisor holds a per-backing-file lock — fcntl OFD locks
  (F_OFD_SETLK, whole file; block/src/io/fcntl.rs). A second writer is
  refused, a reader is refused while a writer holds it, and many concurrent
  readers are fine. The lock is per-host kernel state; cross-node behavior
  depends on whether the backing filesystem propagates fcntl locks.
• Host attach took 4-6 ms and the guest device appeared 5-10 ms later.
  Discovery reads /sys/block/<dev>/serial; no udev involved.
• Cocoon refuses snapshot and hibernate while a hot-attached disk is present,
  so the capture sentinel fails closed independently of the mount mode.
• VM removal closes the device and never deletes the backing file.
• silkd serves one request per connection, so an N-volume unmount is N separate
  vsock dials on a teardown path.

Design

A journaling filesystem already makes a hard VM kill crash-consistent: the
image's committed state survives. What the lifecycle actually has to solve is
(a) flushing the guest page cache on graceful paths so workload writes are
durable, (b) the dirty-journal × ro-reader interaction — ext4 refuses a
read-only mount that needs replay, and replay needs the write lock — and
(c) never mounting a tenant-written filesystem on the host, so recovery must
happen inside a guest. Three mechanisms carry all of it; the rest is plumbing.

1. Graceful quiesce. A new engine method unmounts every rw mount in
reverse order over silkd exec before VM removal, on every termination path
that still has a guest: release, TTL reap (volume claims cannot hibernate, so
they always take the destroy arm and never archive), rollback after a
persist/egress-arm failure, and best-effort during restart quarantine. Budget:
2 s per umount, ~5 s total. Failure never blocks teardown — it falls through
to the marker. Claim-setup failure skips quiesce entirely: the claim was never
handed out, no workload writes happened, and the marker converges it.

2. Write-ahead dirty marker. Before the first rw attach of an image,
sandboxd durably creates <path>.dirty beside the image; it is removed only
after the image's rw mount was cleanly unmounted. A killed sandboxd, a dead
VMM, or a quiesce timeout leaves the marker in place. Same invariant style as
the archive-delete markers (#65): marker durably down before the transition
that would otherwise be unrecoverable. Sidecar placement over data_dir is
deliberate: operator-visible, survives a data_dir wipe, and on shared storage
it travels with the image. (A backing path that is not a regular file — e.g. a
block device — has no sidecar home; if that combination ever lands, the marker
falls back to data_dir keyed by name. Out of scope here.)

3. Dirty handling = block, don't heal.

• dirty + next rw claim → allowed: mount rw replays the journal as a
  side effect, and a clean release then clears the marker. Recovery needs no
  new machinery.
• dirty + next ro claim → 409 (ErrVolumeNeedsRecovery). Auto-heal is
  rejected: it would be a hidden rw window inside a ro claim, and its replay
  takes the write lock, conflicting with concurrent readers.

Admission registry. The pool manager keeps an in-memory
map[name]{writers, readers}, checked in resolveVolumes under the existing
m.mu section and rebuilt from adopted claims at restart. Live writer → rw
and ro both refused; live reader → rw refused; both as 409 (ErrVolumeBusy).
The registry mirrors the hypervisor lock exactly — CH stays the enforcement,
the registry turns its failures into fast 409s before paying attach cost.

Fleet: zero wire change. writable is fleet-uniform catalog metadata
under the existing name-is-fleet-wide-identity invariant (same rule as ACLs),
so local metadata validates the mode and routing reuses the VolumeOwners
intersection unchanged. Cross-node exclusion is not a mechanism: a writable
image is held by exactly one node (operator contract, same weight as
"replacing an attached image is operator error"), and since nodes only
advertise locally present paths, every claim for that name — ro and rw —
funnels to that node's admission registry.

Smaller calls, decided:

• The claim usage event records which applied names were rw, so billing can
  discriminate write access.
• Writable entries keep directio defaulting to off; the durability nuance
  (host page cache between guest flush and device) is documented rather than
  forcing on.
• Per-volume mode is ro or rw, defaulting to ro; rw requires
  operator writable: true on the catalog entry. A writable entry with a
  multi-tenant access list is a bidirectional channel between tenants;
  docs/security.md recommends single-tenant access lists for writable
  entries. Both SDKs carry the full surface.

Concurrency semantics

For one image name:

• rw ∥ rw: refused (409, CH lock as belt).
• rw ∥ ro: refused in both directions (409, CH lock as belt). Not a
  limitation — ext4/xfs are not cluster filesystems; a writer under live
  readers hands every reader torn metadata. The reader cannot corrupt the
  image; the writer corrupts every reader's view.
• ro ∥ ro: fine (the v1 baseline, shared host page cache).
• Sequential rw → release → many ro: the supported publish/update
  workflow. The dirty marker guards exactly that boundary: clean writer
  release → readers proceed; crashed writer → ro claims 409 until one rw
  claim replays and releases cleanly.
• Cross-node: all claims for a writable name route to its single holder,
  so the node-local registry is the single admission point by construction.

Execution plan

component	files	change
types	sandboxd/types/types.go, api.go	Volume.Mode ("" ≡ ro; validated ro|rw in ValidateVolumes). claimDTO persists []types.Volume as-is, so mode rides along. VolumeInfo.Writable so discovery explains a rejected rw claim without probing.
config	sandboxd/config/config.go	VolumeSpec.Writable bool.
engine	sandboxd/engine/volume.go	diskAttachArgs drops --readonly for rw; MountVolume takes the mode (-o ro|rw); new QuiesceVolumes (reverse-order umount over silkd exec, bounded).
pool	sandboxd/pool/volume.go	catalogVolume.writable; resolveVolumes checks mode against writable, the dirty marker (ro + dirty → 409), and registry admission; applyVolumes writes the marker before a rw attach; new teardownVolumes helper (quiesce → clear marker → release registry).
	sandboxd/pool/claim.go	releaseResolved, the reapOnce destroy arm, and rollbackClaim run teardownVolumes before removal (rw claims only).
	sandboxd/pool/reconcile.go	adoption rebuilds the registry; quarantine quiesces best-effort.
	sandboxd/pool/pool.go	registry state; ErrVolumeBusy, ErrVolumeNeedsRecovery; engine interface gains QuiesceVolumes.
server	sandboxd/server/server.go	map both new errors to 409 (same table as ErrVolumeCapture).
mesh	—	no change.
SDK Go	sdk/go	Volume.Mode; Writable in discovery; applied entries echo mode.
SDK Python	sdk/python	_volume_body accepts {name, mount?, mode?} (drop the v1 mode rejection, keep rejecting tuples); client.volumes() surfaces writable.
e2e	fakeengine, volumesmoke, drift pin	ordering assertions: quiesce before remove on release/reap/rollback, no quiesce on setup failure; rw smoke leg; wire-surface pin update.
docs	deploy, sandboxd-api, sdk, sdk-python, security, cluster	writable contract, single-holder rule, marker semantics, directio durability note, bidirectional-channel warning.
cocoon	—	no change: attach without --readonly is the verified rw path, and VM removal closes the device, so detach never happens.

Unchanged by design: all four capture operations stay refused for any volume
claim (ro or rw); idle-sweep exclusion stays; wake is untouched (volume claims
never hibernate).

Hot-path cost: the volume-less and ro claim/release paths are unchanged. The
rw-only cost is N umount execs at release (silkd is one-request-per-connection,
so N vsock dials at ~0.3 ms each plus ms-class umount time). Registry
operations are O(1) inside already-taken m.mu sections.

Estimated size: +250-350 prod LOC across the repo.

Out of scope

Per-claim scratch disks, persistent workspaces, volume create/delete/mutation
APIs, image distribution, virtiofs, and Firecracker — unchanged from #66.
Block-device-backed catalog entries (see the marker note above).

Acceptance

• Bare-metal Cloud Hypervisor proof that a writable claim's data survives
  release.
• The crash sequence proven end-to-end: kill the VM under a writer → marker
  present → ro claim 409 → rw claim mounts (journal replay) → clean release →
  ro claims serve again.
• Admission proven in both directions: ro refused with 409 while a writer is
  live, rw refused while readers are live, second writer refused — as fast
  rejects before attach, with the CH lock verified as belt when admission is
  bypassed.
• Every termination path exercised: release, reap, rollback, quarantine, the
  cancellation-immune removal queue, and startup reconciliation; adopted
  claims rebuild the registry.
• Usage events record rw names; discovery shows writable.
• Release-path latency with rw volumes measured against main, and the
  volume-less release path shown unchanged.
• Full Go -race, dual-GOOS lint, ASL, Python Ruff/pytest, shell checks, and
  documentation diff checks pass.