The box gets a memory boundary: protected services, bounded user work, measured knobs

[pythonlearner1025]

The box was one flat memory pool. A workspace under pressure died one of two ways: a fast squeeze let the kernel pick victims by size alone (dockerd, the actor, or PID 1 — a PID 1 kill lost every session), and a slow squeeze pushed the whole VM into direct reclaim, where cloudflared stalled while alive and the workspace read "connecting" with no OOM line anywhere. The second one took down a real workspace, motivated this work, and was reproduced end-to-end in the load campaign below.

The boundary

• blitz-system.slice — s6, cloudflared, gateway, sshd, dufs, ttyd, watch. memory.min 256M, oom_score_adj -900, pids.max 512. The kernel must not reclaim the services that carry the box to its user.
• blitz-user.slice — terminal tabs, ssh sessions, the actor (its SDK spawns agents beyond any wrapper), Remote Control, DinD. memory.max below the container ceiling, a 500M memory.high throttle band, 2G swap budget, pids.max 4096, and per-leaf memory.oom.group=1 so a runaway dies as one unit while the tab next door survives.
• Placement rides where work actually starts: the tmux pane command, an sshd ForceCommand, dockerd --cgroup-parent, and the service run scripts. uid 1000 is delegated exactly enough to move its own work between its own leaves; the system slice stays root-owned.
• The VM gains zram (25%, with the linux-modules-extra fallback the stock image needs), a container ceiling that reserves 512M for the host, --pids-limit, and a HEALTHCHECK that tests reachability (and treats a missing tunnel token as idle, not sick).
• A box that cannot own cgroups — unprivileged, or box-in-box dev — bails to flat cleanly: no slice, no noise, every enter a passthrough. The smoke suite asserts the full layout where the memory controller is delegated and the clean bail where it is not. Side effect on real VMs: inner containers get real memory accounting for the first time (today's fleet runs DinD with "No memory limit support").

Assertion zero

Enabling controllers on the container root triggers cgroup v2's no-internal-process rule, so the whole design hinged on one unverifiable-in-nested-DinD fact: does docker exec still work? Measured on a real cx23: runc attaches exec processes to PID 1's cgroup (the system slice), not the namespace root. Exec, the bootstrap health loop, the box updater, and the smoke suite all keep working.

The load campaign (real Hetzner VMs, real Cloudflare tunnel)

Two cx23s, label+prefix-guarded lifecycle, pre-run inventory diffed at teardown (proof: all 9 pre-existing servers untouched). Oracles that never sit inside the victim: a 4 Hz host-side probe of the box's sshd, cgroup accounting read from the host cgroupfs, and a real cfd_tunnel polled at 2 Hz through the Cloudflare edge. Baseline = same image, boundary off, no limits, no swap (today's fleet).

Scenario	Baseline (today)	Boundary (this PR)
l1 6G balloon	2303 ms host freeze, probe+tunnel losses, P1+P6 FAIL	272 ms, clean
l2 8x900M build	989 ms, 35 kernel OOM lines	344 ms, 0 lines
l3 DinD bomb	kernel kill, 459 ms	contained in docker.slice, 277 ms
l7 combined	1042 ms, 14 OOM lines	291 ms, kills stay in user leaves
l8 95% ramp	1865 ms freeze, kernel kill	284 ms, bounded leaf kills
l9 hot near-edge hold	collapse at +114 s, 91/474 probes lost, 0 OOM lines, no recovery, even root ssh dead — power reset required. The production failure, reproduced.	0/1710 probes lost, 292 ms worst gap, new session accepted

Tunnel view: 0.0% dead polls inside every treatment window; the baseline l9 window shows a 14.8 s silence and multi-second latencies — the state the webapp renders as "connecting". Recovery from the wedged state: Hetzner power reset to origin-answering tunnel in 45 s (the control plane currently has no reboot action; worth adding as a rescue path).

Knob sweep (10 configs)

No criterion flips across min 128M-384M, gap 250M-1G, zram 0-50%. Shipped: min 256M (~3x the measured 85 MB protected set; validated under the combined scenario), gap 500M, zram 25%. z50 absorbed the l8 ramp with 300x fewer throttle events — the smoothness knob if bigger boxes ever want it. Every knob is a BLITZ_CG_* env var overridable per box via /etc/blitz/box-limits.env without an image rebuild.

Full design and campaign docs: docs/MEMORY-BOUNDARY.md. Harness: packages/box/test/{memory-load.sh,hetzner-load-lab.sh,tunnel-oracle.sh,analyze-load.py}.

Rollout notes: image + control-plane bootstrap ship together in the next tag; existing VMs keep their image for life, so the boundary reaches the fleet as workspaces recycle. The maximal recipe bootstrap stays under Hetzner's 32K user-data cap (1032 bytes of headroom; the zram/limits reasoning lives in TS comments, which never ship).

🤖 Generated with Claude Code