The box gets a memory boundary: protected services, bounded user work, measured knobs

docs/MEMORY-BOUNDARY.md

@@ -0,0 +1,109 @@
+# The box memory boundary
+
+Every workspace VM runs one box container, and until this boundary existed
+the container was one flat memory pool. This page records the failure that
+motivated the split, the design, the knobs, and how to re-run the load
+campaign that sized them.
+
+## The failure
+
+A workspace under memory pressure has two ways to die, and the loud one is
+the rare one:
+
+- **Reclaim stall** (the common one). The VM has no swap, so the kernel
+  drops page cache and then spins in direct reclaim. Every process stalls —
+  including cloudflared, which stays alive but stops answering tunnel
+  heartbeats. The workspace reads "connecting". `dmesg` shows nothing,
+  `docker ps` shows a healthy container. Nothing restarts anything, because
+  nothing died.
+- **OOM kill** (the loud one). The kernel picks the biggest RSS. That is
+  usually the offending build or agent, but dockerd, the session actor, and
+  container PID 1 are all legal victims. A PID 1 kill restarts the whole box
+  and loses every tmux session in it.
+
+## The boundary
+
+```
+VM (Hetzner cloud server, Ubuntu 24.04)
+│  zram swap, 25% of RAM        — pressure degrades before it kills
+│  host reserve 512 MB          — host sshd + dockerd + blitz-box-update
+│                                 survive any box-internal storm
+└── box container   --memory <RAM-512M> --memory-swap <+2G> --pids-limit 8192
+    │
+    │  s6 oneshot `cgroups` runs blitz-cgroup init before anything else:
+    │  drain every pid out of the container root, then delegate +memory +pids
+    │
+    ├── blitz-system.slice        memory.min 256M (measured: no flip down to 128M; 256M is ~3x the protected set) · oom_score_adj -900 · pids 512
+    │     s6 tree, cloudflared, gateway, sshd, dufs, ttyd, watch
+    │     — the services that carry the box to its user
+    │
+    └── blitz-user.slice          memory.max total-min-headroom · high = max-500M
+          pids 4096 · swap 2G      — everything a member's work can grow
+          ├── tab-<session>        one per terminal tab      oom.group=1
+          ├── ssh-<pid>            one per ssh session       oom.group=1
+          ├── actor.scope          chat agents (SDK spawns)  oom.group=1
+          ├── rc.scope             claude.ai Remote Control  oom.group=1
+          ├── dockerd.scope        the DinD daemon itself
+          └── docker.slice         every inner container     max = user/2, oom.group=1
+```
+
+`memory.min` closes the stall: the kernel must not reclaim the protected
+set, so cloudflared keeps answering while user work thrashes. `memory.max`
+plus `oom.group` closes the wrong-victim kill: a runaway dies as one unit
+inside its own leaf, and the tab next door survives. `memory.high` sits a
+band below the ceiling so a runaway throttles visibly before it dies.
+
+## Where placement happens, and why there
+
+| Work enters through | Placed by | Why there |
+|---|---|---|
+| Terminal tab | the tmux **pane command** in `blitz-term` | tmux forks panes from its server, not from the launcher |
+| SSH / sftp | `ForceCommand /usr/local/libexec/blitz-ssh-session` | sessions fork from sshd, which lives in the protected slice |
+| Chat agents | the actor's own s6 `run` | the SDK spawns its own CLI copy beyond any wrapper |
+| Remote Control | its s6 `run` | it drives a full agent |
+| Inner containers | `dockerd --cgroup-parent` | dockerd otherwise creates cgroups outside every limit |
+
+uid 1000 is delegated the container-root `cgroup.procs` and the user slice —
+enough to move its own work between leaves it owns, and structurally unable
+to park anything in the protected slice, which stays root-owned.
+
+## Verified invariants
+
+- **docker exec survives delegation.** runc attaches exec processes to the
+  container init's cgroup (the system slice), not the namespace root, so the
+  no-internal-process rule never breaks exec. Proved on a real cx23
+  ("assertion zero" in the lab) — this was the one fact no nested
+  environment could answer, and the whole design rested on it.
+- **A box that cannot own cgroups boots flat.** Unprivileged containers and
+  box-in-box dev workspaces get a clean bail: no slice directory, no join
+  noise, every `blitz-cgroup enter` a silent passthrough. The smoke suite
+  asserts the full layout where the memory controller is delegated and
+  asserts the clean bail where it is not.
+- **Every knob is an env var** (`BLITZ_CG_*`, see `blitz-cgroup`), read from
+  the container environment, overridable per box via
+  `/etc/blitz/box-limits.env` on the VM without an image rebuild.
+
+## The load campaign
+
+`packages/box/test/` carries the whole rig:
+
+- `memory-load.sh` — runs ON the VM; 8 scenarios (agent balloon, parallel
+  build, DinD bomb, cache flood, fork storm, slow leak, combined, stall
+  edge). The oracle is an out-of-box probe of the box's sshd plus cgroup
+  accounting read from the host cgroupfs — never `docker exec`, which is
+  itself under test.
+- `hetzner-load-lab.sh` — orchestrates real cx23 VMs. Destructive calls are
+  double-constrained (label `blitz-test=oom` AND name prefix `oomtest-`),
+  the pre-run inventory is diffed at teardown, and only the orchestrator
+  holds the API key; run agents drive load purely over ssh.
+- `tunnel-oracle.sh` — a real cfd_tunnel through the box under test, polled
+  from outside. Any HTTP status is a live path; only transport failure is a
+  dead one.
+- `analyze-load.py` — folds probe, cgroup, dmesg, and tunnel logs into one
+  verdict table per arm.
+
+Pass criteria: P1 host probe gap < 2 s, P2 zero system-slice OOM kills,
+P5 a new session within 30 s of any kill, P6 tunnel error rate < 1%.
+
+Results and the sizing rationale for the shipped defaults live in the PR
+that introduced this page.

packages/box/Dockerfile

@@ -141,7 +141,7 @@ COPY packages/box/rootfs/ /
 RUN set -eux; \
     chmod 0755 /usr/local/bin/blitz /usr/local/bin/claude /usr/local/bin/codex \
       /usr/local/bin/blitz-cred-claude /usr/local/bin/blitz-cred-codex \
-      /usr/local/bin/blitz-rules; \
+      /usr/local/bin/blitz-rules /usr/local/bin/blitz-cgroup; \
     chmod 0755 /usr/local/libexec/blitz-* /etc/s6-overlay/s6-rc.d/*/run; \
     install -d -m 0755 /workspace /var/lib/blitz /srv/blitz-files; \
     ln -s /workspace /srv/blitz-files/workspace; \
@@ -169,4 +169,10 @@ ENV HOME=/var/lib/blitz/home \
 VOLUME ["/var/lib/blitz"]
 EXPOSE 22 7443 7444 7445
 STOPSIGNAL SIGTERM
+# A running container is not a reachable box. The start period covers the s6
+# graph and the first tunnel connect; after that an unhealthy verdict means the
+# box stopped answering while it kept running, which is exactly the state that
+# used to be invisible.
+HEALTHCHECK --interval=30s --timeout=10s --start-period=180s --retries=3 \
+  CMD /usr/local/libexec/blitz-healthcheck
 ENTRYPOINT ["/init"]

packages/box/actor/test/recipe-invocation-guest.test.ts

@@ -227,6 +227,15 @@ function tmuxCreateArgv(session: string): string[] {
   ];
 }
 
+/** Every pane command rides through the memory boundary before it execs:
+ * tmux forks panes from its long-lived server, so blitz-term wraps the
+ * command it hands tmux rather than joining a cgroup itself. On boxes where
+ * the boundary cannot exist the wrapper execs straight through, so the argv
+ * is identical everywhere. */
+function cgroupWrapArgv(session: string): string[] {
+  return ["/usr/local/bin/blitz-cgroup", "enter", `user/tab-${session}`, "--"];
+}
+
 function harnessCommand(harness: TuiHarness): string[] {
   return harness === "claude"
     ? ["claude", "--dangerously-skip-permissions", "--permission-mode", "bypassPermissions"]
@@ -297,6 +306,7 @@ describe("blitz-term recipe delivery", () => {
       expect(created.status, `${invocation.name}: ${created.stderr}`).toBe(0);
       expect(recordedArgv(box, "new-session-argv"), invocation.name).toEqual([
         ...tmuxCreateArgv(`${harness}-run`),
+        ...cgroupWrapArgv(`${harness}-run`),
         ...harnessCommand(harness),
         ...recipeFlags(harness, invocation.descriptor),
         invocation.promptBytes,
@@ -317,6 +327,7 @@ describe("blitz-term recipe delivery", () => {
     expect(second.status, second.stderr).toBe(0);
     expect(recordedArgv(box, "new-session-argv")).toEqual([
       ...tmuxCreateArgv("claude-second"),
+      ...cgroupWrapArgv("claude-second"),
       ...harnessCommand("claude"),
     ]);
   });
@@ -332,6 +343,7 @@ describe("blitz-term recipe delivery", () => {
     expect(attach.status, attach.stderr).toBe(0);
     expect(recordedArgv(box, "new-session-argv")).toEqual([
       ...tmuxCreateArgv("claude-same"),
+      ...cgroupWrapArgv("claude-same"),
       ...harnessCommand("claude"),
     ]);
     // The re-planted files sit untouched for a genuinely new session to take.
@@ -368,6 +380,7 @@ describe("blitz-term recipe delivery", () => {
     expect((await runTerm(crossed, ["codex", "run"])).status).toBe(0);
     expect(recordedArgv(crossed, "new-session-argv")).toEqual([
       ...tmuxCreateArgv("codex-run"),
+      ...cgroupWrapArgv("codex-run"),
       ...harnessCommand("codex"),
     ]);
     expectPlanted(crossed, true);
@@ -381,6 +394,7 @@ describe("blitz-term recipe delivery", () => {
         expect(result.status, `${invocation.name}: ${result.stderr}`).toBe(0);
         expect(recordedArgv(box, "new-session-argv"), invocation.name).toEqual([
           ...tmuxCreateArgv(`${harness}-run`),
+          ...cgroupWrapArgv(`${harness}-run`),
           ...harnessCommand(harness),
         ]);
         expectPlanted(box, true);

packages/box/rootfs/etc/blitz/sshd_config

@@ -17,3 +17,6 @@ AllowAgentForwarding yes
 PermitUserEnvironment no
 PrintMotd no
 Subsystem sftp internal-sftp
+# Places every session under the user memory ceiling. See
+# /usr/local/libexec/blitz-ssh-session for why this is a ForceCommand.
+ForceCommand /usr/local/libexec/blitz-ssh-session

packages/box/rootfs/etc/s6-overlay/s6-rc.d/actor/run

@@ -1,5 +1,13 @@
 #!/command/with-contenv bash
 set -euo pipefail
-exec /command/s6-setuidgid blitz /usr/bin/env \
+# The actor's whole job is to host agents, so it is user load. Its Claude
+# adapter spawns through the SDK, which starts its own CLI copy out of
+# reach of any wrapper here — placing the actor itself is what puts every
+# agent it starts under the ceiling, whichever adapter runs.
+#
+# The gateway, not the actor, is what the control plane talks to. A dead
+# actor costs the chat surface; it does not cost the box its tunnel, its
+# terminal, or its files.
+exec /usr/local/bin/blitz-cgroup enter user/actor.scope -- /command/s6-setuidgid blitz /usr/bin/env \
   HOME=/var/lib/blitz/home USER=blitz \
   /usr/local/bin/node --env-file=/etc/blitz/env.defaults /opt/blitz/actor/dist/main.js

packages/box/rootfs/etc/s6-overlay/s6-rc.d/cgroups/type

@@ -0,0 +1 @@
+oneshot

packages/box/rootfs/etc/s6-overlay/s6-rc.d/cgroups/up

@@ -0,0 +1,2 @@
+/command/with-contenv
+/usr/local/bin/blitz-cgroup init

packages/box/rootfs/etc/s6-overlay/s6-rc.d/cloudflared/run

@@ -10,6 +10,6 @@ while [ ! -f "$tunnel_token_file" ] || [ ! -s "$tunnel_token_file" ] || \
   [ ! -f "$webapp_token_file" ] || [ ! -s "$webapp_token_file" ]; do
   sleep 5
 done
-exec /command/s6-setuidgid blitz /usr/bin/env \
+exec /usr/local/bin/blitz-cgroup enter system -- /command/s6-setuidgid blitz /usr/bin/env \
   HOME=/var/lib/blitz/home USER=blitz \
   /usr/local/bin/cloudflared --no-autoupdate tunnel run --token-file "$tunnel_token_file"

packages/box/rootfs/etc/s6-overlay/s6-rc.d/dockerd/run

@@ -4,8 +4,17 @@ if ! grep -Eq '^CapEff:[[:space:]]*[0-9A-Fa-f]*[2367ABEFabef][0-9A-Fa-f]{5}$' /p
   echo "dockerd: skipped (container is not privileged)"
   exec /command/s6-pause
 fi
-exec /usr/local/bin/dockerd \
+# The daemon and the containers it starts sit in DIFFERENT cgroups on purpose.
+# cgroup v2 forbids a cgroup from holding processes and controller-enabled
+# children at once, so the daemon cannot live in the same slice it parents. It
+# also means a container bomb takes docker.slice — every inner container — and
+# leaves the daemon alive to serve the next `docker run`.
+#
+# Without --cgroup-parent the daemon creates container cgroups at the root of
+# the mount, outside blitz-user.slice, and the user ceiling bounds nothing.
+exec /usr/local/bin/blitz-cgroup enter user/dockerd.scope -- /usr/local/bin/dockerd \
   --host=unix:///run/docker.sock \
   --data-root=/var/lib/blitz/docker \
   --pidfile=/run/blitz-docker/dockerd.pid \
+  --cgroup-parent=/blitz-user.slice/docker.slice \
   --group=docker

packages/box/rootfs/etc/s6-overlay/s6-rc.d/dufs/run

@@ -1,6 +1,6 @@
 #!/command/with-contenv bash
 set -euo pipefail
-exec /command/s6-setuidgid blitz /usr/bin/env \
+exec /usr/local/bin/blitz-cgroup enter system -- /command/s6-setuidgid blitz /usr/bin/env \
   HOME=/var/lib/blitz/home USER=blitz \
   /usr/local/bin/dufs --bind 127.0.0.1 --port 17445 --allow-all --allow-symlink \
   /srv/blitz-files

packages/box/rootfs/etc/s6-overlay/s6-rc.d/gateway/run

@@ -1,5 +1,5 @@
 #!/command/with-contenv bash
 set -euo pipefail
-exec /command/s6-setuidgid blitz /usr/bin/env \
+exec /usr/local/bin/blitz-cgroup enter system -- /command/s6-setuidgid blitz /usr/bin/env \
   HOME=/var/lib/blitz/home USER=blitz \
   /usr/local/bin/blitz-box-gateway

packages/box/rootfs/etc/s6-overlay/s6-rc.d/remote-control/run

@@ -89,7 +89,9 @@ cd /workspace || {
 }
 
 launch() {
-	exec /command/s6-setuidgid blitz /usr/bin/env \
+	# Remote Control drives a full agent, so it is user load and belongs under
+	# the user ceiling — not in the reservation that keeps the tunnel alive.
+	exec /usr/local/bin/blitz-cgroup enter user/rc.scope -- /command/s6-setuidgid blitz /usr/bin/env \
 		-u CLAUDE_CODE_OAUTH_TOKEN -u ANTHROPIC_API_KEY -u ANTHROPIC_BASE_URL \
 		HOME="$home" USER=blitz LANG=C.UTF-8 LC_ALL=C.UTF-8 \
 		/usr/bin/script -qec \

packages/box/rootfs/etc/s6-overlay/s6-rc.d/sshd/run

@@ -1,3 +1,3 @@
 #!/command/with-contenv bash
 set -euo pipefail
-exec /usr/sbin/sshd -D -e -f /etc/blitz/sshd_config
+exec /usr/local/bin/blitz-cgroup enter system -- /usr/sbin/sshd -D -e -f /etc/blitz/sshd_config

packages/box/rootfs/etc/s6-overlay/s6-rc.d/ttyd/run

@@ -1,6 +1,6 @@
 #!/command/with-contenv bash
 set -euo pipefail
-exec /command/s6-setuidgid blitz /usr/bin/env \
+exec /usr/local/bin/blitz-cgroup enter system -- /command/s6-setuidgid blitz /usr/bin/env \
   HOME=/var/lib/blitz/home USER=blitz LANG=C.UTF-8 LC_ALL=C.UTF-8 \
   /usr/local/bin/ttyd --interface 127.0.0.1 --port 7443 --writable --check-origin --url-arg \
   /usr/local/libexec/blitz-term

packages/box/rootfs/etc/s6-overlay/s6-rc.d/watch/run

@@ -20,6 +20,6 @@ while [ ! -s "$state_dir/broker.json" ]; do
 	sleep 5
 done
 
-exec /command/s6-setuidgid blitz /usr/bin/env \
+exec /usr/local/bin/blitz-cgroup enter system -- /command/s6-setuidgid blitz /usr/bin/env \
 	HOME="$state_dir/home" USER=blitz \
 	/usr/local/bin/blitz-cred watch