feat(compute): give every workspace its own volume, and bake a golden image

[pythonlearner1025]

Two changes that share one goal: a workspace should survive its VM, and a new
workspace should not pay for work a disk image can already hold.

Validated end to end on real Hetzner resources. Create, destroy and recreate
all run against real infrastructure, and a file written to /workspace survives
the machine being destroyed. Evidence is in the comments below.

Headline numbers, all measured

	create → ready
Stock baseline (GAPS.md:35)	130.6 s
Golden image, first working version	64 s
Golden image, after the speed pass	45.3 s (recreate 45.9 s)

Supporting measurements taken live on cx23 @ hel1 on 2026-08-27:

Phase	Stock ubuntu-24.04	Golden snapshot
POST /servers returns	0.43 s	—
Hetzner reports running	14.6 s	24.1 s
sshd accepts	32.3 s	40.9 s
apt-get update	18.3 s	skipped
apt-get install docker.io	17.4 s	skipped

Two corrections to figures already in the repo:

• GAPS.md:77 records "VM-ready latency varies 46-71 s". That includes the apt
  work. It is not a boot floor, and reading it as one led two earlier analyses
  to conclude a warm pool was mandatory.
• GAPS.md:35 records docker install at 9 s. This run measured 35.7 s for apt
  update plus install together.
• A custom snapshot boots slower than a system image, by about 9 s. The image
  still wins by a wide margin, but that cost is real.

Phase 1 — the volume is the workspace's own disk

Every create on a volume-capable provider makes a 50 GB volume named after the
workspace and attaches it. /var/lib/blitz lives there, which is the docker
store and /workspace both, so a destroyed workspace can come back.

• The volume goes in Hetzner's server-create volumes field instead of an
  attach call afterwards. This fixes a real race: the bootstrap scans
  /dev/disk/by-id once with no retry, so the old ordering could leave a box
  with no persistent disk and no error to show for it.
• Workspace names are not unique and Hetzner volume names must be. The
  workspace name is tried first; a name carrying the workspace id is the
  fallback. Both paths fired during the live runs.
• Destroy detaches and keeps the volume for seven days. A janitor reclaims it
  after that. A volume created through POST /volumes carries no clock and is
  never reclaimed.
• GET /workspaces/history lists destroyed workspaces whose volume is alive.
  POST /workspaces/:id/recreate brings one back on that volume, and takes an
  optional sshPublicKey.
• A create that fails after the volume exists deletes it, and so does an
  ownership row that cannot be written.

Phase 2 — HETZNER_SERVER_IMAGES names a golden snapshot per location

• The bootstrap skips apt when docker and curl are already present, and skips
  the sshd relocation when the image already made it. The stock path is
  untouched, so a plain Ubuntu image still does both.
• A refused snapshot falls back to stock Ubuntu with a warning. Hetzner answers
  before allocating, so that retry cannot duplicate a server. An ambiguous
  failure is never retried.
• The image map reaches the deployment credential only. A snapshot lives in one
  project, so a BYOK organization boots stock Ubuntu.
• scripts/bake-golden-image.mjs builds the snapshot. It shares the emitted
  image-setup bash and its shell helpers with the bootstrap rather than copying
  them, so the two sides of that contract cannot drift.

Five defects the live runs exposed, all fixed here

None were visible to unit tests.

1. The bake hung for 30 minutes. The emitted image setup calls retry,
   which the bootstrap defines in its own preamble. The bake embedded the setup
   without it, so the builder died under set -e and never powered off.
2. Recreate failed with "volume already attached". A Hetzner detach is an
   async action; detachVolume fired it and returned, so destroy reported
   success while the volume was still attached.
3. A recreated workspace had no authorized SSH key. The row never stored
   one, so the restored box came up with no way in.
4. The golden image expired root's password. Undoing it once does not hold;
   cloud-init re-expires on every clone, so the bake disables that default.
   Host debuggability only — workspaces were never affected.
5. The bootstrap had no phase markers, flagged in GAPS.md:36 since
   2026-08-13. blitz_phase now logs seconds since start at four points.

What this does not do

It does not reach 30 s. About 27 s is Hetzner allocation and boot, which no
image can remove, and about 18 s is bootstrap tail. A warm pool of pre-booted
VMs is the only path below 30 s, and it was ruled out on cost: an idle slot
bills about $14.76 a month.

Two of the three speed levers paid nothing and are kept only on their merits:
disabling unused units moved boot to sshd by 0.4 s, inside noise, and shrinking
the snapshot reclaimed 11 MB, because the box image is the snapshot.

Not covered by the live runs: the seven-day janitor was not time-travelled on
real resources, and the BYOK path, arm64 and the hil location were not
exercised. The bake script does not verify its own snapshot; this one was
verified by booting a workspace on it.

Cost

Read from GET /v1/pricing on the deployment account (USD, VAT 0).

Item	Rate	Result
Volume	$0.0767 / GB / month	50 GB = $3.84 per workspace per month
Seven-day retention tail	—	$0.89 per destroyed workspace
Snapshot	$0.0199 / GB / month	$0.05 per month, fleet-wide

Gates

npm run typecheck, npm run lint:gate at the 102 anti-slop baseline with
max-lines back to 7 (hetzner.ts split into compute/hetzner-config.ts), 604
control-plane tests, and the python, bash and scripts conformance suites. Seven
new tests. Extracting boxImageSetupScript and BOX_IMAGE_SETUP_HELPERS out of
bootstrap.ts is a pure refactor: all 44 pinned bootstrap-byte tests pass
unchanged.

To turn Phase 2 on after merge

Snapshot 425047509 (hel1, 2.61 GB) is already baked and kept. Set
HETZNER_SERVER_IMAGES=hel1=425047509. Until that var is set, every create
takes the stock path and behaves exactly as it does today.

Note the live snapshot predates fix 4 above, so a rebake
(npm run golden:bake -- --location hel1) is worth doing before relying on host
SSH for debugging.

Review notes

• migrations/0036_workspace_volumes.sql adds auto_created, detached_at
  and workspace_id to volume_ownership.
• The bootstrap changes emitted bytes, which are a pinned contract.
  test/bootstrap-bash.test.mjs now slices at the docker-presence guard,
  because slicing at apt_watchdog update would cut the enclosing if in half.
• test/org-compute-credentials.test.ts needed its fake to hand out unique
  volume ids and to answer the volume lifecycle. Workspaces now carry a volume
  on every cloud create, so the destroy path detaches one every time.

🤖 Generated with Claude Code

[pythonlearner1025]

Do not merge yet — not validated against real Hetzner.

Every gate in this PR is unit-level. No part of the golden-snapshot path has run against real infrastructure:

• scripts/bake-golden-image.mjs has never executed. The account holds zero snapshots.
• The golden image boot and the bootstrap apt skip have never run on a real snapshot.
• The volumes field in POST /servers has never been exercised live.
• The 50 GB auto-create and its mount at /var/lib/blitz have never run live.
• The retention janitor, history list and recreate route have unit tests only.
• The 40-55 s create-to-ready figure is derived from a boot probe, not measured on this code.

The only live measurement behind this PR is a boot-floor probe on a stock ubuntu-24.04 cx23. It measured the baseline and exercised none of the new code.

Moving to draft until a real create, destroy and recreate cycle passes on real Hetzner resources.

[pythonlearner1025]

Validated against real Hetzner — full cycle passes

Ran the complete cycle on real resources on 2026-08-27, through the branch's control plane (run locally behind a cloudflared tunnel; a branch is never deployed to canary).

1. create           volume e2e-golden, 50 GB, hel1
                    server 163802707 image=425005770 type=snapshot volumes=[106723600]
                    OK: golden snapshot booted, volume attached during create
2. ready            69.9 s
3. marker           wrote /workspace/e2e-marker-1787851244212
4. golden image     apt skipped: true
                    /var/lib/blitz -> /dev/sdb  49G  464K  47G  1%
5. destroy          HTTP 200, volume 106723600 survived, server=null
6. history          GET /workspaces/history lists it
7. recreate         same volume, ready in 68.4 s
8. marker survived  nonce-cq7kss2g8v6 read back intact

create to ready: 69.9 s, down from the 130.6 s baseline. The golden snapshot is 2.62 GB and costs about $0.05 a month.

The run found three real defects, now fixed in b5d0533

1. The bake hung for 30 minutes. boxImageSetupScript emits a call to retry, which the bootstrap defines in its own preamble. The bake embedded the setup without it, so the builder died on retry: command not found under set -e and never powered off. Both callers now emit BOX_IMAGE_SETUP_HELPERS, and the builder traps ERR so a broken bake costs two minutes.
2. Recreate failed with "volume already attached". A Hetzner detach is an async action; detachVolume fired it and returned, so destroy reported success while the volume was still attached. It now polls until no server holds the volume.
3. A recreated workspace had no authorized SSH key. The row never stored one, so the restored box came up with no way in. POST /workspaces/:id/recreate now takes an optional sshPublicKey.

None of these were visible to unit tests. All three are now fixed and the cycle passes end to end.

Gates after the fixes

typecheck, lint ratchet at the 102 baseline, 604 control-plane tests, python 1/1, bash 4/4, scripts 59/59.

State left behind

Snapshot 425005770 (hel1) is kept so HETZNER_SERVER_IMAGES=hel1=425005770 can be set when this lands. Every test server and volume is deleted.

[pythonlearner1025]

Speed pass: 130.6 s → 45.3 s, measured

Re-baked and re-ran the full cycle on real Hetzner (snapshot 425047509).

	create → ready
Stock baseline (GAPS.md:35)	130.6 s
Golden image, before this pass	64 s (57.7 / 61.8 / 69.9 / 68.4)
Golden image, after this pass	45.3 s (recreate 45.9 s)

Three levers went in together. Only one paid, and I am reporting the other two honestly rather than claiming the whole 19 s.

1. Bake the sshd move — the win, ~19 s. The bootstrap relocated host sshd to 2222 and waited for port 22 to free on every boot. The image now holds that state, and the bootstrap skips the block when a listener is on 2222 and 22 is free. Estimated 5-10 s; delivered about 19 s.
2. Disable unused units — no measurable gain. Boot to sshd went 40.9 s → 41.3 s, inside run-to-run noise. Kept because a workspace never uses snapd or the apt timers, not because it is faster.
3. Shrink the snapshot — no gain. 2.6157 GB → 2.6050 GB. The box image is the snapshot; there was nothing to reclaim. Estimated 2-5 s; delivered none.

Two supporting fixes

• Phase markers. blitz_phase now logs seconds since start at the apt, volume, sshd and image points. GAPS.md:36 has flagged their absence since 2026-08-13; without them the tail could only be attributed by subtraction.
• Root password expiry in the image. Hetzner expires root's password on an image built without an SSH key, and undoing it once does not hold — cloud-init re-expires on every clone, so the bake disables that default. Host debuggability only; workspaces were never affected.

Measured, and a correction to an earlier claim

A golden snapshot boots slower than a stock system image: 24.1 s vs 14.6 s to running, 40.9 s vs 32.3 s to sshd. Hetzner materializes a custom snapshot more slowly. The image still wins by far, but that ~9 s cost was not in my earlier estimate.

Where the remaining 45 s goes

About 27 s of Hetzner allocation and boot, which no image can remove, and about 18 s of bootstrap tail. Under 30 s still needs a warm pool, which was ruled out on cost.

State

Snapshot 425047509 is live and kept: set HETZNER_SERVER_IMAGES=hel1=425047509. The previous snapshot is deleted. Every test server and volume is gone; the 8 pre-existing workspace servers and their 2 volumes are untouched.

[pythonlearner1025]

The snapshot id no longer needs setting by hand

2d38929 pins it in canary.yml, beside BLITZ_DEPLOY_VAR_CLOUD_WORKSPACE_CREDENTIAL_POLICY:

BLITZ_DEPLOY_VAR_HETZNER_SERVER_IMAGES: hel1=425047509

It was never a secret — it is a plain [vars] entry, and this is where the other per-deployment non-secret settings already live. Merging now deploys it. A rebake changes one line, in version control.

It deliberately is not a default in code. DEFAULT_HETZNER_MACHINE_TYPES holds names Hetzner offers everyone; a snapshot id exists in exactly one project. A hardcoded default would make every self-hoster and every BYOK org attempt an image they cannot see — the adapter falls back correctly, but logs a warning on every create forever.

release.yml gets no such line yet: client prod needs a bake in its own Hetzner project first.

And the reason it is still inert

Both hosted deployments run byok-required, and the golden image reaches the deployment credential only, so today it reaches no customer.

The intended behaviour — a subscribed org runs on platform infrastructure — is not implemented. Verified three ways: the credential policy is one deployment-wide env var read in two places with no per-org lookup (org-credentials.ts:293,402,426); org_entitlements holds only seat_limit, so the billing service can write seats and orgs.vm_limit and nothing else (migrations/0031); and nothing under core/compute/ references a plan, tier or subscription.

plans/SUBSCRIPTION-COMPUTE.md closes that with one migration, one integer in the entitlements seam, and one branch in resolve() — keeping the rule that core never learns a plan name. It also fixes what a lapse must not do: compute_credential_source is pinned per row, so a downgrade refuses the next create and never touches a running workspace.