Dead-host detection + machine drain/failover

[blakebauman]

Context

Last core reliability gap from the audit. The control plane never noticed a host going away — a crashed or network-partitioned host kept its machines assigned indefinitely, so apps didn't recover and the scheduler could still place new machines onto a dead host. (Now much easier to operate thanks to the structured logs from #5.)

Branched from a clean main (no stacking).

What changed

• Detection — a RunHostMonitor loop marks a host down after it misses heartbeats for FOLD_HOST_TIMEOUT (default 30s); the next heartbeat brings it back up. The scheduler already skips non-up hosts, so placement avoids dead hosts automatically.
• Drain / failover — when a host goes down, its running machines are rescheduled onto healthy hosts: volume-backed machines stay in their AZ (the volume is detached from the dead host first so it can re-attach on the new one), and the stale host-local RAM snapshot is cleared so they cold-start where they land. Machines that aren't wanted-running are simply unplaced.
• Scale-to-zero recovery — wake() now re-places unplaced machines, so a scaled-to-zero app recovers on the next request even if its old host died.
• Recovery — store.Heartbeat clears the down mark; control logs the down/up transitions.
• CLI — fold hosts list shows host state (up|down).

Reschedule is serialized per app (lockApp) against concurrent deploys/scale.

Caveat (documented)

A host that's alive but partitioned from control will have its machines rescheduled elsewhere. The cloud's single-attach guarantee prevents a true split for volume-backed apps, but a hard partition can briefly double-run stateless copies. Tunable via FOLD_HOST_TIMEOUT.

Tests

• drainHost reschedules a started machine onto the live host and leaves it desired-started
• drainHost unplaces a suspended machine and clears its stale snapshot
• sweepHosts marks a stale host down, and a heartbeat recovers it to up

Verification

• gofmt -l clean · go vet ./... clean · go test ./... all pass · go build -tags firecracker ./... compiles
• make e2e-local passes (no regression in deploy/scale-to-zero/wake — wake path was touched)
• Manual: started a local control + agent, killed the agent, and watched fold hosts list flip the host to down (with a WARN host missed heartbeats log), then back to up after restarting the agent

🤖 Generated with Claude Code