ops hardening: swap must archive the outgoing binary; refuse non-release candidates; smoke needs a soak; finish fleet-status systemd port on erlich

[aarontrowbridge]

From the 2026-09-03 hub-wedge incident (see #775 + vault sessions/session-20260903-hub-wedge-forensics.md). Four gaps, each small:

1. hub-restart.sh swap must archive the outgoing binary

The 11:12 swap overwrote c99d6eab — the only binary that provably survived production (4 days) — with no copy kept. The Aug-30-era manual flow kept .bak/.new; the script path doesn't. The rollback path is only as good as the last archive.

• swap saves the outgoing binary to bin/archive/<sha>-<date> (or the opencode.prev convention from the upgrade receipts) before the rename.

2. swap should refuse non-release candidates

This morning's wedge-era binary was a local dev build (0.0.0--202609031509 — no version injection), staged and swapped through the otherwise-safe path. Release-only is policy (amicode-release.yml); enforce it at the seam.

• swap rejects candidates whose --version matches 0.0.0-* unless --force (and document the force escape hatch).

3. hub-upgrade-smoke.sh needs a post-listening soak

Both wedge classes from #775 pass today's smoke ("listening within 25 s"): the dev build served for ~12 min before wedging; the .18 release wedged ~90 s after listening. The harness now exists offline (incident dir); wire it in.

• After "listening", poll /session + CPU for a configurable soak (default ≥ 15 min), fail on timeout or RSS growth anomaly.

4. fleet-status systemd port on erlich is unfinished

No timer ran today (the 11:23 collection was manual); the guard still has launchctl remnants (reported "server not running" while it served 200); DEVICES order/aliases are from the mini-canonical era.

• systemd --user timer (5 min) + systemctl pid lookup + verify the guard's bad-state notification path works on erlich.

[aarontrowbridge]

Client-side hit of the same #270 class (2026-09-03, plane wifi): Go Standalone failed on a fleet client — the vendored amicode.18 binary (pre-#272-guards) crashed on the local DB with duplicate column name: directories because the local journal lacked 20260828201050_normal_stryfe. Heal applied manually (journal row inserted; backup kept). This strengthens the case for shipping a guarded release (the .18 binary is the vendored CLIENT binary too — every fleet client's local DB can be drifted): any pre-guard binary blocks Go Standalone, which is exactly the mode you want when the hub is unreachable.