v0.2.12

v0.2.12 — what it reports is what it checked, and a roll leaves nothing behind

Two kinds of fix, both found the hard way. On 2026-10-26 the site read "10 of 10 invariants hold — no findings" while one host could not read its own configuration. And the 2026-10-23 roll needed a human at every node to do what the roll should have: re-admit disks, clear dead records, keep a second roll from starting.

Reporting what was checked (ADR-206)

• An unreadable config is a failure (bug-208). posture-check reads its configuration once. If it cannot, it records site config unreadable — every check that needs it was skipped: <error> and exits non-zero. It used to read the file five times and treat each failure as "nothing configured", which is how a host that could not parse its config published a clean, smaller count.
• Deploys refuse a mixed fleet (bug-209). deploy-updates, deploy-posture, deploy-observability, deploy-cplb and deploy-resolver --apply first ask every hypervisor which substrate it runs, and refuse unless all match the deploying release. A host that cannot be asked counts as a mismatch.
• A firewall port counts only when the allow-list is proven (bug-211). Held means the denied source was refused AND the allowed source connected. The store's ports are probed only on the host that runs the store. A host that cannot reach its allowed prober now reports the port as information instead of a pass.

A roll that does its own steps (ADR-205, ADR-207)

• One roll at a time (bug-205): a fleet-wide lock in the cluster (kube-system/substrate-roll-lock). A second roll refuses and names the holder. A killed roll leaves the lock, and the refusal says how to clear it after checking the volumes.
• Drain before destroy (bug-203): pods stop cleanly and move while the node still answers. A drain that does not finish in 300 s un-cordons the node and halts the roll with nothing destroyed.
• No dead records left behind (bug-201): the destroyed node's Longhorn replica records are deleted with its etcd member, Node object and volume attachments. Left in place, they fooled the last-replica guard and stopped a detached volume from attaching.
• The guard trusts evidence, not records (bug-201): a replica is a copy only if it never failed, finished building, and sits on a disk that is Ready under the same UUID on a node Longhorn sees Ready. The nightly storage line uses the same rule. The guard fails closed when it cannot read the answer.
• The disk re-admit waits for a verdict (bug-202): it polls until Longhorn calls the new disk stale (then re-admits it) or freshly Ready, ignores a Ready left over from the destroyed disk, and halts after 300 s of neither. The roll will not move on while the rolled node's disk is not admitted.
• --node says what it did (bug-207): " rebuilt on the pinned image — the other nodes were not touched".

Every node trusts every operator (ADR-207, bug-210)

• hypervisors.<name>.ssh_public_key declares each hypervisor's public key, and every node authorizes all of them. Before this, a node trusted only the key of the machine that built it, so after a roll from server1, server2 could reach no node.
• A site that declares no keys behaves as before. If keys are declared and the roller's own key is not among them, roll, provision and rebuild refuse before destroying anything.

One command installs the fleet (ADR-208)

• substrate install-fleet <version> installs a release on every hypervisor from one place. install.sh only ever installed where it ran, which is how server2 was left a release behind twice. The command:
  • is a dry run by default; --apply installs
  • runs each host's installs in one interactive session, so sudo asks once per host
  • installs every release in between, in order, so a host two behind gets the middle one first
  • refuses a downgrade, and refuses any path through a release that has no binary
  • installs nowhere if any host is refused
  • finishes by asking each host what it runs
• Each host still verifies the download itself, with its own cosign. install.sh is now carried in the signed binary, so the host without a checkout no longer fetches it from raw GitHub.

Operator notes — ORDER MATTERS

1. ./install.sh v0.2.12 on server1, then substrate install-fleet v0.2.12 --apply from server1, which brings server2. This bootstrap is needed only once, because v0.2.11 lacks the command. It ends by checking every host's version.
2. Only then add ssh_public_key for both hypervisors to site.yml. v0.2.11 and earlier cannot parse the field. The new version check refuses a deploy to an older host, but it should not be the only thing standing in the way.
3. substrate deploy-posture --apply.
4. The new keys reach the nodes at their next build, so the next roll is when server2 regains access. Until then server2 reports its 9100 firewall probe as information and publishes 18 held invariants, against server1's 19. That is the honest count.
5. The roll_runbook.md hand-steps for disk re-admit, dead replica records and "one node at a time" are now done by the roll. Keep the runbook's checks until one full roll has proven it.

214 tests.