v0.2.14

Fixed

• The App Health "Errors and warnings" panel now respects the namespace dropdown (bug-223). Its LogsQL query was {kubernetes.pod_namespace=~"$namespace"} error OR ERROR OR warn OR WARN OR fatal OR panic. The stream filter bound only to error; every other term matched all logs, from every namespace and the hosts. With "wanderer" selected it returned 45,161 lines in an hour, when the namespace had 568 in total. The OR group is now parenthesised. Verified against VictoriaLogs: that same selection now returns 0, and the per-namespace counts are what each namespace actually logs.

Install

1. From server1: substrate install-fleet v0.2.14 to preview, then --apply.
2. substrate deploy-observability --apply, which re-provisions the dashboard.

No pin changes. 216 tests.

Known

• vmagent logs about 96 "skipping duplicate scrape target" errors a minute. Pods that declare several ports yield one identical target per port, and vmagent scrapes one. No metric is wrong, but the volume dominates observability in this panel. Not changed in this release: the obvious relabel would silently drop a k0s system pod that serves metrics on an undeclared port.

v0.2.13

Fixed

• The roll no longer trusts a Longhorn write it did not check (bug-215). Every controller leave or join makes k0s restart every konnectivity agent, and until they reconnect a call through a Longhorn webhook fails with No agent available. On the 2026-10-27 roll that left a destroyed node's replica records in place while the roll printed that it had removed them, and a volume then could not attach. Now:
  • the roll waits for rollout status ds/konnectivity-agent after the leave and again after the join, before any Longhorn write;
  • Longhorn writes (the replica delete and the disk re-admit patches) retry No agent available for up to a minute;
  • the replica delete's result is checked, and the records are re-read afterwards. Any record on the dead node whose deletion was not accepted halts the roll before the node is re-created, with the record names.
• The flux-operator bump bot no longer rewrites cosign's checksum. Its sed replaced every two-space sha256: in versions.yml, and cosign is the second one. PR #5 (v0.62.1) carried that error and was closed unmerged. The edit is now scoped to the flux_operator block, and the job fails unless exactly those two lines changed.

Pins

Component	From	To	Why
Grafana	v13.2.4	v13.3.1	security: six CVEs fixed in 13.3.1/13.2.6, plus those in 13.3.0
VictoriaMetrics	v1.154.1	v1.156.0	Go 1.27.3 and base-image security updates, bug fixes
VictoriaLogs datasource	v0.33.1	v0.34.0	alerting fix for empty Instant stats queries; needs Grafana ≥ 10.4
flux-operator	v0.61.0	v0.62.1	maintenance; applies at the next node build
rustix (build dependency)	1.1.4	1.1.6	RUSTSEC-2026-0486 (unsound), which fails the release audit

Each sha256 was checked against upstream's published checksum: sha256 for VictoriaMetrics and Grafana; SHA-1 for the datasource, whose sha256 was then computed from the same artefact. The flux-operator sha256 was computed from the downloaded install.yaml.

Install

1. From server1: substrate install-fleet v0.2.13 to preview, then --apply. It installs on every hypervisor and ends by checking each host's version.
2. substrate deploy-observability --apply (Grafana, VictoriaMetrics, the datasource). Then confirm Grafana's VictoriaLogs panels still load.
3. flux-operator changes nothing until the next node build. The roll changes are proven by the next roll.

216 tests.

Known

• The konnectivity wait narrows the window rather than closing it: k0s may rewrite the DaemonSet a moment after the wait returns. The retry and the re-read are what make the roll safe. The proof is the next roll passing without roll_runbook.md §2a done by hand.

v0.2.12

v0.2.12 — what it reports is what it checked, and a roll leaves nothing behind

Two kinds of fix, both found the hard way. On 2026-10-26 the site read "10 of 10 invariants hold — no findings" while one host could not read its own configuration. And the 2026-10-23 roll needed a human at every node to do what the roll should have: re-admit disks, clear dead records, keep a second roll from starting.

Reporting what was checked (ADR-206)

• An unreadable config is a failure (bug-208). posture-check reads its configuration once. If it cannot, it records site config unreadable — every check that needs it was skipped: <error> and exits non-zero. It used to read the file five times and treat each failure as "nothing configured", which is how a host that could not parse its config published a clean, smaller count.
• Deploys refuse a mixed fleet (bug-209). deploy-updates, deploy-posture, deploy-observability, deploy-cplb and deploy-resolver --apply first ask every hypervisor which substrate it runs, and refuse unless all match the deploying release. A host that cannot be asked counts as a mismatch.
• A firewall port counts only when the allow-list is proven (bug-211). Held means the denied source was refused AND the allowed source connected. The store's ports are probed only on the host that runs the store. A host that cannot reach its allowed prober now reports the port as information instead of a pass.

A roll that does its own steps (ADR-205, ADR-207)

• One roll at a time (bug-205): a fleet-wide lock in the cluster (kube-system/substrate-roll-lock). A second roll refuses and names the holder. A killed roll leaves the lock, and the refusal says how to clear it after checking the volumes.
• Drain before destroy (bug-203): pods stop cleanly and move while the node still answers. A drain that does not finish in 300 s un-cordons the node and halts the roll with nothing destroyed.
• No dead records left behind (bug-201): the destroyed node's Longhorn replica records are deleted with its etcd member, Node object and volume attachments. Left in place, they fooled the last-replica guard and stopped a detached volume from attaching.
• The guard trusts evidence, not records (bug-201): a replica is a copy only if it never failed, finished building, and sits on a disk that is Ready under the same UUID on a node Longhorn sees Ready. The nightly storage line uses the same rule. The guard fails closed when it cannot read the answer.
• The disk re-admit waits for a verdict (bug-202): it polls until Longhorn calls the new disk stale (then re-admits it) or freshly Ready, ignores a Ready left over from the destroyed disk, and halts after 300 s of neither. The roll will not move on while the rolled node's disk is not admitted.
• --node says what it did (bug-207): " rebuilt on the pinned image — the other nodes were not touched".

Every node trusts every operator (ADR-207, bug-210)

• hypervisors.<name>.ssh_public_key declares each hypervisor's public key, and every node authorizes all of them. Before this, a node trusted only the key of the machine that built it, so after a roll from server1, server2 could reach no node.
• A site that declares no keys behaves as before. If keys are declared and the roller's own key is not among them, roll, provision and rebuild refuse before destroying anything.

One command installs the fleet (ADR-208)

• substrate install-fleet <version> installs a release on every hypervisor from one place. install.sh only ever installed where it ran, which is how server2 was left a release behind twice. The command:
  • is a dry run by default; --apply installs
  • runs each host's installs in one interactive session, so sudo asks once per host
  • installs every release in between, in order, so a host two behind gets the middle one first
  • refuses a downgrade, and refuses any path through a release that has no binary
  • installs nowhere if any host is refused
  • finishes by asking each host what it runs
• Each host still verifies the download itself, with its own cosign. install.sh is now carried in the signed binary, so the host without a checkout no longer fetches it from raw GitHub.

Operator notes — ORDER MATTERS

1. ./install.sh v0.2.12 on server1, then substrate install-fleet v0.2.12 --apply from server1, which brings server2. This bootstrap is needed only once, because v0.2.11 lacks the command. It ends by checking every host's version.
2. Only then add ssh_public_key for both hypervisors to site.yml. v0.2.11 and earlier cannot parse the field. The new version check refuses a deploy to an older host, but it should not be the only thing standing in the way.
3. substrate deploy-posture --apply.
4. The new keys reach the nodes at their next build, so the next roll is when server2 regains access. Until then server2 reports its 9100 firewall probe as information and publishes 18 held invariants, against server1's 19. That is the honest count.
5. The roll_runbook.md hand-steps for disk re-admit, dead replica records and "one node at a time" are now done by the roll. Keep the runbook's checks until one full roll has proven it.

214 tests.

v0.2.11

v0.2.11 — a volume survives losing a machine, and the platform checks it

On 2026-10-23 every replica of every Longhorn volume was on one hypervisor. The settings promised replicas would reach the other one; nothing checked, and nothing made it happen. Losing that machine would have meant a restore, not a survival.

Three gaps lined up. The replica anti-affinity was per node, and three nodes on one machine satisfy it. No node carried a zone: the hypervisor was declared under invariant-platform.io/hypervisor, which Longhorn ignores, and the standard key had been left out on the belief that a kubelet may not set *.kubernetes.io/ labels on itself. That is false for this key — topology.kubernetes.io/zone is on the kubelet allow list. And with every node in one zone, Longhorn placed replicas by free disk.

Fixed

• bug-200 — every node now registers with topology.kubernetes.io/zone=<hypervisor>, rendered in the same --labels flag as invariant-platform.io/hypervisor and with the same value, so the two keys cannot describe different domains. Kubelet applies it only when it creates the Node, so nodes built before this release need a one-time label (below).

Added

• A storage line in posture-check. Every volume must hold a usable replica on every hypervisor, judged on where the replicas are rather than on how the cluster is configured — a correct label with replicas still piled up fails. A stopped replica of a detached volume counts; a failed one, or one on a node that no longer exists, does not. One finding per volume short of a machine, naming the volume and how many machines it reaches. Fewer than two hypervisors, or no volumes, is information and uncounted (ADR-199). 18 → 19.
• substrate gate checks the zone label, separately from the hypervisor label, and prints the one-time remedy. It is not one of the roll's between-node checks, so it cannot halt a roll part-way.

Changed — YAML is now parsed in safe Rust

• serde_yaml_ng → serde-saphyr (ADR-204). The first build of this release stopped at the RustSec gate: unsafe-libyaml, the parser underneath serde_yaml_ng, is unmaintained (RUSTSEC-2024-0449, assigned 2026-10-12) — a machine translation of libyaml's C into unsafe Rust that will not be fixed. Rather than ignore the advisory, the parser is gone: serde-saphyr is written in safe Rust, and every YAML call goes through one module (substrate_core::yaml).
• Booleans are strict: an unquoted yes, no, on or off stays a string, as it did before.
• On the real configuration the old and new parsers produce identical plans for every deploy. The two YAML files substrate writes — Grafana's alert provisioning and the merged kubeconfig — carry the same data with their keys in sorted order, so the next deploy-observability restarts Grafana once.
• The advisory gate now runs on every push and daily, not only after a release exists, so the next new advisory is a red build on main rather than a release with no binary.

Security — RUSTSEC-2026-0387

• p256 0.13 → 0.14 (with x509-cert 0.3, sha2 0.11). crypto-bigint 0.5, underneath p256 0.13, could run secret-dependent branches in operations meant to be constant-time — a timing side channel on the key that signs the just-in-time admin certificate requests. Published 2026-10-23, after this release's first build; fixed in crypto-bigint 0.7.6+. The generated key and certificate request were verified independently with OpenSSL (signature, subject, P-256, key matches request).

This release was first published without assets and has been rebuilt in place under the same number: nothing was ever installed from it.

Operator notes

• The matching cluster-side change is in substrate_config: replica-zone-soft-anti-affinity: "true" (deliberately soft — with three replicas and two zones a hard rule can never place the third) and replica-auto-balance: "best-effort", which moves replicas that are already lopsided.
• Nodes built before v0.2.11 need the zone label once, through break-glass: kubectl label node <node> topology.kubernetes.io/zone=<hypervisor>. The next roll makes it permanent.
• install.sh v0.2.11 on each host, then substrate deploy-posture --apply from the host that reaches the others.

187 tests.

v0.2.10

v0.2.10 — the platform checks that its own backups ran

Fifteen invariants were re-asserted every night and not one of them was "the backups ran". The whole claim is that a node is rebuilt rather than repaired, and a rebuild restores from a backup, so the backup was the one load-bearing thing nothing asserted. The restore drills prove the mechanism worked once; they say nothing about whether last night produced a file.

Added

• posture.backups — a declared list of {namespace, cronjob, max_age_hours}. Each one's last SUCCESS must be recent, and each gets its own line. 15 → 18 invariants.
  • Reads lastSuccessfulTime, never lastScheduleTime: a CronJob that fires nightly and fails nightly keeps a fresh schedule time forever, which is exactly the silent failure worth catching.
  • DECLARED rather than discovered. A check that went looking for CronJobs that look like backups would fall silent the moment one was renamed or deleted — the two failures most worth catching. A declared name with no matching CronJob is a FAILURE, so deleting a backup is not a way to stop being asked about it.
  • A site that declares none prints backups: none declared, not asserted as info, never as a note (ADR-199): a line that cannot fail must not raise the number that holds.

Fixed

• bug-198 — deploy-observability installed a new binary and left the old process running. The installer already restarted a unit whose FILE changed and one whose startup-only CONFIG changed; a replaced EXECUTABLE was covered by neither, because the unit file and every config file stayed byte-identical and systemctl enable --now does nothing to a running unit. So the deploy printed "fetching v1.153.0", "active" and "host matches what was deployed" while two stale processes served — from two SECURITY releases, which left a patched binary on disk and an unpatched process answering requests.

  The install shapes now record what they replaced, and the restart pass asks what the unit loop already asked one layer up: is this process older than the artefact it is executing? That phrasing handles a fresh install for free, since a unit just started is newer than the binary it started from.

• The verification line reports the version the PROCESS returns, not systemctl is-active. The old line was true and said nothing about which binary, which is how this passed its own check.

Operator notes

• install.sh v0.2.10 on each host, then from the host that reaches the others: substrate deploy-posture --apply (ships the site.yml carrying the new backups: block) and substrate deploy-observability --apply.
• That deploy-observability --apply is the one that finally moves VictoriaMetrics to v1.153.0 and VictoriaLogs to v1.53.0 in the running processes. Confirm with the new line: victoria-metrics: active running v1.153.0.
• The invariant count on the site follows the hosts, not this release. It reads 15 until a host on v0.2.10 with the new site.yml records a run (ADR-173).

175 tests.

v0.2.9

v0.2.9 — a roll stops breaking the peers' probes

A roll replaces nodes, and replaced nodes present new SSH host keys. Only the hypervisor that ran the roll learned them; every other hypervisor kept the old ones, so its peer probe failed on a key mismatch and degraded to [ -- ] without ever saying why. The probe looked like it was running. It was refusing to connect.

Fixed

• bug-177 — the host keys are scanned once, from one vantage, and the same lines are then installed on every hypervisor, so the fleet trusts one set of keys rather than each host accepting whatever it is shown first. scan_host_keys, install_host_keys and refresh_client_access are called from the roll and from both rebuild paths. The addresses are passed as positional parameters, never interpolated into the script — a test passes 10.99.0.21; rm -rf / as a node address and asserts no address appears in the script text.
• bug-179 — deploy-updates --repo . no longer needs the fleet to preview. The dry run plans a placeholder where the kubeconfig goes and says so; the fetch happens only under --apply, and an apply whose kubeconfig was never fetched refuses rather than writing the placeholder to a host.

Added

• deploy-updates joins the other two in the CI preview step, in both workflows. All three deploys now plan against the fixture site after every build, which is what bug-179 was blocking.

Operator notes

• install.sh v0.2.9 on each host. No --apply is required: every shipped file is byte-identical to v0.2.8, and the two fixes are in code paths the roll and the rebuild take.
• The bug-177 fix lands on the next roll. Until then, a host whose peer probe shows [ -- ] can be cleared by hand: ssh-keygen -R <node-ip> && ssh -o StrictHostKeyChecking=accept-new <admin>@<node-ip> true.

163 tests.

v0.2.8

v0.2.8 — deploy-posture plans again

v0.2.7 could not run deploy-posture: it asked the release payload for versions.yml, which is configuration and lives in --repo, and refused itself with versions.yml is not part of this release's payload. The other two deploys were correct. Nothing reached a host from v0.2.7's posture deploy, so nothing on a host is wrong — but the version could not do its job, and under ADR-200 the only fix is the next release.

Fixed

• bug-178 — deploy_posture::plan reads versions.yml from --repo, as site.yml already was. A test now plans against a repo whose versions.yml was edited and asserts the edited bytes are what would reach the host — the plan takes exactly two files from --repo and everything else from the payload.

Added

• Both workflows run deploy-posture and deploy-observability previews against tests/fixtures after the build. The preview that would have caught v0.2.7 was never run by hand; it is now run by the machine, every time. (deploy-updates is not yet in that step: its preview still fetches a kubeconfig over ssh — bug-179, open.)
• tests/fixtures/site.yml carries the posture status ids the deploy needs.

Operator notes

• install.sh v0.2.8 on each host, then from the host that reaches the others: substrate deploy-updates --apply (first PEER_HOSTS= apply) and substrate deploy-posture --apply.

162 tests.

v0.2.7

v0.2.7 — the release carries its payload

ADR-200. Until now the signed binary was a release but the files it put on a host — the nightly-maintenance script, the posture and observability units, the Grafana templates and dashboards — were read at run time from whatever --repo pointed at: any commit, any branch, a hand-edited tree. deploy-updates --apply from a checkout could ship a script no release ever contained. It cannot now. Every file a deploy-* command installs is embedded in the binary at build time, so the only way a script reaches a hypervisor is inside a signed release.

Changed

• payload.rs — all 29 shipped files (hypervisor-update.sh, hypervisor-uncordon.sh, notify.sh, grafana-health.sh, substrate-reconcile.sh, four update units, four posture units, nine observability units, the Grafana and Traefik templates, provisioning, three dashboards) are include_bytes! in an explicit list. No build script walks a directory: a test walks systemd/ and observability-host/ and fails when a file is on disk but not listed, or listed but gone. The three auto-roll units are recorded as deliberately unshipped (ADR-186) with the reason beside them.
• deploy-updates, deploy-posture and deploy-observability read only the payload. --repo supplies site.yml and versions.yml and nothing else — /etc/substrate on a host is a complete repo. Dashboards are enumerated from the payload, not from a directory at deploy time.
• Every deploy opens with payload: substrate 0.2.7 (v0.2.7) — every file below is this release's, not the checkout's.
• release.yaml lints every shipped script with the same shellcheck the test workflow runs. v0.2.6's commit was red on test and green on release (bug-175); one commit now gets one verdict.

Fixed

• bug-175 — SC2086 in hypervisor-update.sh (peer count). Rides in this release, which is now the only way a script fix ships.
• bug-176 — a deploy could install a script from the working tree. Closed by construction: the working tree is no longer an input.

Operator notes

• install.sh v0.2.7 on each host. Then, once, from the host that reaches the others: substrate deploy-updates --apply — the first apply that writes PEER_HOSTS= (v0.2.6 introduced the list; this is the release that installs the script honouring it) — and substrate deploy-posture --apply. Both fan out to every hypervisor in site.yml.
• A change to any shipped script or unit is a release from here on. There is no other path.

161 tests.

v0.2.6

v0.2.6 — the fleet is not two hosts

Phase A of ADR-199: substrate stops assuming a second hypervisor exists, and stops assuming only one does. Nothing on a two-host estate changes — the posture line, the count and the unit files render as before — but a one-host site and a three-host site now get the same tooling with the truth told about each.

Fixed

• bug-173 — peers are a list. deploy-updates paired each hypervisor with the first other host by name, so at three hosts two of them gated their nightly reboot on the same third host and could reboot together. The reboot interlock now checks every other host's lock, VMs and reachability (PEER_HOSTS); posture-check probes every peer's SELinux and units and asks the firewall matrix from every vantage point; a one-host site has an empty list, says so, and still checks the cluster.
• bug-174 — nothing unasserted is counted. "unreachable, not checked", "not configured, check skipped" and "site config incomplete" were filed as invariants that held. They are now [ -- ] information: printed, never counted, never published. The count on the strip is the number of things actually asserted. check_credentials is not asked at all where no secrets manager is configured.
• validate no longer demands a peer_target on a site with one hypervisor.

Added

• tests/fixtures/site-one-host.yml and site-three-hosts.yml — every fleet-shape assumption is a test against one of them. A node's cloud-config is proven byte-identical across one, two and three hosts.
• updates::peers_of, peer_hosts_value; Report.info; posture::note_no_peers.

Operator notes

• install.sh v0.2.6 on each host, then substrate deploy-updates --apply (rewrites /etc/homelab/update.env with PEER_HOSTS=; the script honours an old PEER_HOST= until then) and substrate deploy-posture --apply.
• Expect the same fifteen lines tomorrow morning.

157 tests.

v0.2.5

v0.2.5 — the roll owns its steps

The first roll on v0.2.4 (2026-09-16, Kairos 4.2.0 → 4.3.0, k0s 1.36.3 → 1.36.4) replaced all five nodes and needed a human four times. This release makes each of those a step in substrate roll, so the operator's terminal stays empty from --yes to roll complete.

Fixed

• bug-154 — the re-created node's Longhorn disk is re-admitted automatically. Longhorn keeps its node record across a same-name replacement, so every fresh disk came back as DiskNotReady: record diskUUID doesn't match the one on the disk and the three-patch procedure (evict → remove → re-add under a new key) was run by hand on every node of both rolls. roll now does it after the join, keeping the disk's own storageReserved (bug-159 cannot recur), retrying while Longhorn's webhook is still syncing, and halting with the condition message if the disk is not Ready and Schedulable within two minutes. forget_node no longer spends two minutes per node waiting for Longhorn to forget a node it never forgets.
• bug-170 — the dead node's volumes are released at once. A destroyed node's VolumeAttachments lingered until the attach/detach controller's six-minute unmount wait expired, so every single-writer tenant (PocketBase, Meilisearch) served 500s for ~3.5 minutes per node. roll deletes them right after the Node object; the rescheduled pod mounts in seconds.
• bug-171 — the gate's local kubectl always uses the break-glass context. It inherited the operator's current context; with brad (read-only, ADR-071) selected, /healthz/etcd was forbidden on every node, the etcd wait reported "not serving" across a healthy cluster, and the roll was three minutes from halting after one node. Regression-tested.
• bug-158 — refuse to destroy a volume's last healthy replica. Before destroying a node, roll reads every Longhorn replica and refuses — naming the volumes — if any volume would be left with no never-failed replica elsewhere. Detached volumes are held to the same standard as attached ones; exempting them is how garmin-sync-state eroded to zero replicas across four replacements on v0.2.3.

Operator notes

• Nothing changes on the hypervisors beyond the binary: install.sh v0.2.5 on each, then substrate deploy-posture --apply to re-stage /etc/substrate from the checkout (first live run of the installer's 0600 enforcement on a kept publish-status.env).
• The acceptance test is the next bump's roll: no WARNING, no disk dance, no context switch, no 500 on trails.

141 tests. Verified: cosign identity bradpenney/substrate/.github/workflows/release.yaml@refs/tags/v0.2.5.