Skip to content

Releases

v1.25.0: as-of restores find same-day backups; nix run .#rekey

Choose a tag to compare

@thatbagu thatbagu released this 21 Nov 12:33
b7aba8c

Highlights

Restores "as of" a backup find it, even later the same day (#65)

restic's forget kept only the newest backup of each day. A --new-cluster rebuild backs everything up, then the new cluster takes its first backup of its fresh volumes the same day, and that backup's forget deleted the rebuild's. restore-all --as-of then silently restored an older snapshot. On a live cluster, Pi-hole and WireGuard came back as they were a day earlier. Per-app backups now also keep every backup of the last 7 days (retain.within). With one backup a day, that costs nothing extra.

nix run .#rekey: replace the age key safely (#62)

The documented manual steps could leave nodes unable to decrypt secrets.yaml. rekey adds the new key next to the old one everywhere (your workstation, every node, the cluster's copy) and re-encrypts secrets.yaml for it. After you push, rekey --finish removes the old key everywhere.

Docs (#64)

  • A troubleshooting page: symptom → which tool to run.
  • What nixlab's version numbers mean for you.

Upgrading

Nothing to change in vars.nix. If you set backups.apps.retain yourself, consider adding within = "7d";.

v1.24.2: backups keep working after a restore

Choose a tag to compare

@thatbagu thatbagu released this 20 Nov 21:48
bdb9ef9

Fix

Backups keep working after a restore. A restore could leave its restic lock in the repository. Every later backup then saved its snapshot but failed at forget ("repository is already locked"), and the volume's backup stayed Failed. nixlab-restore and nixlab-restore-all now make the next backup unlock the repository first, using VolSync's own mechanism (#61).

Found after an unattended --new-cluster rebuild of a live cluster, where Nextcloud's backup failed on every run.

Upgrading

Nothing to change in vars.nix. If a volume's backup fails with "repository is already locked" since an earlier restore, run sudo nixlab-backup-now <namespace>/<claim> once after setting its ReplicationSource's spec.restic.unlock to any new value.

v1.24.1: a fresh cluster's first Longhorn sync creates its namespace

Choose a tag to compare

@thatbagu thatbagu released this 20 Nov 21:11
875d4a1

Fix

A fresh cluster bootstraps by itself. The bootstrap started Longhorn's first sync without the app's sync options or retry, so the sync failed with namespaces "longhorn-system" not found. The failure was final, and every later sync wave waited on Longhorn. The sync now creates the namespace and keeps retrying (#60).

Found by the fully unattended nix run .#reinstall -- <master> --new-cluster on a live 3-node encrypted cluster. With this fix applied by hand, that run finished by itself: all 28 apps came back, every volume and Nextcloud's database were restored, and the data was identical before and after.

Upgrading

Nothing to change in vars.nix. This matters only when a cluster is created; running clusters aren't affected.

v1.24.0: the getting-started guide works as written; safe power cuts after upgrades; upgrade alerts

Choose a tag to compare

@thatbagu thatbagu released this 20 Nov 18:21
16d857f

Highlights

Found by a readiness pass on a live 3-node cluster: a newcomer's walk through the getting-started guide, a security scan, VPN access tests with real peer keys, and power cuts during upgrades.

The getting-started guide works as written (#52)

  • nix flake check no longer fails on the template's example. The example names its node master with hostname mymaster, and four modules confused the two (attribute 'mymaster' missing).
  • Tools take a node's hostname too. install, reinstall, unlock and wake accept mymaster as well as master, as the guide writes it.
  • nix run .#init adds what's missing. It runs before vars.nix is filled in, so a node set to encrypt = true afterwards had no disk passphrase, and install stopped. Run init again: it adds the missing secrets and leaves the rest alone.

Power cuts right after an upgrade (#58)

On real hardware, a power cut even 3 seconds after a finished upgrade booted the previous system: the new boot menu was still in the FAT boot partition's write cache. GRUB's install now flushes it, and the same cut boots the new system.

Alerts for nixlab's own failures (#56)

With monitoring on, NixlabUpgradeFailed fires when a node's nightly upgrade fails, and NixlabUpgradeNotRunning when one hasn't run for 3 days. Before, a node silently stayed on an old version.

Security (#51)

  • lan.restrictClusterPorts (opt-in): the ports the nodes use among themselves (API, kubelet, flannel's unauthenticated VXLAN, MetalLB, NodePorts) answer only lan.subnet. Recommended once lan.subnet is right.
  • The docs say plainly that sudo needs no password, so your SSH key is root on every node.

The scan found nothing else open: the kubelet and the API reject anonymous requests, and SSH is keys-only.

Smaller fixes

  • Nextcloud knows its own address for links made outside a request, like emails from background jobs and federated cloud IDs. Before, those read localhost (#53).
  • Leftover pods get cleaned up. After a node restart, the pods that ran on it stayed listed (Error, Completed) next to their replacements. Masters now delete those hourly; Job pods stay (#54).
  • CI retries image-download flakes and caches the VM tests' images (#57).

Also verified live: VPN groups are enforced as documented (admin gets everything; family gets only DNS and Nextcloud), VPN auto-login creates the Nextcloud account on first visit, certificates renew, and DDNS follows the public IP.

Upgrading

Nothing to change in vars.nix. Once your lan.subnet is right, consider lan.restrictClusterPorts = true;.

v1.23.1: reinstall encrypted nodes; fresh clusters bootstrap by themselves

Choose a tag to compare

@thatbagu thatbagu released this 19 Nov 23:26
b52a97f

Fixes

Found by running the fully automated nix run .#reinstall -- <master> --new-cluster on a live cluster whose nodes are all encrypted (#50).

  • Reinstalling an encrypted node works. Its /persist is inside LUKS, which GRUB can't read, so the installer didn't boot: the node fell back to NixOS, and nothing was erased. reinstall now detects this and boots the installer through iPXE: a tiny iPXE on the node's plain boot partition, booted once (falling back to NixOS), fetches it over HTTP from another node.
  • Longhorn's first sync on a fresh cluster no longer fails with "app project default not found". The bootstrap waits for Argo CD's default project first.
  • The gateway no longer stalls a fresh cluster. Traefik rejects a Gateway outright while its certificates don't exist yet, which v1.22 didn't account for. A Gateway rejected only for missing certificates now counts as healthy; any other rejection is still Degraded.

Result: an encrypted master rebuilt from scratch over the network, the workers joining, and every volume and Nextcloud's database restored, with data identical before and after.

Upgrading

Nothing to change in vars.nix.

v1.23.0: unlock encrypted nodes remotely

Choose a tag to compare

@thatbagu thatbagu released this 19 Nov 21:05
bfc361d

Highlights

nix run .#unlock -- <node>: unlock an encrypted node remotely

When a node's TPM doesn't unlock its disk (say, after a firmware update), the node waits in its boot stage. unlock reaches it over SSH, gives it the recovery passphrase from secrets.yaml, waits for it to boot, and enrolls its TPM again, so the next boot unlocks itself. It replaces the manual ssh -p 2222 and systemd-cryptenroll steps (#45).

The boot stage keeps the node's usual address

The boot stage's DHCP identified itself differently from the booted system's (DUID vs. MAC), so the router handed it another address each time. On a test node that was .130, then .132, never its usual .129, and remote unlock couldn't find it. It now asks by MAC and gets the node's usual address (#45).

Both were tested on real hardware (Beelink EQ12, Intel N100) with the TPM slot removed on purpose: the boot stage came up on the usual address, unlock unlocked the node and re-enrolled the TPM, and the next reboot unlocked unattended.

Monitoring: room for Prometheus

The light flavor's Prometheus used ~660Mi of its 768Mi limit on a 3-node cluster, and it grows while replaying its WAL after a restart. The limit is now 1536Mi (request 512Mi) (#48).

Docs: network boot on Intel N100 mini PCs

The firmware lists no PXE entry even with the network stack on, and pixiecore's chain-loaded iPXE hangs. iPXE with an embedded script (DHCP, then the installer over plain HTTP from another node) boots the installer fine (#49).

Upgrading

Nothing to change in vars.nix. An encrypted node gets the new boot stage with its next upgrade.

v1.22.1: the parent Application follows upgrades

Choose a tag to compare

@thatbagu thatbagu released this 19 Nov 19:38
5e0149c

Fix

The parent Application follows nixlab upgrades

The bootstrap applied the parent Application (apps) only once, on a fresh cluster, and nothing else manages it. So its settings never reached clusters bootstrapped earlier, e.g. v1.22.0's sync retries. The bootstrap service now re-applies it at boot. It's idempotent and touches only that object (#47).

Upgrading

Nothing to change. Deploying this version to your master applies it.

v1.22.0: rebuild from scratch, fresh clusters bootstrap by themselves

Choose a tag to compare

@thatbagu thatbagu released this 19 Nov 19:05
fd27e81

Highlights

Found by rebuilding a live cluster's only master from scratch (encrypted) and restoring it from backups. The data came back intact, but a fresh cluster needed manual help at seven points. It doesn't anymore (#46).

Rebuild the cluster from scratch: nix run .#reinstall -- <master> --new-cluster

Reinstalls the only master, which starts a new, empty cluster, and brings everything back:

  1. Backs up every volume (and a fresh dump of Nextcloud's database) right before.
  2. Reinstalls the master remotely, encrypted if encrypt = true.
  3. Lets the workers forget the old cluster's certificates and join the new one.
  4. Waits for Argo CD to bring the apps back from git.
  5. Restores every volume as it was before the reinstall.

Without per-app backups, it tells you the apps come back empty.

A fresh cluster bootstraps by itself

  • Longhorn is manual-sync, and its Application blocked every later sync wave. The bootstrap now syncs it once.
  • Longhorn's pre-upgrade hook waited forever for a service account only the sync creates. It's off, as Longhorn recommends for Argo CD.
  • The gateway stayed Degraded until cert-manager, in a later wave, issued its certificates: a deadlock. A Gateway is now healthy once its controller accepted it.
  • Every app keeps retrying a failed sync (backoff up to 5 minutes) instead of giving up after 5 attempts.
  • nixlab-secrets no longer puts Roles into namespaces that another app owns. Those namespaces don't exist yet on a fresh cluster, which was a regression from v1.19.0. There's a scoped ClusterRole instead, and the Job creates the namespace if needed. Kubernetes can't limit "create" to a name, so in that case the Job may create Secrets in any namespace; get, patch and update stay limited by name.

Restores that can't go wrong quietly

  • sudo nixlab-restore-all --as-of <time> restores every volume as it was at that time. A new cluster's first backups are of its empty volumes, so "latest" would restore nothing.
  • sudo nixlab-restore-nextcloud-db [<dump>] restores Nextcloud's database from a dump and sets up the database user again: a dump doesn't contain it, so Nextcloud couldn't log in. Then it tells sync clients the server went back in time. It replaces the manual kubectl run + pg_restore steps in the docs.
  • nixlab-restore no longer prints the --as-of time twice.

Upgrading

Nothing to change in vars.nix. Longhorn is manual-sync, so press Sync on the longhorn app once to drop its pre-upgrade hook.

v1.21.0: reinstall a running node remotely

Choose a tag to compare

@thatbagu thatbagu released this 18 Nov 20:32
f4b42e1

Highlights

nix run .#reinstall -- <node>

Reinstalls a running node from scratch, over SSH, with nobody at the machine. For example, to turn on encryption or to start clean. install can't do this itself: running nixlab nodes have kexec disabled on purpose, and kexec hangs some hardware anyway (Intel N100 mini PCs).

The node reboots into a NixOS installer put on its own disk, booted once through GRUB: a real reboot, which falls back to NixOS if the entry fails. The installer reboots back into NixOS by itself on a panic, on a hang (hardware watchdog), or if it isn't claimed within 15 minutes. Then install runs, the node rejoins the cluster, and Longhorn forgets its old disk so replicas rebuild onto the new one. It refuses the only master, since that would start a new cluster.

Ran on real hardware: an encrypted reinstall of a worker in a live 3-node cluster, with unattended TPM unlock (#43).

nixlab-longhorn-reset-node <node> (on masters)

After a reinstall, Longhorn still expects the node's old disk ("record diskUUID doesn't match the one on the disk") and schedules no replicas there. This forgets the stale disk record and its replicas, so Longhorn adds a fresh disk. It refuses if any volume would lose its last healthy replica. reinstall runs it for you.

Upgrading

Nothing to change in vars.nix. Deploy this version to your masters before using reinstall, since it needs the new helper there (it checks before erasing anything).

v1.20.0: recovering from dead nodes, alerts for per-app backups

Choose a tag to compare

@thatbagu thatbagu released this 18 Nov 18:12
125f80f

Highlights

Apps recover from a dead node by themselves

When a node hangs or loses power, rather than shutting down, its Longhorn volumes stay attached to it. On a live cluster, Pi-hole (the LAN's DNS) and Nextcloud's database couldn't start on another node until their pods were force-deleted by hand. Longhorn's pod deletion policy when a node is down is now delete-both-statefulset-and-deployment-pod: once Kubernetes marks the node down, Longhorn deletes those pods so they start elsewhere with their volume (#42).

Alerts for per-app backups

Nothing alerted when a per-app backup (VolSync) failed. With monitoring and backups.apps on, Prometheus now scrapes VolSync's metrics (both flavors), and AppBackupOutOfSync fires when a volume's last scheduled backup failed or didn't finish before the next one was due, for an hour (#42).

Docs

  • Monitoring flavors: the default light flavor has no Grafana; Grafana comes with monitoring.flavor = "full". The docs used to describe the full stack as the default. They now have a flavor table and light's service names.
  • Network boot (PXE): the official guides (NixOS manual and wiki, pixiecore, iPXE, nixos-images, nixos-anywhere), and what was found on Beelink EQ12 mini PCs.

Upgrading

Nothing to change in vars.nix. Longhorn is manual-sync, so press Sync on the longhorn app once to apply the new setting.