Skip to content

v1.20.0: recovering from dead nodes, alerts for per-app backups

Choose a tag to compare

@thatbagu thatbagu released this 18 Nov 18:12
125f80f

Highlights

Apps recover from a dead node by themselves

When a node hangs or loses power, rather than shutting down, its Longhorn volumes stay attached to it. On a live cluster, Pi-hole (the LAN's DNS) and Nextcloud's database couldn't start on another node until their pods were force-deleted by hand. Longhorn's pod deletion policy when a node is down is now delete-both-statefulset-and-deployment-pod: once Kubernetes marks the node down, Longhorn deletes those pods so they start elsewhere with their volume (#42).

Alerts for per-app backups

Nothing alerted when a per-app backup (VolSync) failed. With monitoring and backups.apps on, Prometheus now scrapes VolSync's metrics (both flavors), and AppBackupOutOfSync fires when a volume's last scheduled backup failed or didn't finish before the next one was due, for an hour (#42).

Docs

  • Monitoring flavors: the default light flavor has no Grafana; Grafana comes with monitoring.flavor = "full". The docs used to describe the full stack as the default. They now have a flavor table and light's service names.
  • Network boot (PXE): the official guides (NixOS manual and wiki, pixiecore, iPXE, nixos-images, nixos-anywhere), and what was found on Beelink EQ12 mini PCs.

Upgrading

Nothing to change in vars.nix. Longhorn is manual-sync, so press Sync on the longhorn app once to apply the new setting.