Skip to content

v1.22.0: rebuild from scratch, fresh clusters bootstrap by themselves

Choose a tag to compare

@thatbagu thatbagu released this 19 Nov 19:05
fd27e81

Highlights

Found by rebuilding a live cluster's only master from scratch (encrypted) and restoring it from backups. The data came back intact, but a fresh cluster needed manual help at seven points. It doesn't anymore (#46).

Rebuild the cluster from scratch: nix run .#reinstall -- <master> --new-cluster

Reinstalls the only master, which starts a new, empty cluster, and brings everything back:

  1. Backs up every volume (and a fresh dump of Nextcloud's database) right before.
  2. Reinstalls the master remotely, encrypted if encrypt = true.
  3. Lets the workers forget the old cluster's certificates and join the new one.
  4. Waits for Argo CD to bring the apps back from git.
  5. Restores every volume as it was before the reinstall.

Without per-app backups, it tells you the apps come back empty.

A fresh cluster bootstraps by itself

  • Longhorn is manual-sync, and its Application blocked every later sync wave. The bootstrap now syncs it once.
  • Longhorn's pre-upgrade hook waited forever for a service account only the sync creates. It's off, as Longhorn recommends for Argo CD.
  • The gateway stayed Degraded until cert-manager, in a later wave, issued its certificates: a deadlock. A Gateway is now healthy once its controller accepted it.
  • Every app keeps retrying a failed sync (backoff up to 5 minutes) instead of giving up after 5 attempts.
  • nixlab-secrets no longer puts Roles into namespaces that another app owns. Those namespaces don't exist yet on a fresh cluster, which was a regression from v1.19.0. There's a scoped ClusterRole instead, and the Job creates the namespace if needed. Kubernetes can't limit "create" to a name, so in that case the Job may create Secrets in any namespace; get, patch and update stay limited by name.

Restores that can't go wrong quietly

  • sudo nixlab-restore-all --as-of <time> restores every volume as it was at that time. A new cluster's first backups are of its empty volumes, so "latest" would restore nothing.
  • sudo nixlab-restore-nextcloud-db [<dump>] restores Nextcloud's database from a dump and sets up the database user again: a dump doesn't contain it, so Nextcloud couldn't log in. Then it tells sync clients the server went back in time. It replaces the manual kubectl run + pg_restore steps in the docs.
  • nixlab-restore no longer prints the --as-of time twice.

Upgrading

Nothing to change in vars.nix. Longhorn is manual-sync, so press Sync on the longhorn app once to drop its pre-upgrade hook.