Repository navigation
v1.22.0: rebuild from scratch, fresh clusters bootstrap by themselves
Highlights
Found by rebuilding a live cluster's only master from scratch (encrypted) and restoring it from backups. The data came back intact, but a fresh cluster needed manual help at seven points. It doesn't anymore (#46).
Rebuild the cluster from scratch: nix run .#reinstall -- <master> --new-cluster
Reinstalls the only master, which starts a new, empty cluster, and brings everything back:
- Backs up every volume (and a fresh dump of Nextcloud's database) right before.
- Reinstalls the master remotely, encrypted if
encrypt = true. - Lets the workers forget the old cluster's certificates and join the new one.
- Waits for Argo CD to bring the apps back from git.
- Restores every volume as it was before the reinstall.
Without per-app backups, it tells you the apps come back empty.
A fresh cluster bootstraps by itself
- Longhorn is manual-sync, and its Application blocked every later sync wave. The bootstrap now syncs it once.
- Longhorn's pre-upgrade hook waited forever for a service account only the sync creates. It's off, as Longhorn recommends for Argo CD.
- The gateway stayed Degraded until cert-manager, in a later wave, issued its certificates: a deadlock. A Gateway is now healthy once its controller accepted it.
- Every app keeps retrying a failed sync (backoff up to 5 minutes) instead of giving up after 5 attempts.
- nixlab-secrets no longer puts Roles into namespaces that another app owns. Those namespaces don't exist yet on a fresh cluster, which was a regression from v1.19.0. There's a scoped ClusterRole instead, and the Job creates the namespace if needed. Kubernetes can't limit "create" to a name, so in that case the Job may create Secrets in any namespace; get, patch and update stay limited by name.
Restores that can't go wrong quietly
sudo nixlab-restore-all --as-of <time>restores every volume as it was at that time. A new cluster's first backups are of its empty volumes, so "latest" would restore nothing.sudo nixlab-restore-nextcloud-db [<dump>]restores Nextcloud's database from a dump and sets up the database user again: a dump doesn't contain it, so Nextcloud couldn't log in. Then it tells sync clients the server went back in time. It replaces the manualkubectl run+pg_restoresteps in the docs.nixlab-restoreno longer prints the--as-oftime twice.
Upgrading
Nothing to change in vars.nix. Longhorn is manual-sync, so press Sync on the longhorn app once to drop its pre-upgrade hook.