Skip to content

v1.21.0: reinstall a running node remotely

Choose a tag to compare

@thatbagu thatbagu released this 18 Nov 20:32
f4b42e1

Highlights

nix run .#reinstall -- <node>

Reinstalls a running node from scratch, over SSH, with nobody at the machine. For example, to turn on encryption or to start clean. install can't do this itself: running nixlab nodes have kexec disabled on purpose, and kexec hangs some hardware anyway (Intel N100 mini PCs).

The node reboots into a NixOS installer put on its own disk, booted once through GRUB: a real reboot, which falls back to NixOS if the entry fails. The installer reboots back into NixOS by itself on a panic, on a hang (hardware watchdog), or if it isn't claimed within 15 minutes. Then install runs, the node rejoins the cluster, and Longhorn forgets its old disk so replicas rebuild onto the new one. It refuses the only master, since that would start a new cluster.

Ran on real hardware: an encrypted reinstall of a worker in a live 3-node cluster, with unattended TPM unlock (#43).

nixlab-longhorn-reset-node <node> (on masters)

After a reinstall, Longhorn still expects the node's old disk ("record diskUUID doesn't match the one on the disk") and schedules no replicas there. This forgets the stale disk record and its replicas, so Longhorn adds a fresh disk. It refuses if any volume would lose its last healthy replica. reinstall runs it for you.

Upgrading

Nothing to change in vars.nix. Deploy this version to your masters before using reinstall, since it needs the new helper there (it checks before erasing anything).