Skip to content

Hydra's Packet Builders

Graham Christensen edited this page Sep 29, 2019 · 3 revisions

with the exception of one, packet-t2a-2, which is not managed this way

All of the hydra builders on Packet are stateless, netbooted machines.

How they're updated

The builders are regenerated every day at 00:00UTC at https://buildkite.com/grahamc/packet-nix-builder. The builders will also be regenerated and rebooted after a push to the master branch.

The source for the builders lives at https://github.com/grahamc/packet-nix-builder. The build uses the current stable version of NixOS.

Once the build has completed, a rolling-reboot script reboots one machine at a time, waiting for its ssh port to come back up.

How they work

This boot process erases all of the disks and creates a single ZFS stripe across all of them. No state is preserved between boots.

On the hydra server, hydra-packet-importer queries Packet's API for machines with the hydra tag and adds them to the Hydra machines file. This happens every few minutes.

Dealing with Failure

It is safe to, at any time, reboot any of the machines. When they come back up, they will have empty disks and no state, and will hopefully come up just fine.