Proposal: drills

[Zaba505]

Note

Revision 1 of Proposal: the rebuild entry point makes the entry point a Dagger module. rebuild drill becomes dagger call drill [--through <phase>], and a drill still runs up with --no-pause. A Dagger function can't read the host, so the GitHub token that posts the drill commit status reaches it as a secret argument, like every other credential. This proposal's next revision takes these up.

Note

Wave 1 of Proposal: rebuilding infra from scratch, as amended by revision 3. It builds on Proposal: the rebuild entry point, Proposal: infrastructure modules and Proposal: platform topology while they're decided. Revision 3 leaves module tests to infrastructure modules, and gives this proposal everything that runs after them.

Summary

Every phase is drilled before it touches the live platform, at three levels:

1. Checks run on every pull request, in Build, with no credentials: format, validation, lint, module tests with mocked providers, and Go tests.
2. A phase drill stands a fresh scratch target up through one phase, checks it, and tears it down. It runs before a change to that phase reaches the live platform.
3. A whole drill runs the full rebuild on a fresh scratch target, checks it, times it, and tears it down. It runs after a significant change, and at least once a quarter. A whole drill that takes over 60 minutes fails.

A scratch target is the live target's shape with its own values: a new Google Cloud project, its own DNS names on a drill domain, and virtual hosts on an isolated drill VLAN in the home lab. Drill and live differ only in the target.

For now, checks run in CI, and drills run from the operator's workstation. Cloud drills need a credential CI doesn't hold yet, and home-lab drills need the LAN.

Background

• TR-06 requires drills on scratch infrastructure after significant changes and at least quarterly, without touching the live platform, with drill and live differing only in the target (technical requirements).
• ADR-0004 decided how the home lab is drilled (ADR-0004):
  • a drill's home-lab hosts are virtual machines on the home-lab LAN, not in the cloud, so the tunnel is still a real crossing between two environments;
  • they sit on an isolated segment whose DHCP points at the drill target's boot endpoint, because a drill VM that booted live artifacts would pass a test it never ran;
  • a drill doesn't prove firmware, BMC power-on, disk controllers or NIC drivers;
  • a drill's teardown deletes VMs, while a live teardown powers hosts off and wipes them, so drills don't test the live teardown of the home lab.
• ADR-0003 made phase 0 take the target as its parameter, so a drill makes its own state bucket.
• The charter's principle 3: every phase runs against scratch targets before it runs against the live platform, and every proposal shows its drill.
• Proposal: the rebuild entry point gives up --through <phase>, a down that's safe to run again, a preflight check, run records with each phase's time, and leaves to this proposal what a drill over 60 minutes means.
• Proposal: infrastructure modules owns module tests, with mocked providers.
• Proposal: definitions, values and secrets has phase 0 create the target's project. A deleted project's ID can never be used again.
• Proposal: platform topology lists the checks each path's phase must show in its drill.
• The home lab's network has three VLANs today: IPMI, network boot, and the main home-lab network (Hardware).

Proposed design

The levels

Level	What runs	Against	Where	When	Credentials
checks	tofu fmt, tofu validate, TFLint, tofu test, golangci-lint, go test	nothing	Build and verify	every pull request	none
phase drill	rebuild drill --through <phase>	a fresh scratch target	the operator's workstation	before a change to the phase reaches live	the operator's
whole drill	rebuild drill	a fresh scratch target	the operator's workstation	after a significant change, and at least quarterly	the operator's

A phase drill stands up every phase before the one it names too, since each phase stands on the ones below it.

The drill command

Proposal: the rebuild entry point gets one more command:

dagger run go run ./cmd/rebuild drill [--through <phase>]

It:

1. makes a fresh scratch target from targets/drill.yaml in the values repository, with a new drill ID, such as drill-20261024-a1b2, in every name that must be unique: the project, and the DNS names;
2. runs up against it with no pauses, so each phase's verify is the only gate, as in an unattended live run;
3. runs preflight, which must pass, because nothing has changed since the apply;
4. runs down, even when an earlier step failed;
5. checks that nothing is left: the project is gone, and no DNS record carries the drill ID;
6. reports the result, each phase's time and the total, and posts the result as a drill commit status on the commit it ran.

A drill fails when any step fails, when anything is left after teardown, or, for a whole drill, when up takes over 60 minutes. If teardown fails, the drill says what's left and how to remove it: rebuild down --target drill-….

targets/drill.yaml holds what every drill shares, such as the billing account and the drill domain. The drill ID fills in the rest.

Scratch targets

• Cloud. A new Google Cloud project for every drill. Phase 0 creates it and its teardown deletes it, so no drill sees another's state. A new ID per drill also matters because a deleted project's ID can never be used again.
• DNS and the edge. A drill's names live under its ID on a drill domain: <name>.<drill ID>.<drill domain>. The drill domain's zone is permanent, and each drill only adds and removes its own records. Proposal: cloud anchor and edge defines the zone.
• The home lab. A drill's hosts are VMs on an isolated drill VLAN, whose DHCP points at boot.<drill domain>. The drill points that name at its own boot endpoint while it runs, so the VLAN's DHCP setting never changes. Only one home-lab drill runs at a time. Proposal: home-lab hosts and Proposal: operations tunnel decide how the VLAN and its DHCP setting are defined.

CI and the workstation

• Checks run in CI now.
• Cloud drills could run in CI once it holds a Google Cloud credential that can create projects under the billing account, and a Cloudflare token for the drill zone. Proposal: operator access decides how such credentials are held and revoked. Until it lands, cloud drills run from the workstation, with the operator's credentials.
• Home-lab drills need the LAN. They run from a workstation in the home lab, or, once Proposal: operations tunnel gives a run from elsewhere a way in, from CI.

What a drill proves, and what it doesn't

A drill proves the phases, their order, the definitions, the shape of the values, teardown, and the rebuild's time, all on scratch infrastructure. It doesn't prove:

• that this hardware boots: firmware, BMC power-on, disk controllers and NIC drivers;
• the live teardown of the home lab, which wipes disks instead of deleting VMs;
• the live zone's DNS;
• the live platform's capacity.

The Drills page says so next to every result, so a green drill is never read as more than it is.

Design decisions

• A fresh target for every drill. A scratch target that's used again accumulates state, which is what a drill exists to rule out.
• Drills run without pauses. They test the same gates an unattended live run relies on: each phase's verify, and the preflight check.
• Teardown always runs, and it's checked. A drill that leaves resources behind fails, even when every phase passed, because a teardown that doesn't work is the failure TR-04 exists to catch.
• A whole drill over 60 minutes fails. A live rebuild never stops for time, so the whole drill is where the 60-minute target is held. An overrun is a defect in the platform, found before a real rebuild needs the time.
• Results go on the commit. A drill commit status shows which commits were drilled, without a new place to look.
• The workstation first. No CI credential exists that could create projects, and creating one is Proposal: operator access's decision.
• One fixed boot name for drills. It keeps the drill VLAN's DHCP setting fixed, at the cost of one home-lab drill at a time.

Drill

The drill command's own tests use fake phases, as the entry point's do, with no Dagger and no credentials. They show that:

• down runs after an up that failed at each phase in turn;
• a drill fails when a fake phase leaves something behind after teardown;
• a whole drill fails when, on a fake clock, up runs past 60 minutes, and a phase drill doesn't;
• each drill gets a new ID, and that ID is in every unique name.

The first real drill is phase 0, once Proposal: definitions, values and secrets lands it:

dagger run go run ./cmd/rebuild drill --through 0

Documentation

• Fundamentals → Drills (docs/content/fundamentals/drills.md, weight 40): the three levels, scratch targets, what runs where, and what a drill proves and doesn't.
• Operating → Running a drill (docs/content/operating/running-a-drill.md, weight 30): running a phase drill and a whole drill, reading the result, and cleaning up after a failed teardown.
• Getting started → A first drill: the rebuild drill step, in the page Proposal: the rebuild entry point writes.

What it replaces

• ADR-0004's drill decisions.
  • Kept: VMs on the home-lab LAN, an isolated segment with its own DHCP setting, and the named limits of a drill.
  • Added: a fresh target per drill, the drill command, a 60-minute limit for whole drills, and which runs happen in CI.
• Nothing in the tree. There have never been drills.

Open questions

1. DNS for drills. A second domain, with a zone of its own, keeps drills out of the live zone entirely, for the cost of a domain. The alternative is drill records under a prefix in the live zone, which would mean a drill changes live state. Which?
2. Where drill VMs run. On the workstation that runs the drill, bridged to the drill VLAN, no live host is involved, but the workstation needs the memory for every drill host. On a home-lab host's hypervisor, the drill uses live capacity, and needs that host up. Which?
3. Only drilled commits go live? Should rebuild up against the live target refuse a commit without a passing drill status? Principle 3 would then be enforced instead of remembered, but every change to live would wait for a drill.
4. Drills on a schedule. Once Proposal: operator access gives CI a credential, should a whole cloud drill run in CI on a schedule, and how often, given that each one costs money for the time its project exists?

Out of scope

• Module tests: Proposal: infrastructure modules.
• The credentials CI holds: Proposal: operator access.
• How the drill zone, the drill VLAN and its DHCP setting are defined: Proposal: cloud anchor and edge, Proposal: home-lab hosts and Proposal: operations tunnel.

References

• Proposal: rebuilding infra from scratch, the charter, and its principle 3
• ADR-0003 and ADR-0004
• Hardware