Repository navigation
Replies: 2 comments
Revision 1: answers to the open questionsThis revision applies the answers to all four open questions:
Because of the second answer, a checkpoint is now the end of a call, and resuming is how a run goes on, whether it paused or failed. The command, the cache rule and the files change too, and so do parts of three other proposals. See Effects on other proposals below. Each section below replaces the section of the same name in the post, or says which part of it changes. Everything else in the post is unchanged and still applies. Decisions
SummaryThe command and the paragraph after it are replaced by: It takes one argument that matters, the target. It checks the target for state it can't explain, then applies and verifies the next phase, and stops at a checkpoint. The operator then continues the run with The entry point is a Dagger module in this repository. Its functions compose every container a phase needs from pinned images and from devex's modules, so a rebuild runs the same pinned tools wherever it starts. BackgroundTwo items are added:
Proposed designThe commandThis section replaces the post's.
They're functions of one Dagger module, which What a run reads from the machine that starts it. A Dagger function can't read the host. Everything it reads from there is an argument: the values checkout, and the operator's credentials, which are secrets. The operator sets them once, outside the repository, as Dagger user defaults. A default holds a reference to a secret, such as PhasesThe interface is unchanged. One sentence is replaced:
The order is fixed in one place, CheckpointsThis section replaces the post's. A Dagger function runs to completion, so a checkpoint is the end of a call, not a prompt inside one. After each phase verifies,
With A failed apply or verify stops the run at once. ResumingThis section is new, and goes after Checkpoints. A run that stopped, at a checkpoint or on a failure, is unfinished.
The operator decides whether a target is fit to resume. A failure can leave a phase in a state its A run can resume from a different commit than the one it started from, such as one that fixes the phase that failed. The record keeps the commits each phase was applied from. TargetsThe second rule is replaced:
The preflight checkStep 2 is replaced:
RecordsThe list is replaced:
The record lives only in the target's state backend. TimeThe first sentence is replaced: A run's time is measured from the start of its first Side effects and the cacheThis section replaces The one helper for side effects. Every function that reads or changes a target is marked
Where it runsThis section replaces the post's. A run can start wherever a Dagger engine runs: a workstation, a CI runner, or a remote engine. Every step is a container the module composes. A phase that reaches the home lab still needs a network path there. On the home-lab LAN, that's the LAN. Proposal: operations tunnel decides how a run from anywhere else reaches it. Proposal: drills decides which runs happen in CI. FilesThis section replaces the post's. The first story adds the workspace and the module, with a Design decisionsThe first decision is replaced, and five are added. The rest stand.
The decision on no default target now reads: No default target, and live teardown always needs DrillThe example is replaced, since it now shows resuming: func TestUp_AResumedRunStartsAtThePhaseThatFailed(t *testing.T) {
target := runtest.NewTarget(t, runtest.Drill)
phases := []run.Phase{
runtest.Phase("0"),
runtest.Phase("1a", runtest.FailApplyOnce(errNoEdge)),
runtest.Phase("1b"),
}
err := run.Up(t.Context(), target, phases, run.NoPause())
require.ErrorIs(t, err, errNoEdge)
err = run.Up(t.Context(), target, phases, run.NoPause(), run.Resume())
require.NoError(t, err)
require.Equal(t, []string{
"0 apply", "0 verify", "1a apply",
"0 verify", "1a apply", "1a verify", "1b apply", "1b verify",
}, target.Calls())
}Added to what the tests show:
The test that a phase run twice really runs its execs twice moves to the module's own The first drill against real infrastructure is unchanged, in the new form: Documentation
What it replacesADR-0004's entry is replaced:
Effects on other proposalsThis section is new. Each proposal takes these up in its own next revision.
Open questionsNone. This revision answers all four. ReferencesAdded:
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Note
Revision 1 answers all four open questions. The entry point is now a Dagger module on Dagger 1.0 beta, run as
dagger call up --target <name>, and Go never calls Dagger: its phases compose z5labs/devex's modules, starting withopentofu. A checkpoint is the end of a call, andup --resumecontinues a run that paused or failed. A run can start wherever a Dagger engine runs, and its record stays only in the target. The rest of this post is unchanged.Note
Revision 1 of Proposal: infrastructure modules adds a
checkcommand to this one, so infrastructure modules now builds on this proposal too.dagger run go run ./cmd/rebuild checkrunstofu fmt -check,tofu validate, TFLint andtofu teston every module and root module, in the same container a rebuild applies root modules in. It touches no target and needs no credentials, so it runs through the helper that may cache, notRun.Exec.Buildandverifyboth run it.Note
Wave 1 of Proposal: rebuilding infra from scratch, as amended by revision 3. It depends only on the charter. Proposal: definitions, values and secrets and Proposal: drills build on it, so they're discussed alongside it.
Summary
One command stands the platform up, on a workstation or in CI:
It takes one argument that matters, the target. It checks the target for state it can't explain, then runs the platform's phases in order. Each phase applies, then verifies. On a terminal, the command then pauses for the operator before the next phase.
rebuild downtears the phases down in reverse, with phase 0 last.The command is a Go program in this repository. It runs every tool a phase needs in a container, through Dagger, so a rebuild runs the same pinned tools wherever it starts. This proposal decides:
It lands no phase. Each phase comes back with its own proposal, starting with phase 0 in Proposal: definitions, values and secrets.
Background
ADR-0004 decided the shape (ADR-0004). It chose a purpose-built orchestrator in Go, as a Dagger module in this repository, with the forge as a thin invoker. Its phases run 0, 1a, 1b, 1c, 2, 3 and 4, and teardown unwinds in reverse with phase 0 last. Its reasons still hold:
Two of ADR-0004's mechanisms leaned on things the charter removes.
rebuild.ymlin Zaba505/homelab. Between jobs, a step posted to an issue on the engagement thread and polled forcontinue.Zaba505/homelab and the engagement process are both out of scope now.
Dagger has no way to wait for a person. A Dagger function runs to completion, and
Terminal()is for debugging. That's why ADR-0004 put checkpoints outside the module.Dagger can cache a side effect away. It caches function calls and the
withExeclayers inside them, andcache="never"on a function doesn't reach its execs. Deploys and destroys have returned cached success while nothing ran (dagger#7090, dagger#9607). ADR-0004 made a per-run cache-buster, added by one helper, its hardest rule.The preflight drift check was never designed. story(adr): preflight drift-detection mechanism — self-hosted-application-platform #758 asked what the last-known-good reference is and how the rebuild finds unexplained differences. ADR-0003 fixed only its form: a commit SHA on Zaba505/homelab's
main.The 60-minute target counted everything. ADR-0004 measured wall-clock time from invocation to the canary going green, including time spent at checkpoints. It treated the target as a target, not a gate.
The requirements behind all of this are TR-02 (one entry point, 60 minutes), TR-04 (teardown at every checkpoint), TR-05 (preflight that fails closed) and TR-06 (drill and live differ only in the target), in the technical requirements.
Proposed design
The command
upruns the preflight check, then applies and verifies each phase in order, through the last phase or the one--throughnames.downtears down every phase in reverse order, phase 0 last. It's safe to run again after it fails partway, and it succeeds when there's nothing left.preflightruns only the preflight check.There's no default target. A workstation needs the Dagger CLI and a container runtime. Go's version is pinned by
go.mod'stoolchainline, the Dagger SDK's bygo.mod, and the Dagger CLI's in the repository next to them, so Renovate moves the SDK and the CLI together.Phases
A phase is a Go value that meets one interface:
Runcarries the target, the run's ID, the Dagger client and the commit being rebuilt. A phase reads every value it needs from the target, and nothing from the environment it runs in.The order is fixed in one place,
rebuild/phases.go:A phase joins the list when its proposal's stories land it. A rebuild runs every phase that exists.
Checkpoints
After each phase verifies,
upprints a summary: what the phase made, how long it took, and the run's time so far. On a terminal, it then waits for the operator to choose:With
--no-pause, or with no terminal attached, as in CI,updoesn't wait. Each phase's verify is then the only gate.A failed apply or verify stops the run at once and prints what failed. On a terminal,
upoffers teardown. Unattended, it leaves the target as it is for the operator to look at, and exits non-zero.rebuild downis always the way back to nothing.Targets
--targetnames one target. A target holds every value a phase may read, such as the cloud project, the state bucket, the DNS zone, the boot endpoint and the home-lab hosts. Drill and live differ only in the target, as TR-06 requires and ADR-0004 kept.Proposal: definitions, values and secrets decides what a target holds and where it's kept. This proposal decides how a target is used:
downagainst the live target asks for its name to be typed. Unattended, it needs--confirm <name>.The preflight check
Before
upapplies anything, it checks the target:upstops. TR-04 forbids carrying partial state into a new run, so the operator runsdownfirst.uprunsVerifyfor every phase the last run applied. Any failure is unexplained drift, andupstops before it changes anything. The operator fixes the target or tears it down.Verifychecks the target against what its phase last applied, not against today's definitions. For an OpenTofu phase, that's a refresh-only plan with no changes, then the phase's own checks that what it made works. So the definition of drift is exact: drift is a difference between the target and what the rebuild last applied to it. A difference between what was last applied and today's definitions isn't drift. It's the change this run applies, and history explains it.Records
Each run writes a record to the target's state backend, which phase 0 provides:
The record of the last run that finished is the target's last-known-good. Once the canary tenant exists, a run finishes only when the canary goes green. Last-known-good then means what ADR-0003 wanted: the definitions a canary last passed on.
Time
A run's time is measured from the start of
upto the last phase's verify, including the preflight check and every pause at a checkpoint, as ADR-0004 decided. The summary prints each phase's time and the total, and marks a run that went over 60 minutes. The record keeps both, so the trend is visible across runs.A run never stops because of time. A live rebuild that's running long is still the fastest way back. Proposal: drills decides what a drill over 60 minutes means.
The one helper for side effects
Every exec that reads or changes a target goes through
Run.Exec.Run.Execadds the run's ID to the container's environment, so Dagger can never return a cached result for it. This covers applies, teardowns, and every plan or check that reads live state, since a cached read would report no drift when there is some.A forbidigo rule, added to bedrock's list, bans calling
WithExecanywhere else. Work that doesn't touch a target, such astofu validate, goes through a second helper that may cache.Where it runs
The command runs wherever the operator starts it. A phase that reaches the home lab needs a network path there. On a workstation in the home lab, that's the LAN. Proposal: operations tunnel decides how a run from anywhere else reaches it. ADR-0004's runner handoff, a self-hosted runner in the cloud anchor from phase 1b on, is a candidate there. Proposal: drills decides which runs happen in CI.
Files
The first story adds golangci-lint, with bedrock's forbidigo list and the
WithExecrule, andgo test -race ./...toBuildand toverifyin.claude/backlog.json.rebuild/is a library package, so it's held to 80% statement coverage.Design decisions
Verifyreads what was last applied, not today's definitions. That lets one method serve both after apply and in the preflight check, and it keeps a change in the definitions from looking like drift.uprefuses to build on top of a partial run.Drill
The entry point's whole is the run loop, and its tests drive the loop with fake phases, without Dagger or credentials:
The tests show that:
downruns them in reverse with phase 0 last;downsucceeds when run twice, and after anupthat failed at each phase in turn;downagainst live refuses without the name;The first drill against real infrastructure is phase 0 on a scratch target, once Proposal: definitions, values and secrets lands it.
Documentation
docs/content/fundamentals/rebuild.md, weight 20): the command, the phase order, the phase contract, checkpoints, the preflight check, records and time.docs/content/getting-started/first-drill.md, weight 20) and Going live (docs/content/getting-started/going-live.md, weight 30). This proposal writes the commands and what each checkpoint shows. Drills and definitions add their steps. This proposal also adds the tools to What you need, which Proposal: definitions, values and secrets writes.docs/content/operating/routine-maintenance.md, weight 20): keeping Dagger and Go current, which matters more than usual while Dagger is pre-1.0.Each page links to the next page in the reading order that exists, and each story fixes the link on the page before its own.
What it replaces
rebuild.ymland no target definitions in Zaba505/homelab.Open questions
up --resumecontinue from the phase that failed, once the operator has looked?daggerCLI and modules as the way in, and have said the SDK starting its own engine may be deprecated. This design runs underdagger run, which is the supported path, but it still depends on Dagger keeping library use. Is that acceptable?Out of scope
References
All reactions