Proposal: definitions, values and secrets

[Zaba505]

Note

Revision 1 of Proposal: the rebuild entry point changes how values and credentials reach a run. The entry point is now a Dagger module, and a Dagger function can't read the host. So the values checkout and the operator's credentials are arguments, set once as Dagger user defaults, which hold references to secrets, never their values. The entry point no longer reads credentials from the operator's environment or defaults to ../infra-values. The drill's commands become dagger call …, starting with dagger call up --target <drill> --through 0. This proposal's next revision takes these up.

Note

Wave 1 of Proposal: rebuilding infra from scratch, as amended by revision 3. It builds on Proposal: the rebuild entry point, Proposal: infrastructure modules and Proposal: platform topology while they're decided. It takes over ADR-0003.

Summary

Besides this repository, a rebuild is given exactly two things: a target's values, and the operator's credentials.

• Definitions are this repository. It's public, and holds no secret and no environment's real values.
• Values are each target's settings, such as its project and its domain. They live in one private repository, Zaba505/infra-values, with one file per target. Every run records the commit of the values it ran with.
• Secrets come in three kinds:
  • the operator's credentials, which the operator holds and hands to a run, and which the platform never stores;
  • carried secrets, the few a rebuild from nothing can't make again without a manual step, kept in the values repository and encrypted with SOPS;
  • generated secrets, everything else, which a phase writes to the target's Secret Manager, and which the next rebuild from nothing makes again.
• Phase 0 creates a target's Google Cloud project and its state bucket. It runs first against a local state backend, then moves its own state into the bucket it made. It's the first phase up and the last one down.

Background

• ADR-0003 split the definitions across two repositories (ADR-0003). Public, reusable modules lived here. Each environment's bindings, values and apply workflows lived in Zaba505/homelab. A module ref joined them, and ADR-0003 made that ref a tracked definition. Last-known-good was a commit SHA on Zaba505/homelab's main. Zaba505/homelab is out of scope now, so that split goes.
• Much of ADR-0003 still stands:
  • state is derived, lives in the target, and is never committed;
  • a definitions-driven phase 0 creates the state bucket against a local backend, then adopts its own state; it counts against the 60 minutes, takes the target as its parameter, and is torn down last;
  • secrets live in Secret Manager, referenced by name, and no secret value is committed;
  • values are private because of aggregation, pre-disclosure and a mistake's blast radius, not because hiding them is a defense.
• The charter's principles 2 and 4: everything the platform is lives in this repository or in the values a rebuild is given, and this repository holds no secrets and no environment's real values.
• Proposal: the rebuild entry point takes one --target. Every phase reads its values from the target and nowhere else. A live run refuses uncommitted changes, and each run's record lives in the target's state backend.
• Proposal: infrastructure modules picks OpenTofu, which can encrypt state on the client.
• Proposal: platform topology puts the state backend in the Google Cloud anchor, with Cloudflare as the edge.
• A deleted Google Cloud project's ID can never be used again (Delete and restore projects). A project waits 30 days before it's fully deleted, and its ID stays reserved after that.
• A successor holds sealed credentials. Proposal: operator access decides how they're sealed.

Proposed design

What a rebuild is given

	Where it lives	Who can read it	Changes recorded by
definitions	this repository	anyone	git
values	Zaba505/infra-values, targets/<name>.yaml	the operator and the successor	git
carried secrets	Zaba505/infra-values, targets/<name>.secrets.yaml, encrypted	whoever holds an age key it's encrypted to	git
operator credentials	with the operator	the operator	none
generated secrets	the target's Secret Manager	the phases that use them	the rebuild
state	the target's state bucket	the rebuild	the bucket's versioning

Values

A target's values are one YAML file, read strictly: an unknown key, a missing key or a wrong type stops the rebuild.

# targets/live.yaml, in Zaba505/infra-values
kind: live
google:
  billingAccount: 000000-000000-000000
  project: zaba505-infra-live-1
  region: us-central1
cloudflare:
  account: 0123456789abcdef0123456789abcdef
  zone: example.com

• The entry point's Target type is the file's schema. Each phase's proposal adds the keys it needs.
• Names that must be unique come from the project. The state bucket is <project>-state, so a target needs no second globally unique name.
• The values repository is private for ADR-0003's reasons, not because any value is secret: the whole set, in one place, maps the platform for anyone who reads it.
• Proposal: drills decides how a drill target's file is made.

Secrets

1. The operator's credentials are what a rebuild from nothing needs before anything exists:

   • the operator's own Google Cloud login, through gcloud auth application-default login;
   • a Cloudflare API token;
   • the operator's age key, which opens the carried secrets.

   On a workstation, they come from the operator's environment. The entry point passes each one into the containers that need it as a Dagger secret, so it never appears in a log, an image layer or a cache key. Proposal: operator access decides how the operator keeps them and how the successor's sealed copy is made. Proposal: drills decides what CI gets.

2. Carried secrets are encrypted with SOPS, using age, to two recipients: the operator's key and the successor's. They're the few secrets that a rebuild from nothing can't make again without a manual step or without losing data. Each part's proposal names its own. For example:

   • a key the router holds, if the operations tunnel's home-lab end can't be configured by the rebuild;
   • the key that encrypts backups, which backup, restore and export will need to outlive any rebuild.

3. Generated secrets are everything else, such as origin certificates, service account keys, and tunnel keys the rebuild configures on both ends. A phase that makes one writes it to the target's Secret Manager, and a phase that needs one reads it by name. They're destroyed with the target.

The rule between 2 and 3: a secret is generated unless making it again would need a manual step or lose data.

How values reach a phase

1. The entry point reads targets/<target>.yaml from a checkout of Zaba505/infra-values, at the path --values gives, and ../infra-values by default.
2. If targets/<target>.secrets.yaml exists, the entry point decrypts it with SOPS's Go package, using the age key from the operator's environment.
3. It decodes both into Target, strictly.
4. A phase passes values to OpenTofu as a generated .tfvars.json file, and secrets as Dagger secrets. A value or secret never appears on a command line, where a process list or a log could show it.

The record of each run holds the values repository's commit, so the preflight check and the last-known-good record cover values as well as definitions.

Phase 0: the state backend

Phase 0 is the root module and Go phase in phases/statebackend/, and it uses modules/google-project and modules/google-state-bucket.

• Apply. On its first run against a target, its state is local, inside the run's container. It creates the project under the billing account, enables the APIs the next phases need to start (Cloud Storage and Secret Manager), and creates the state bucket with versioning on, uniform access, and public access prevented. It then moves its own state into the bucket it just made (tofu init -migrate-state). Later runs find the bucket and use it.
• Verify. A refresh-only plan shows no change, and the run can write, read and delete an object in the bucket.
• Teardown. Phase 0 goes last. It moves its own state back to the run's container, empties and deletes the bucket, then deletes the project.

Every other phase's root module keeps its state in the same bucket, under a prefix named for its phase. The entry point passes each one its backend settings from the target, so no root module names a bucket.

Design decisions

• A new private repository for values, not Zaba505/homelab. The charter takes Zaba505/homelab out of scope. A new repository keeps values in git, so a run can record which values it ran with, and the preflight check can tell what changed.
• Values in git, not in the cloud. A rebuild from nothing starts with no cloud, and values kept there would be gone with it.
• Carried secrets are encrypted in the values repository, not kept in a password manager. One age key opens all of them, so the sealed set the successor holds stays small, and each secret is versioned with the values it goes with. This changes ADR-0003's rule that no secret value is committed even to the private repository. The new rule is that only ciphertext is committed, and .sops.yaml lists who can read it.
• Generated is the default. A rebuilt platform that makes its secrets fresh carries nothing a leak could have touched. That's "rebuilt, not repaired", applied to secrets.
• Phase 0 creates the project. A drill then needs only the billing account, and a target is entirely what its definitions and values make. The cost is in open question 1.
• Credentials only ever enter a run as Dagger secrets. Dagger keeps them out of logs, layers and cache keys, which a plain environment variable or file in a container wouldn't.

Drill

Phase 0 is the first phase with a drill against real infrastructure. Against a drill target:

dagger run go run ./cmd/rebuild up --target <drill> --through 0
dagger run go run ./cmd/rebuild preflight --target <drill>
dagger run go run ./cmd/rebuild down --target <drill>
dagger run go run ./cmd/rebuild down --target <drill>

It shows that, starting from a billing account and nothing else:

• phase 0 makes a project and a bucket, moves its own state in, and verifies;
• the preflight check then passes, because nothing has changed since the apply;
• teardown leaves no project and no bucket behind;
• teardown run a second time succeeds, with nothing to do.

Without credentials, in Build:

• phase 0's root module and its two modules have tests with mocked providers, as Proposal: infrastructure modules requires;
• the values loader's tests show that an unknown key, a missing key, a secrets file that won't decrypt, and a live target with uncommitted values each stop the run.

Documentation

• Fundamentals → Values and secrets (docs/content/fundamentals/values-and-secrets.md, weight 30): definitions, values, the three kinds of secret, phase 0, and how a value reaches a phase.
• Getting started → What you need (docs/content/getting-started/prerequisites.md, weight 10): the accounts, the operator's credentials, the age key and a checkout of the values repository. Proposal: the rebuild entry point adds the tools.

What it replaces

• ADR-0003, taken over without Zaba505/homelab.
  • Kept: state derived, in the target and never committed; phase 0 as ADR-0003 designed it; generated secrets in Secret Manager, by name; private values, for ADR-0003's reasons.
  • Changed: values in Zaba505/infra-values; no module ref, because a rebuild runs this repository's commit directly; last-known-good is the entry point's record, not a commit on Zaba505/homelab; carried secrets are committed, encrypted.
• Nothing in the tree. The reset deleted every module, and no root module was ever here.

Open questions

1. Who owns a target's project? In this design, phase 0 creates the project and its teardown deletes it, so a drill needs only a billing account. But a deleted project's ID can never be used again. After a live teardown, the live target needs a new project ID, which is a one-line change to its values. The alternative is that each target's project exists before any rebuild, and teardown leaves it empty. That keeps the ID, but the project is then something set up outside the rebuild. Which?
2. Is state encrypted? OpenTofu can encrypt state with a Cloud KMS key, so a copy of the bucket would reveal no secret. But Cloud KMS key rings can't be deleted. Under this design, a key ring would go when its project does. Under the alternative in question 1, a rebuilt target would need a new key ring name each time. Should state be encrypted now, or should it rely on the bucket's access rules?
3. SOPS and age, or a password manager? A password manager would keep carried secrets out of git entirely. It's another vendor, though, and the successor would need access to it as well. Which?
4. The repositories' rulesets. The default ruleset was set by hand, which principle 2 calls memory, and ADR-0003 asked for it to become a definition. Should a phase or a script apply it from definitions, and which proposal owns that: this one, or Proposal: operator access?

Out of scope

• How the operator keeps credentials, and the successor's sealed set: Proposal: operator access.
• CI's credentials: Proposal: drills and Proposal: operator access.
• Which secrets each part carries: each part's proposal.
• Backups of tenant data, and their keys: Proposal: backup, restore and export.

References

• Proposal: rebuilding infra from scratch, the charter
• ADR-0003
• Delete and restore projects, on project IDs after deletion