From e511590113497723f9fdf50d1cb06020c8e17169 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 4 Sep 2026 22:18:19 +0000 Subject: [PATCH 1/2] feat(secrets): hold a second age recipient, prove each one separately (#106) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #106 argues the runbook frames a loss problem as a headcount one: "worth doing on the day the lab stops being a one-person project" describes who holds the key, when the risk is that the key stops existing. ADR-0015 then answered the same question the other way four months later, and both halves of that answer are about `oracle` specifically — a powered, network-attached host in the same room that already holds the ciphertext it would then be able to read. That rejection stands and is reaffirmed; it does not generalise to a key held offline. What decides it is an argument neither document makes. For a one-person lab a second recipient and a second copy of the existing key are almost the same object: two secret artefacts, either one decrypts everything, same exposure. They differ in one respect and it is not cryptographic — .sops.yaml records a recipient and nothing records a copy. verify-key- backup.sh can name which recovery path a run just proved; it cannot tell two copies of one key apart. A copy in a drawer is the .env of key backup, which is the trade secrets/README.md already refused once. That exposed a defect the decision would otherwise introduce. SecretsKeyBackupUnproven fired on a single series that run-scheduled.sh writes whatever KEY= pointed at, so with two recipients proving either would reset the ninety-day clock for both and the other could rot behind a green alert — the alert getting more wrong the more recovery paths existed. It now fires per recipient, off homelab_key_recipient_last_proof_timestamp_seconds, and a recipient that has never been verified is recorded as 0 rather than omitted so it is loud rather than invisible. The recipient list is read from the sops: block inside the encrypted file, not from .sops.yaml: the first is the keys that can open the bytes on disk, the second is policy for the next encryption, and they diverge for exactly as long as it takes to add a recipient and forget sops updatekeys. verify-key-backup.sh matched against .sops.yaml before, so in that window it passed a key that then failed at the decrypt with "the secret half is damaged", sending you to re-copy a good backup. `make secrets-add-recipient PUBKEY=age1...` takes the public half only. It resolves the creation_rule from the recipients the file already uses rather than re-implementing sops' first-match-wins path_regex, which makes ADR-0020's mistake structurally impossible; refuses a key whose private half is on this host; re-keys in the same run; and rolls .sops.yaml back if that fails, because a recipient advertised as a recovery path that cannot decrypt is worse than either end state. bootstrap.sh now points at it instead of advising a hand-edit. Exercised against a synthetic repo with throwaway keys: add, re-key, independent decrypt by the new key alone, idempotent re-add, the own-key/truncated/private-key guards, per-recipient carry-forward, and the multi-line recipient form parsing under sops 3.9.4. promtool passes including a new two-recipient case asserting exactly one alert naming the unproven key — the state the previous rule was silent about. The second keypair itself is not created here and cannot be: its private half has to be generated where it will live. Until it is, the estate has one recipient and every check behaves as it did before. Co-Authored-By: Claude Opus 5 --- .sops.yaml | 13 + Makefile | 23 ++ .../adr/0015-give-oracle-the-off-host-jobs.md | 10 + ...recipient-and-prove-each-one-separately.md | 180 +++++++++++++ docs/observability.md | 10 +- docs/runbooks/back-up-the-age-key.md | 96 ++++++- scripts/add-recipient.sh | 253 ++++++++++++++++++ scripts/bootstrap.sh | 12 +- scripts/install-timers.sh | 7 + scripts/key-recipients.sh | 173 ++++++++++++ scripts/verify-key-backup.sh | 56 +++- secrets/README.md | 22 +- .../prometheus/rules/backup.rules.yaml | 43 ++- .../prometheus/tests/backup.test.yaml | 45 +++- 14 files changed, 896 insertions(+), 47 deletions(-) create mode 100644 docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md create mode 100755 scripts/add-recipient.sh create mode 100755 scripts/key-recipients.sh diff --git a/.sops.yaml b/.sops.yaml index 70f5198..030b54a 100644 --- a/.sops.yaml +++ b/.sops.yaml @@ -22,6 +22,19 @@ # # Run `make secrets-init` on a fresh machine to generate a keypair and write its # public half into this file. +# +# ADDING A RECIPIENT +# +# `make secrets-add-recipient PUBKEY=age1...`, not a hand-edit. Editing this +# file without running `sops updatekeys` produces a recipient the repository +# advertises as a recovery path which cannot decrypt anything, and that is the +# DEFAULT outcome of doing it by hand — see +# docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md. +# +# Recipients are comma-separated, written one per line in the folded scalar. +# sops reads the value as a single string, splits it on commas and trims each +# part, so the line break is cosmetic and the trailing comma is not. Measured on +# sops 3.9.4; a wrong guess here parses as YAML and decrypts nothing. creation_rules: # The lab stack, FIRST — SOPS takes the first rule whose path_regex matches, diff --git a/Makefile b/Makefile index bd9444d..ca38b5a 100644 --- a/Makefile +++ b/Makefile @@ -102,6 +102,29 @@ secrets-edit: ## Edit the encrypted secrets in $$EDITOR @# unencrypted disk. scripts/secrets-edit.sh silences the editor first. ./scripts/secrets-edit.sh $(STACK) +.PHONY: secrets-add-recipient +secrets-add-recipient: ## Add a second age recipient and re-key (PUBKEY=age1...) + @# Under Secrets and not Maintenance because it is part of setting the + @# secrets up, but it is the one target here that rewrites a committed file + @# — .sops.yaml and secrets/$(STACK).sops.yaml both change and must be + @# committed together. ADR-0024 says why more than one recipient exists. + @# + @# PUBKEY rather than ARGS, for the reason secrets-verify-backup takes KEY: + @# exactly one argument, required, and an empty ARGS would reach the script + @# as no argument at all and print usage, which reads like a broken target. + @# + @# The PUBLIC half only. The private half of the key being added must never + @# be generated on, copied to, or pasted into this host — that is the whole + @# property the second recipient exists to have. + @[[ -n "$(PUBKEY)" ]] || { \ + printf '\033[0;31merror:\033[0m PUBKEY is required\n' >&2; \ + printf 'Generate the keypair where it will LIVE, then bring back its public half:\n' >&2; \ + printf ' make secrets-add-recipient PUBKEY=age1...\n' >&2; \ + printf 'See docs/runbooks/back-up-the-age-key.md\n' >&2; \ + exit 2; \ + } + ./scripts/add-recipient.sh "$(PUBKEY)" $(STACK) + .PHONY: secrets-show secrets-show: ## Print the decrypted secrets to stdout (careful) sops --decrypt $(SECRETS) diff --git a/docs/adr/0015-give-oracle-the-off-host-jobs.md b/docs/adr/0015-give-oracle-the-off-host-jobs.md index 1725839..f1464eb 100644 --- a/docs/adr/0015-give-oracle-the-off-host-jobs.md +++ b/docs/adr/0015-give-oracle-the-off-host-jobs.md @@ -2,6 +2,16 @@ **Status:** Accepted · 2026-09 +> [!NOTE] +> "Not a second age recipient", below, is narrowed by +> [ADR-0024](0024-hold-a-second-age-recipient-and-prove-each-one-separately.md). +> The rejection of **`oracle`** as a recipient is unchanged and still the design +> — this machine holds ciphertext and no key. What ADR-0024 sets aside is the +> broader framing quoted with it, that a second recipient is worth having only +> once the lab stops being a one-person project: that makes a loss problem read +> as a headcount one. A second recipient, held offline and off this estate, is +> now the design. The text here is left as written, per ADR-0001. + ## Context [#94](https://github.com/Gerrrt/HomeLab/issues/94) opens with "`oracle` diff --git a/docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md b/docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md new file mode 100644 index 0000000..e7aca56 --- /dev/null +++ b/docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md @@ -0,0 +1,180 @@ +# ADR-0024: Hold a second age recipient, and prove each one separately + +**Status:** Accepted · 2026-09 + +## Context + +[#106](https://github.com/Gerrrt/HomeLab/issues/106) reopens a question +[`back-up-the-age-key.md`](../runbooks/back-up-the-age-key.md) had already +answered, and is right to. That runbook closes with: + +> One key, one person. … It is worth doing on the day the lab stops being a +> one-person project, and not before — every extra recipient is another key that +> can leak. + +The issue's objection is that **the risk being described is not about team +size**. Nobody joining is what makes a second *holder* worth having; it has +nothing to do with whether the key survives a disk failure. Losing the key does +not degrade anything — every encrypted value in git history becomes permanently +undecryptable, and the recovery path is re-deriving each credential from the +device it belongs to. That is four SNMP rotations on live hardware, one of which +(`neo`) cannot persist a community deletion and needs a reboot, which in turn +cannot happen during working hours because that switch carries every VLAN. The +cost of the failure is measured in scheduled outages on someone else's calendar, +not in effort. + +[ADR-0015](0015-give-oracle-the-off-host-jobs.md) then answered the same +question the other way, four months later, and it is worth being precise about +what it actually rejected: + +> **Not a second age recipient.** … A second key on a powered, network-attached +> host in the same room, administered by the same person, adds no person and no +> offline copy. It only adds a key. Worse, it would be *this* host. … Giving it +> a private key that decrypts the estate's secrets makes it the one place where +> the backups and the means to open them sit on the same 5400 rpm disk. + +**Both of those paragraphs stand.** That argument is about `oracle`, and it does +not generalise: it turns on the machine being powered, network-attached, in the +same room, and already holding the ciphertext it would then be able to read. +[ADR-0023](0023-keep-the-household-recovery-path-outside-the-estate.md) has +since sharpened the same point in a different context — `oracle` is "the same +VLAN, the same rack, the same power feed and the same room", and every failure +worth insuring against reaches both boxes in one event. A key held offline, off +this estate, shares none of those properties. `oracle` remains a machine that +holds ciphertext and no key. + +### The argument #106 does not make, and it is the one that decides this + +For a one-person lab, a second *recipient* and a second *copy of the existing +key* are almost the same object. Either way there are two secret artefacts, each +of which alone decrypts everything, and losing both is total loss. The +cryptography does not care which one is chosen, and the confidentiality cost — +"another key that can leak" — is identical, because a second copy is also +another thing that can leak. + +They differ in exactly one respect, and it is not a cryptographic one: + +**`.sops.yaml` records a recipient in git. Nothing records a copy.** + +`verify-key-backup.sh` reads the public half out of the backup being tested and +matches it against the recipients of the encrypted file, so it can say *which* +recovery path a given run just proved. Two copies of one key are indistinguishable +to it and to everything else here — the same public half, the same match, the +same green result. Prove one and the tooling reports, accurately as far as it +can tell, that the backup works. + +That is the distinction this repository has already decided it cares about, in +this exact domain. [`secrets/README.md`](../../secrets/README.md) rejects a +gitignored `.env` and gives the reason: + +> A gitignored `.env` keeps secrets out of the repository, but it also keeps +> them out of any backup, review or history. + +And [`run-scheduled.sh`](../../scripts/run-scheduled.sh) exists because: + +> the requirement is not "run the job", it is "make NOT having run the job +> observable". + +A second copy of the key in a drawer is the `.env` of key backup. It works, and +nothing in this repository can see it, check it, or notice when it goes bad. + +### What that exposes about the existing deadline + +`SecretsKeyBackupUnproven` fires on +`homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup"}`, +which is one series. `run-scheduled.sh` writes it on every successful run +whatever `KEY=` pointed at. With one recipient that is exactly right and the +alert means what it says. + +With two it stops being true. Proving either copy resets the ninety-day clock +for both, so the second can rot behind a green alert — and the alert would be +*more* wrong the more recovery paths existed, which is the opposite of what +adding them is for. This is not a consequence of the decision below; it is a +defect the decision would introduce if left alone, and it is the reason the +mechanism is not, as #106 puts it, free. + +## Decision + +**The estate's secrets are encrypted to more than one age recipient, and every +recipient carries its own ninety-day proof.** Three parts. + +The mechanism and the per-recipient deadline are built with this ADR. **The +second keypair itself is not**, and cannot be: its private half must be +generated where it will live, which is somewhere this repository cannot reach — +see part 1. Until the operator does that, the estate has one recipient and every +check below behaves exactly as it did before, which is the honest state to leave +it in rather than pretending a key exists. + +**1. A second recipient, held offline and off this estate.** Its private half is +generated on the medium or machine that will keep it and never touches the +monitoring host — `scripts/add-recipient.sh` refuses a public key whose private +half it can find at `~/.config/sops/age/keys.txt`, because a key generated here +and added here is a second copy on the disk being insured wearing the costume of +a second recovery path. It is not `oracle`, for ADR-0015's reasons, restated +above and unchanged. Where it goes instead is an operational choice recorded in +`back-up-the-age-key.md`; the only constraint this ADR imposes is that it must +not fail at the same time as the first copy, which rules out the same drawer as +firmly as it rules out the same disk. + +**2. `make secrets-add-recipient PUBKEY=age1...` is how one is added.** Public +half only. It resolves which `creation_rule` governs the stack by looking at the +recipients the encrypted file *already uses*, rather than re-implementing sops' +first-match-wins `path_regex` resolution — which makes writing a key into the +wrong rule structurally impossible, and that matters because +[ADR-0020](0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) gives +`lab` a rule of its own precisely so a lab-guest key cannot decrypt the estate's +SNMP communities. It re-keys in the same run and rolls `.sops.yaml` back if that +fails, because the intermediate state — a recipient this repository advertises +as a recovery path which cannot decrypt anything — is worse than either end. + +**3. The proof is per recipient, read out of the ciphertext.** +`scripts/key-recipients.sh` emits +`homelab_key_recipient_last_proof_timestamp_seconds` with one series per +recipient, and `SecretsKeyBackupUnproven` fires on that instead. A recipient +that has never been verified is recorded as `0`, not omitted, so it is loud +rather than invisible — the same choice `run-scheduled.sh` makes for a job that +has never run. + +The recipient list comes from the `sops:` block inside +`secrets/.sops.yaml` and **not** from `.sops.yaml`. Those answer different +questions: `.sops.yaml` is the policy for the next encryption, and the file's own +metadata is the set of keys that can open the bytes on disk. They diverge for +exactly as long as it takes somebody to add a recipient and forget +`sops updatekeys`, which is a window in which the repository advertises a +recovery path that does not exist. Reading the ciphertext means every check here +is a statement about what can actually be recovered. + +## Consequences + +- **The confidentiality cost is real and is accepted.** There is now a second + private key that decrypts every secret in this repository, and ADR-0015's + "another key that can leak" applies to it in full. What changes the balance is + that the alternative being compared against is not "one key" — it is "one key + and an unaudited second copy of it", which carries the same exposure and + cannot be checked. +- **Revocation is still rotation.** Removing a recipient and re-keying protects + future values only; every historical ciphertext in git remains readable by the + removed key. A leaked second recipient means rotating every credential, exactly + as a leaked first one does. Nothing here improves that and the runbook says so. +- **The ninety-day deadline gets stricter on its own.** The threshold is still + declared once, in the `JOBS` table in `install-timers.sh`, but it now applies + to each recipient independently. Adding a recipient adds an alert that fires + immediately and keeps firing until that specific copy has been mounted and + tested. That is the intended behaviour and it is also the main ongoing cost: + two copies means two trips to wherever they are kept, four times a year. +- **`ScheduledJobNeverRan` still covers the cold start.** Before any + verification has ever happened, `key-recipients.sh` has not written its file + and there are no per-recipient series to fire on, so the generic rule speaks — + unchanged from today. +- **This does not close the "one person" half of #106.** The title says "one + holder and one copy" and this ADR answers the copy. A second holder is still a + question about who else should be able to open the estate's secrets, which is + a decision about people and is not made here. What changes is that the + mechanism for it now exists and is exercised: adding a second *holder* later is + the same command with someone else's public key. +- **`make render` is the canary if sops changes.** Recipients are written one per + line in a folded scalar, comma-separated, which relies on sops trimming each + entry. That was measured on sops 3.9.4 rather than assumed, and + `add-recipient.sh` re-reads the file after re-keying so a future regression + fails at add time. If it ever regressed silently instead, decryption on the + deployment host is where it would surface. diff --git a/docs/observability.md b/docs/observability.md index b832400..8c0f53a 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -504,7 +504,15 @@ checking, with the key that is on that machine, against the disk that is in it. disk or a fire. The only job that proves off-host recoverability is `secrets-verify-backup`, and it is precisely the one that cannot be automated — it needs a human to mount removable media, so `SecretsKeyBackupUnproven` nags at -ninety days instead. One output does leave: `backup-firewall` copies each export +ninety days instead. That alert is the one rule in `backup.rules.yaml` not keyed +on `homelab_job`: +[ADR-0024](adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +allows the secrets to be encrypted to more than one age recipient, so it fires +per recipient off `homelab_key_recipient_last_proof_timestamp_seconds` rather +than off the job. One timestamp for every copy would mean proving either one +vouched for the other, which is backwards when the whole point of the second +copy is that it fails independently. With a single recipient it behaves exactly +as it always has. One output does leave: `backup-firewall` copies each export to `oracle` and fails if it cannot, so its failure alert doubles as "the config has stopped leaving this host". The volume sets do not leave; that is [#92](https://github.com/Gerrrt/HomeLab/issues/92). diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md index df29b49..5f8c2e9 100644 --- a/docs/runbooks/back-up-the-age-key.md +++ b/docs/runbooks/back-up-the-age-key.md @@ -203,28 +203,96 @@ the live key by device and inode on purpose: what gets tested has to be a copy o removable media, and no timer can mount that. So it is enforced from the other end. `make secrets-verify-backup` records the -timestamp of a successful run, and `SecretsKeyBackupUnproven` fires when that -proof passes ninety days old — routed to the normal alert channel like any other -warning. Until the first verification there is no timestamp at all and -`ScheduledJobNeverRan` says so instead, which is the honest reading of a key -backup nobody has ever tested. +timestamp of a successful run **against the recipient it just proved**, and +`SecretsKeyBackupUnproven` fires when any recipient's proof passes ninety days +old — routed to the normal alert channel like any other warning. Until the first +verification there is no timestamp at all and `ScheduledJobNeverRan` says so +instead, which is the honest reading of a key backup nobody has ever tested. + +The per-recipient part is [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +and it only starts to matter once there is more than one. With one recipient +this is the same alert it always was. With two, one timestamp for both would +mean that proving either copy vouched for the other — which is exactly backwards, +because the point of the second copy is that it fails independently. Nagging is not as good as running it. It is a great deal better than remembering. The threshold lives in the `JOBS` table in [`install-timers.sh`](../../scripts/install-timers.sh); the mechanism is in [`schedule-maintenance.md`](schedule-maintenance.md). +## Adding a second recipient + +The section this replaces said a second recipient was "worth doing on the day +the lab stops being a one-person project, and not before". +[#106](https://github.com/Gerrrt/HomeLab/issues/106) pointed out that this +frames a loss problem as a headcount problem. Nobody joining is what makes a +second *holder* worth having; it has nothing to do with whether the key survives +a disk dying. [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +decides the question on the loss grounds and explains why a second *recipient* +beats a second *copy* — briefly: `.sops.yaml` records a recipient and nothing +records a copy, so the tooling can name and separately nag about a recipient and +cannot tell two copies of one key apart. + +Generate the keypair **on the machine or the medium that will keep it**. Not +here. A key generated on the monitoring host and then added as a recipient is a +second copy on the disk you are insuring, and `make secrets-add-recipient` +refuses it for that reason. + +```bash +# ...somewhere that is not this host: +umask 077 +age-keygen -o /path/to/its/keys.txt +age-keygen -y /path/to/its/keys.txt # the PUBLIC half — this is what travels +``` + +Then, on the monitoring host, with only that public half: + +```bash +make secrets-add-recipient PUBKEY=age1... +``` + +That writes the key into the `creation_rule` this stack's file already uses and +re-keys the file in the same run. sops shows the group change and asks before +committing to it; answering no rolls `.sops.yaml` back, so the two never +disagree. Both files change and belong in one commit: + +```bash +git add .sops.yaml secrets/observability.sops.yaml +``` + +The new recipient starts unproven, and says so: `SecretsKeyBackupUnproven` fires +against it immediately and names it. Clear it the same way as the first — +`make secrets-verify-backup KEY=/path/to/the/new/copy` — and note that this is +the point of the whole exercise. **Proving one recipient does not clear another.** +Each copy has to be mounted and tested on its own, which is four trips a year +rather than two. + +The table in [Copy the key](#1-copy-the-key) and the disqualifiers in +[Confirm the copy is not republishing itself](#3-confirm-the-copy-is-not-republishing-itself) +apply to the second copy as much as the first, plus one rule that only exists +once there are two: **it must not fail +at the same time as the first.** Two USB sticks in the same drawer is one copy +with extra steps, and so is a second recipient whose private half is in the same +password-manager vault as the first — the failure mode of a vault is losing the +master password, and that takes everything in it at once. + ## What this still does not solve -One key, one person, one copy plus the original. If the answer to "who else can -recover this" needs to be more than one, the mechanism already exists: generate a -second keypair that lives only offline, add its public half to `.sops.yaml` as an -additional recipient, and re-key with -`sops updatekeys secrets/observability.sops.yaml` from a host that can already -decrypt. That is the same procedure as bringing a second host in, described in -[`secrets/README.md`](../../secrets/README.md). It is worth doing on the day the -lab stops being a one-person project, and not before — every extra recipient is -another key that can leak. +**One person.** ADR-0024 answers the copy and deliberately does not answer the +holder: whether somebody else should be able to open the estate's secrets is a +question about people, not about mechanism. What has changed is that the +mechanism is now built and exercised — a second holder is the same +`make secrets-add-recipient` with their public key, not a procedure to work out +on the day it matters. + +**Revocation is still rotation.** Removing a recipient and re-keying protects +values encrypted from then on. Every historical ciphertext in git stays readable +by the removed key, because the commits are still there. A second recipient that +leaks means rotating every credential — four SNMP communities on hardware, the +Grafana password, the four notification URLs and the renderer token — exactly as +a leaked first one does. That is why "every extra recipient is another key that +can leak" is still the right thing to weigh; ADR-0024 weighs it and accepts it +against an alternative that carries the same exposure unaudited. ## If something goes wrong diff --git a/scripts/add-recipient.sh b/scripts/add-recipient.sh new file mode 100755 index 0000000..2c0052c --- /dev/null +++ b/scripts/add-recipient.sh @@ -0,0 +1,253 @@ +#!/usr/bin/env bash +# +# Add a second age recipient to a stack's secrets: register its public half in +# .sops.yaml and re-key the encrypted file so the new key can actually open it. +# +# ADR-0024 decides that this repository holds more than one recipient and why. +# This is the mechanism, and it exists rather than "edit .sops.yaml and run sops +# updatekeys" for the two reasons bootstrap.sh already refuses to be a one-liner: +# +# - .sops.yaml has more than one creation_rule, and writing a key into the +# wrong one is silent. ADR-0020 gives `lab` a rule of its own precisely so a +# lab-guest key cannot decrypt the estate's SNMP communities; a hand-edit +# into the general rule undoes that and still looks like it worked. +# - Editing .sops.yaml WITHOUT running `sops updatekeys` produces the worst +# possible state: a recipient this repository advertises as a recovery path +# which cannot decrypt anything. That is not a hypothetical failure, it is +# the default outcome of doing this by hand and forgetting the second step. +# +# ONLY THE PUBLIC HALF EVER REACHES THIS SCRIPT. The private half of the key +# being added must never exist on this host — see the guard in section 2, and +# docs/runbooks/back-up-the-age-key.md for where it should live instead. +# +# Usage: scripts/add-recipient.sh [stack] +# make secrets-add-recipient PUBKEY=age1... + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +PUBKEY="${1:-}" +STACK="${2:-observability}" +SECRETS_FILE="${REPO_ROOT}/secrets/${STACK}.sops.yaml" +SOPS_CONFIG="${REPO_ROOT}/.sops.yaml" +LIVE_KEY="${SOPS_AGE_KEY_FILE:-${HOME}/.config/sops/age/keys.txt}" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } +warn() { printf '\033[0;33m!!\033[0m %s\n' "$*"; } + +if [[ -z "${PUBKEY}" ]]; then + cat >&2 < [stack] + or: make secrets-add-recipient PUBKEY=age1... + +Adds a recipient to secrets/${STACK}.sops.yaml. Public half only — generate the +keypair on the machine that will HOLD it, never here. +Procedure: docs/runbooks/back-up-the-age-key.md +EOF + exit 2 +fi + +command -v sops >/dev/null 2>&1 || die "sops not found. + https://github.com/getsops/sops/releases" + +[[ -f "${SECRETS_FILE}" ]] || die "no encrypted secrets at secrets/${STACK}.sops.yaml +Run 'make secrets-init' first." +[[ -f "${SOPS_CONFIG}" ]] || die "no ${SOPS_CONFIG}" + +# --------------------------------------------------------------------------- +# 1. Is it an age public key at all? +# --------------------------------------------------------------------------- +# Checked before anything is written, because the failure mode of a typo is a +# .sops.yaml that sops refuses to parse — which breaks `make render` and takes +# the stack down at the next converge, for a value nobody can decrypt with +# anyway. +# +# An age X25519 public key is bech32 with the `age1` HRP and is always 62 +# characters. Length is checked rather than left to the regex so that a +# truncated paste is named as truncated. +[[ "${PUBKEY}" =~ ^age1[a-z0-9]+$ ]] \ + || die "not an age public key: ${PUBKEY} +It should start with 'age1' and contain only lowercase letters and digits. +A private key starts with AGE-SECRET-KEY-1 and must never be pasted here." + +((${#PUBKEY} == 62)) \ + || die "'${PUBKEY}' is ${#PUBKEY} characters; an age public key is 62. +Usually a truncated or wrapped paste. Take it from 'age-keygen -y ' on +the machine that holds the private half." + +# --------------------------------------------------------------------------- +# 2. Is it already a recipient? +# --------------------------------------------------------------------------- +mapfile -t CURRENT < <("${REPO_ROOT}/scripts/key-recipients.sh" --list --stack "${STACK}") + +# key-recipients.sh dies with its own message when the file lists none, but it +# dies inside a process substitution, whose exit status this shell never sees. +# Without this the next line would index an empty array and report a bash error +# instead of the real one. +((${#CURRENT[@]})) || die "could not read the recipients of secrets/${STACK}.sops.yaml — see above" + +if printf '%s\n' "${CURRENT[@]}" | grep -qxF "${PUBKEY}"; then + info "secrets/${STACK}.sops.yaml is already encrypted to ${PUBKEY}" + info "nothing to do" + exit 0 +fi + +# --------------------------------------------------------------------------- +# 3. It must not be a key this host already holds +# --------------------------------------------------------------------------- +# The whole value of a second recipient is that its private half is somewhere +# this machine is not. A key generated here and then added here is a second copy +# on the disk being insured, wearing the costume of a second recovery path — and +# it would pass every other check in this script and in verify-key-backup.sh. +# +# ADR-0015 rejects `oracle` as a recipient on the same argument one host over: +# the backups and the means to open them must not share a disk. +if [[ -r "${LIVE_KEY}" ]] && command -v age-keygen >/dev/null 2>&1; then + if age-keygen -y "${LIVE_KEY}" 2>/dev/null | grep -qxF "${PUBKEY}"; then + die "that is this host's own key. + +Its private half is at ${LIVE_KEY}, on the disk these secrets already live on. +Adding it as a second recipient records a recovery path that dies with the +machine it is supposed to survive. + +Generate the keypair on the machine or the medium that will hold it, and bring +only the public half back here: age-keygen -y /path/to/its/keys.txt" + fi +fi + +# --------------------------------------------------------------------------- +# 4. Which creation_rule does this stack's file actually use? +# --------------------------------------------------------------------------- +# Derived from the file rather than by re-implementing sops' first-match-wins +# path_regex resolution. The rule that governs this file is, by definition, the +# one listing the recipients the file is already encrypted to — so the anchor is +# a fact about the ciphertext, not a guess about the policy. +# +# That also makes the ADR-0020 mistake structurally impossible: the lab rule +# does not list the estate's recipient, so `--stack observability` cannot land +# in it, and vice versa. +ANCHOR="${CURRENT[0]}" + +mapfile -t ANCHOR_LINES < <(grep -nE "^[[:space:]]+${ANCHOR},?[[:space:]]*$" "${SOPS_CONFIG}" | cut -d: -f1) + +if ((${#ANCHOR_LINES[@]} == 0)); then + die "secrets/${STACK}.sops.yaml is encrypted to + ${ANCHOR} +but no creation_rule in .sops.yaml lists that key. + +The policy and the ciphertext disagree, and this script cannot tell which one is +right. Fix .sops.yaml by hand — the recipients the file actually uses are: +$(printf ' %s\n' "${CURRENT[@]}")" +fi + +if ((${#ANCHOR_LINES[@]} > 1)); then + die "${ANCHOR} +appears in ${#ANCHOR_LINES[@]} creation_rules in .sops.yaml (lines: ${ANCHOR_LINES[*]}). + +One key covering two rules is the collapse those rules exist to prevent — +bootstrap.sh refuses the same thing. Untangle .sops.yaml by hand first." +fi + +LINE="${ANCHOR_LINES[0]}" +ANCHOR_TEXT="$(sed -n "${LINE}p" "${SOPS_CONFIG}")" + +# The anchor line matched ^[[:space:]]+age1...,?[[:space:]]*$ to get here, so +# these two are total: everything before the key is the indent, and the key +# itself is what the new line has to line up with. +INDENT="${ANCHOR_TEXT%%age1*}" +ANCHOR_TEXT="${ANCHOR_TEXT%"${ANCHOR_TEXT##*[![:space:]]}"}" + +# --------------------------------------------------------------------------- +# 5. Write it in +# --------------------------------------------------------------------------- +# One recipient per line, comma-separated. sops reads `age:` as a single string +# and splits it on commas, trimming each part — so a folded scalar that puts one +# key per line is the same value as one long line, and reviews far better. +# Measured on sops 3.9.4 rather than assumed, because a wrong guess here is a +# .sops.yaml that parses as YAML and decrypts nothing. +# +# The anchor keeps whatever trailing comma it had: if it already ended in one it +# was not the last entry, and the new line needs one too. +if [[ "${ANCHOR_TEXT}" == *, ]]; then + NEW_LINE="${INDENT}${PUBKEY}," + REWRITTEN="${ANCHOR_TEXT}" +else + NEW_LINE="${INDENT}${PUBKEY}" + REWRITTEN="${ANCHOR_TEXT}," +fi + +BACKUP="$(mktemp)" +cp "${SOPS_CONFIG}" "${BACKUP}" +# Restores .sops.yaml on any exit before the re-key is confirmed. A half-applied +# change here is the state described at the top of this file: an advertised +# recovery path that cannot decrypt. +restore() { cp "${BACKUP}" "${SOPS_CONFIG}"; } +trap 'restore; rm -f "${BACKUP}"' EXIT INT TERM + +python3 - "${SOPS_CONFIG}" "${LINE}" "${REWRITTEN}" "${NEW_LINE}" <<'PY' +import sys +path, line, rewritten, new_line = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4] +lines = open(path).read().split("\n") +lines[line - 1] = rewritten +lines.insert(line, new_line) +open(path, "w").write("\n".join(lines)) +PY + +info "added ${PUBKEY} to .sops.yaml" + +# --------------------------------------------------------------------------- +# 6. Re-key, which is the half that does the work +# --------------------------------------------------------------------------- +# Needs the private half of a key that can ALREADY decrypt — normally the live +# key on this host. sops prints the group change and asks; that prompt is worth +# keeping, because it is the last point at which a wrong recipient is cheap. +info "re-keying secrets/${STACK}.sops.yaml" +if ! sops updatekeys "${SECRETS_FILE}"; then + die "sops updatekeys failed — .sops.yaml has been rolled back. + +Nothing changed. The usual cause is that this host cannot decrypt the file, and +re-keying requires a key that can. Run this where the existing key is." +fi + +# --------------------------------------------------------------------------- +# 7. Do not trust the exit status alone +# --------------------------------------------------------------------------- +# The same rule bootstrap.sh and verify-key-backup.sh apply. `updatekeys` exits +# 0 when the user answers no at its prompt, which leaves .sops.yaml advertising +# a recipient the ciphertext has never heard of. +if ! "${REPO_ROOT}/scripts/key-recipients.sh" --list --stack "${STACK}" | grep -qxF "${PUBKEY}"; then + die "sops exited 0 but secrets/${STACK}.sops.yaml is still not encrypted to +${PUBKEY} — answering 'no' at the prompt does exactly this. + +.sops.yaml has been rolled back so the two still agree. Re-run when ready." +fi + +trap - EXIT INT TERM +rm -f "${BACKUP}" + +# The new recipient starts life unproven, and says so out loud rather than +# waiting for the next verification of some OTHER key to notice it exists. +# key-recipients.sh writes 0 for it, SecretsKeyBackupUnproven reads that as +# "never", and the nagging starts now instead of in ninety days. +"${REPO_ROOT}/scripts/key-recipients.sh" --record --stack "${STACK}" || true + +cat <.sops.yaml is fact: the recipients this file is +# encrypted to right now. They differ for exactly as long as it takes somebody +# to add a key and forget `sops updatekeys` — a window in which .sops.yaml +# promises a recovery path that does not exist, and every check reading it +# reports a backup that cannot open anything. +# +# So this reads the file. No decryption is involved: the recipient list is +# plaintext metadata, which is the same reason the key names are. +# +# WHY THE TIMESTAMPS ARE PER RECIPIENT +# +# ADR-0024. `homelab_job_last_success_timestamp_seconds{homelab_job= +# "verify-key-backup"}` is one series, and `make secrets-verify-backup` records +# it whichever key was mounted. With one recipient that is exactly right. With +# two it says "a key was proved" when the question is "was THIS key proved" — +# so proving one resets the ninety-day deadline for both and the other is free +# to rot behind a green alert. One series per recipient is the smallest thing +# that cannot lie about that. +# +# A recipient that has never been verified is recorded as 0 rather than omitted. +# time() - 0 is about 1.8 billion seconds, which exceeds every threshold, so +# "never proved" and "not proved lately" are one alert — the same choice +# run-scheduled.sh makes and for the same reason. +# +# Usage: +# scripts/key-recipients.sh --list [--stack ] +# scripts/key-recipients.sh --record [--stack ] [--proved ] +# +# Environment: +# TEXTFILE_DIR where the .prom files go +# (default /var/lib/node_exporter/textfile_collector) + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +TEXTFILE_DIR="${TEXTFILE_DIR:-/var/lib/node_exporter/textfile_collector}" + +STACK="observability" +MODE="" +PROVED="" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +warn() { printf '\033[0;33m!!\033[0m %s\n' "$*" >&2; } + +usage() { sed -n '/^# Usage:/,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } + +while (($#)); do + case "$1" in + --list) MODE="list"; shift ;; + --record) MODE="record"; shift ;; + --stack) STACK="${2:-}"; shift 2 ;; + --proved) PROVED="${2:-}"; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac +done + +[[ -n "${MODE}" ]] || { usage >&2; die "--list or --record is required"; } + +# Same shape run-scheduled.sh enforces on a job name, for the same reason: this +# becomes a path segment and a Prometheus label value. +[[ "${STACK}" =~ ^[a-z][a-z0-9-]{0,30}$ ]] \ + || die "stack name '${STACK}' must match ^[a-z][a-z0-9-]{0,30}\$" + +SECRETS_FILE="${REPO_ROOT}/secrets/${STACK}.sops.yaml" +[[ -f "${SECRETS_FILE}" ]] || die "no encrypted secrets at secrets/${STACK}.sops.yaml" + +# --------------------------------------------------------------------------- +# The recipients, read out of the file's own metadata +# --------------------------------------------------------------------------- +# grep and not a YAML parser. The `recipient:` keys sit inside the `sops:` block +# and nowhere else in these files — every other value is ciphertext — and a +# bech32 age public key cannot appear in an ENC[...] literal or an armoured age +# blob, both of which are base64. Keeping this dependency-free matters because +# verify-key-backup.sh calls it on a host that has just been rebuilt from bare +# metal, where python3 and PyYAML are not yet a given. +mapfile -t RECIPIENTS < <(grep -oE '^[[:space:]]*(-[[:space:]]+)?recipient:[[:space:]]*age1[a-z0-9]+' "${SECRETS_FILE}" \ + | grep -oE 'age1[a-z0-9]+' | sort -u) + +((${#RECIPIENTS[@]})) \ + || die "secrets/${STACK}.sops.yaml lists no age recipients. +Either it is not SOPS-encrypted, or it is encrypted to a KMS this repository +does not use. Check: grep -A3 '^sops:' secrets/${STACK}.sops.yaml" + +if [[ "${MODE}" == "list" ]]; then + printf '%s\n' "${RECIPIENTS[@]}" + exit 0 +fi + +# --------------------------------------------------------------------------- +# Record +# --------------------------------------------------------------------------- +if [[ -n "${PROVED}" ]]; then + [[ "${PROVED}" =~ ^age1[a-z0-9]+$ ]] || die "--proved is not an age public key: ${PROVED}" + # Refused rather than silently ignored. Reaching here means a key decrypted + # the file while not being one of the recipients the file records, which is + # not a thing that can happen — so it is a bug in the caller, and writing a + # proof row for a recipient that does not exist would age out and fire an + # alert naming a key nobody can find. + printf '%s\n' "${RECIPIENTS[@]}" | grep -qxF "${PROVED}" \ + || die "${PROVED} is not a recipient of secrets/${STACK}.sops.yaml" +fi + +# Same two failure modes run-scheduled.sh distinguishes, and the same answers: +# a missing directory is a human on a workstation and must not break the +# command; an unwritable one is a broken monitoring host and must not be +# reported as a recorded outcome. +if [[ ! -d "${TEXTFILE_DIR}" ]]; then + warn "no textfile directory at ${TEXTFILE_DIR} — not recording which recipients are proved" + exit 0 +elif [[ ! -w "${TEXTFILE_DIR}" ]]; then + die "${TEXTFILE_DIR} is not writable by $(id -un). +Fix the directory, then re-run: + sudo install -d -m 0755 -o $(id -un) -g $(id -gn) ${TEXTFILE_DIR}" +fi + +PROM="${TEXTFILE_DIR}/key-recipients.prom" +NOW="$(date +%s)" + +# Carry forward what is already there, exactly as run-scheduled.sh carries +# forward prior_success: this file is rewritten in full on every run, so a +# recipient's proof would otherwise be lost the moment a different one was +# verified — which is the failure this whole file exists to prevent. +prior_for() { + local want="$1" + [[ -r "${PROM}" ]] || { printf '0'; return; } + awk -v want="${want}" ' + $0 ~ /^homelab_key_recipient_last_proof_timestamp_seconds\{/ { + if (index($0, "recipient=\"" want "\"")) value = $NF + } + END { print (value ~ /^[0-9]+$/) ? value : "0" } + ' "${PROM}" +} + +# Every recipient of every stack shares one file, because node_exporter merges +# the directory and a metric name may carry only one HELP string across it. A +# second stack writing its own file would collide on that, not on the series. +# So rows for other stacks are carried through untouched. +others="" +if [[ -r "${PROM}" ]]; then + others="$(grep -F 'homelab_key_recipient_last_proof_timestamp_seconds{' "${PROM}" \ + | grep -vF "stack=\"${STACK}\"" || true)" +fi + +tmp="${PROM}.$$" +{ + printf '# HELP homelab_key_recipient_last_proof_timestamp_seconds Unix time an offline copy of this age recipient private key was last proved to decrypt the stack secrets. 0 means never.\n' + printf '# TYPE homelab_key_recipient_last_proof_timestamp_seconds gauge\n' + [[ -n "${others}" ]] && printf '%s\n' "${others}" + for recipient in "${RECIPIENTS[@]}"; do + if [[ "${recipient}" == "${PROVED}" ]]; then + ts="${NOW}" + else + ts="$(prior_for "${recipient}")" + fi + printf 'homelab_key_recipient_last_proof_timestamp_seconds{stack="%s",recipient="%s"} %s\n' \ + "${STACK}" "${recipient}" "${ts}" + done +} > "${tmp}" + +# 0644 explicitly and rename to publish, for the two reasons run-scheduled.sh +# gives: a 0600 .prom is invisible to the collector, and a truncate-in-place +# exposes a half-written file to a scrape. +chmod 0644 "${tmp}" +mv -f "${tmp}" "${PROM}" diff --git a/scripts/verify-key-backup.sh b/scripts/verify-key-backup.sh index 2fb86bb..5b15f22 100755 --- a/scripts/verify-key-backup.sh +++ b/scripts/verify-key-backup.sh @@ -152,10 +152,21 @@ An age key file holds a line beginning AGE-SECRET-KEY-1. If you transcribed this from paper, check for a truncated or wrapped line." fi +# Matched against the RECIPIENTS OF THE FILE, not against .sops.yaml. +# +# .sops.yaml is the policy for the next encryption; the `sops:` block inside the +# encrypted file is the list of keys that can open the bytes on disk. They +# diverge for exactly as long as it takes somebody to add a recipient and forget +# `sops updatekeys` — and in that window this check used to pass on a key that +# then failed at section 4 with "the secret half is damaged or truncated", +# sending you to re-copy a backup that was never the problem. +mapfile -t RECIPIENTS < <("${REPO_ROOT}/scripts/key-recipients.sh" --list --stack "${STACK}") +((${#RECIPIENTS[@]})) || die "could not read the recipients of secrets/${STACK}.sops.yaml — see above" + matched="" while IFS= read -r pub; do [[ -n "${pub}" ]] || continue - if grep -qF "${pub}" "${SOPS_CONFIG}"; then + if printf '%s\n' "${RECIPIENTS[@]}" | grep -qxF "${pub}"; then matched="${pub}" break fi @@ -164,14 +175,18 @@ done <<< "${PUBLIC_KEYS}" if [[ -z "${matched}" ]]; then die "this is a valid age key, but not one the secrets are encrypted to. - backup holds: $(printf '%s' "${PUBLIC_KEYS}" | tr '\n' ' ') - .sops.yaml wants: $(grep -oE 'age1[a-z0-9]+' "${SOPS_CONFIG}" | tr '\n' ' ') + backup holds: $(printf '%s' "${PUBLIC_KEYS}" | tr '\n' ' ') + the file is encrypted to: $(printf '%s ' "${RECIPIENTS[@]}") + .sops.yaml lists: $(grep -oE 'age1[a-z0-9]+' "${SOPS_CONFIG}" | tr '\n' ' ') + +A freshly generated keypair looks exactly like this. If the first two lines +agree and the third does not, .sops.yaml has drifted from the ciphertext. -A freshly generated keypair looks exactly like this. If you meant to add a new -recipient rather than restore an old one, that is 'sops updatekeys' — see -secrets/README.md." +If you meant to ADD this key as a recipient rather than restore an old one: + make secrets-add-recipient PUBKEY=$(printf '%s' "${PUBLIC_KEYS}" | head -n1)" fi -info "recipient matches .sops.yaml: ${matched}" +info "recipient ${matched}" +info "this file has ${#RECIPIENTS[@]} recipient(s); this run proves one of them" # --------------------------------------------------------------------------- # 4. The real test: decrypt with nothing but the backup available @@ -253,8 +268,35 @@ if [[ -n "${mode}" && "${mode: -2}" != "00" ]]; then warn "mode ${mode} — group or other can read this copy: chmod 600 $(printf '%q' "${BACKUP_ABS}")" fi +# --------------------------------------------------------------------------- +# 7. Record WHICH recipient was proved +# --------------------------------------------------------------------------- +# ADR-0024. run-scheduled.sh wraps this call and records one timestamp under +# homelab_job="verify-key-backup" whichever key was mounted, so with more than +# one recipient it answers "a key was proved" to the question "was THIS key +# proved" — and proving one silences the ninety-day deadline for all of them. +# key-recipients.sh keeps a series per recipient, which is what +# SecretsKeyBackupUnproven actually reads. +# +# Before the ok line and allowed to be fatal, deliberately. An unrecorded proof +# is the failure #77 is about: the job ran, nothing can see that it ran, and the +# gap is invisible until the day it matters. +"${REPO_ROOT}/scripts/key-recipients.sh" --record --stack "${STACK}" --proved "${matched}" + printf '\033[0;32mok\033[0m — %s decrypts secrets/%s.sops.yaml (%d/%d keys)\n' \ "$(basename "${BACKUP}")" "${STACK}" "${found}" "${#REQUIRED[@]}" printf ' Proven: this key, on its own, recovers every secret in the repo.\n' + +# The line the single-series metric could never say. Proving one recipient used +# to read as proving the backup; with more than one it proves exactly one of +# them, and the other copies are still only as good as the last time somebody +# went and got them out. +if ((${#RECIPIENTS[@]} > 1)); then + printf ' Not proven: the other %d recipient(s) of this file. Each needs its own\n' \ + "$((${#RECIPIENTS[@]} - 1))" + printf ' run against its own copy — SecretsKeyBackupUnproven names any that go\n' + printf ' ninety days without one.\n' +fi + printf ' Not proven: that where you keep it will still exist after a fire,\n' printf ' a theft, or a forgotten password. That part is your judgement.\n' diff --git a/secrets/README.md b/secrets/README.md index 55f2010..125bb53 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -17,10 +17,24 @@ decrypted only in memory at deploy time. > > On a host that holds none of those private keys, `make render` fails at > decryption rather than starting the stack with defaults. To bring a second -> host in, run `make secrets-init` there to generate its keypair, add its public -> half to `.sops.yaml` as an additional recipient, and re-key the file with -> `sops updatekeys secrets/observability.sops.yaml` from a host that can already -> decrypt it. `secrets-init` refuses to overwrite an existing encrypted file. +> host in, run `make secrets-init` there to generate its keypair, bring back the +> public half it prints, and add it here: +> +> ```bash +> make secrets-add-recipient PUBKEY=age1... +> ``` +> +> That writes the key into the `creation_rule` this stack's file already uses — +> resolved from the file's own recipients, so it cannot land in another stack's +> rule — and re-keys in the same run. `secrets-init` refuses to overwrite an +> existing encrypted file, and `secrets-add-recipient` refuses a key whose +> private half is already on this host. +> +> A recipient that is a *backup* rather than a host follows the same procedure +> and one extra rule: generate it where it will live, never here. +> [ADR-0024](../docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +> and [`back-up-the-age-key.md`](../docs/runbooks/back-up-the-age-key.md) cover +> why the estate keeps more than one and how each is proved separately. ## Why encrypted-in-git rather than a gitignored `.env` diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index a0ec92c..6e00c30 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -174,23 +174,44 @@ groups: # a real off-host copy. That means a human has to mount removable media, # and no timer can do it. What a timer CAN do is notice that nobody has. # - # The threshold comes from the same declaration series as everything - # else, so the ninety days lives in scripts/install-timers.sh and nowhere - # else. Until the first verification there is no last_success series at - # all and ScheduledJobNeverRan is what speaks instead. + # ONE SERIES PER RECIPIENT, and this is the only rule in the file that + # is not keyed on homelab_job. ADR-0024 added a second age recipient, and + # the moment there were two, homelab_job_last_success_timestamp_seconds + # {homelab_job="verify-key-backup"} became a wrong answer: it is a single + # series that run-scheduled.sh writes whichever key was mounted, so + # proving one copy reset the ninety-day deadline for every copy and the + # other could rot behind a green alert. scripts/key-recipients.sh emits + # homelab_key_recipient_last_proof_timestamp_seconds per recipient, read + # out of the encrypted file's own metadata, and this fires per recipient. + # + # The threshold still comes from the same declaration series as + # everything else, so the ninety days lives in scripts/install-timers.sh + # and nowhere else. It is joined on(instance) rather than on(homelab_job) + # because the left side has no homelab_job label to join on — both series + # are written into the same textfile directory on the same host. + # group_left() for the reason ScheduledJobStale gives above: without it + # the result carries only the labels named in on(), and `recipient` — the + # entire point of this rule — would be silently dropped. + # + # A recipient that has never been verified is recorded as 0, not omitted, + # so it fires here with an absurd duration rather than being invisible. + # ScheduledJobNeverRan still speaks for the case before ANY verification + # has happened, because key-recipients.sh has not written its file yet + # and there are no per-recipient series to fire on. expr: | - (time() - homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup"}) - > on(homelab_job) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + (time() - homelab_key_recipient_last_proof_timestamp_seconds) + > on(instance) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} for: 1h labels: component: backup severity: warning category: correctness annotations: - summary: "The age key backup has not been verified in {{ $value | humanizeDuration }}" + summary: "No proven backup of age recipient {{ $labels.recipient }} in {{ $value | humanizeDuration }}" description: >- - Nothing else here proves the secrets are recoverable from anywhere - but this host — every other job runs on the same machine, with the - same key, against the same disk. Get the offline copy out and run - `make secrets-verify-backup KEY=`: + Nothing else here proves the {{ $labels.stack }} secrets are + recoverable from anywhere but this host — every other job runs on the + same machine, with the same key, against the same disk. Proving a + DIFFERENT recipient does not clear this one. Get that copy out and + run `make secrets-verify-backup KEY=`: docs/runbooks/back-up-the-age-key.md. diff --git a/stacks/observability/prometheus/tests/backup.test.yaml b/stacks/observability/prometheus/tests/backup.test.yaml index 0c2011d..db55774 100644 --- a/stacks/observability/prometheus/tests/backup.test.yaml +++ b/stacks/observability/prometheus/tests/backup.test.yaml @@ -89,24 +89,55 @@ tests: alertname: ScheduledJobStale exp_alerts: [] + # And SecretsKeyBackupUnproven does not fire off the JOB series either. + # It reads the per-recipient series and nothing else, so a host where + # verify-key-backup has run but key-recipients.sh has not written its file + # is covered by ScheduledJobNeverRan, not by a rule that would have to + # invent a recipient label it does not have. + - eval_time: 2h30m + alertname: SecretsKeyBackupUnproven + exp_alerts: [] + + # --- SecretsKeyBackupUnproven: one alert per recipient -------------------- + # + # The assertion ADR-0024 turns on. Two recipients, one of them verified + # 30 minutes ago and one never — the shape the estate is in the moment a + # second key is added. Exactly one alert, naming the unproven key. + # + # Under the single-series rule this replaced, the fresh proof of `age1aaa` + # was the only series there was, so this state was silent. That is the whole + # defect: proving one copy vouched for every copy. + - interval: 1m + input_series: + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="observability", + recipient="age1aaa", instance="prometheus"}' + values: "9000+0x180" + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="observability", + recipient="age1bbb", instance="prometheus"}' + values: "0+0x180" + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup", instance="prometheus"}' + values: "3600+0x180" + alert_rule_test: - eval_time: 2h30m alertname: SecretsKeyBackupUnproven exp_alerts: - exp_labels: alertname: SecretsKeyBackupUnproven - homelab_job: verify-key-backup + stack: observability + recipient: age1bbb instance: prometheus component: backup severity: warning category: correctness exp_annotations: - summary: "The age key backup has not been verified in 2h 30m 0s" + summary: "No proven backup of age recipient age1bbb in 2h 30m 0s" description: >- - Nothing else here proves the secrets are recoverable from - anywhere but this host — every other job runs on the same - machine, with the same key, against the same disk. Get the - offline copy out and run `make secrets-verify-backup - KEY=`: docs/runbooks/back-up-the-age-key.md. + Nothing else here proves the observability secrets are + recoverable from anywhere but this host — every other job runs + on the same machine, with the same key, against the same disk. + Proving a DIFFERENT recipient does not clear this one. Get that + copy out and run `make secrets-verify-backup KEY=`: + docs/runbooks/back-up-the-age-key.md. # --- ScheduledJobFailed: fires on non-zero, silent on zero --------------- - interval: 1m From 5eecc0a268f68bc6eb0a30127c2a7da4feeb2b90 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 4 Sep 2026 22:19:23 +0000 Subject: [PATCH 2/2] docs(runbook): the handover is where the second recipient goes (#106) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #124's successor-handover.md landed while this branch was open and says two things this change makes stale: that the mechanism for a second recipient costs nothing, and that SecretsKeyBackupUnproven tracks one verification. Neither survives ADR-0024 — the mechanism is a command now, it is not free, and the deadline is per recipient. The section is also the best place in the repository for that command. ADR-0024 answers the copy and deliberately leaves the holder open, and a successor is precisely the holder question: the key gets generated where they will keep it, and only the public half comes back here. "If you do not have the age key" now starts by checking whether it is actually gone, because after ADR-0024 .sops.yaml may list a recipient somebody else holds — which turns a four-device rotation into a phone call, and is worth ruling out before anyone starts rotating. Co-Authored-By: Claude Opus 5 --- docs/runbooks/successor-handover.md | 41 ++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/docs/runbooks/successor-handover.md b/docs/runbooks/successor-handover.md index a618308..eb3ae0a 100644 --- a/docs/runbooks/successor-handover.md +++ b/docs/runbooks/successor-handover.md @@ -193,7 +193,7 @@ inherits without knowing. | --- | --- | --- | | **Alert delivery to a destination you do not own** | Immediately, and silently | You do not. This is step 2 above, and it is the reason it is step 2 | | **The external heartbeat watcher** — a free-tier cron-monitor on somebody else's account | Whenever that account lapses | Nothing here can tell you. A watcher on this host would fail with the thing it watches, which is why it is off-host and therefore outside anything this repository can check | -| **The age key backup goes unproven** | 90 days after the last verification | `SecretsKeyBackupUnproven`, routed to the normal alert channel. Until the first verification there is no timestamp at all and `ScheduledJobNeverRan` says so instead | +| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). Until the first verification there is no timestamp at all and `ScheduledJobNeverRan` says so instead | | **Grafana's leaf certificate** | 825 days from issue; the APC card's own certificate expires on its own clock | `TlsCertificateExpiringSoon` at 30 days, `TlsCertificateExpiryImminent` at 7 — read off the served handshake by `blackbox-exporter`, not off a file. Let it lapse and `up{job="grafana"}` goes to 0 as well | | **The UPS battery pack** | A pack was fitted 2026-08-28 and passed its self-test; packs are consumables and this one is on a biweekly test schedule | `UpsSelfTestFailed` and `UpsBatteryUnproven` key on the self-test result, which is the single honest signal this card emits — every charge, runtime and alarm value it reports was fabricated while the bay was empty. Two things remain open: the card's test *schedule* is unwatched ([#249](https://github.com/Gerrrt/HomeLab/issues/249)), and `upsBasicBatteryLastReplaceDate` still reads a pre-fit date, so it is not a usable record of the pack's age | | **Mains power to the monitoring path** | Any cut | The rack is on the UPS; the switch carrying `prometheus` and `oracle` is not, so both laptops keep running and go deaf. Stated in [`security.md`](../security.md#threat-model) | @@ -243,15 +243,36 @@ outgoing operator directly: ### If you do not have the age key Say so out loud before anything else. This is -[#106](https://github.com/Gerrrt/HomeLab/issues/106) — one key, one holder, one -copy plus the original — and it is open. Without it every encrypted value in -this repository and its history is permanently undecryptable, and the recovery -path is re-deriving each credential from the device it belongs to: four SNMP -rotations on hardware, one of which cannot persist a community deletion and -needs the switch rebooted to change. The mechanism for a second recipient costs -nothing (`.sops.yaml` takes a list, and `sops updatekeys` re-keys the file from -any host that can already decrypt); what it has always needed is somewhere to -put the second key. A handover is that somewhere. +[#106](https://github.com/Gerrrt/HomeLab/issues/106). Without the key every +encrypted value in this repository and its history is permanently undecryptable, +and the recovery path is re-deriving each credential from the device it belongs +to: four SNMP rotations on hardware, one of which cannot persist a community +deletion and needs the switch rebooted to change. + +Check first whether it is actually gone. `.sops.yaml` may list more than one +recipient — [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +made a second one the design, and a second *copy* may exist that is not the one +you were handed: + +```bash +grep -A3 creation_rules .sops.yaml # every key that can open the secrets +``` + +If a recipient there is one somebody else holds, the secrets are recoverable and +this is a phone call rather than a rotation. + +**A handover is the moment to add one.** ADR-0024 answers the copy and +deliberately leaves the holder open, which is exactly the question a successor +is. Generate the keypair where *they* will keep it, and add its public half from +a host that can already decrypt: + +```bash +make secrets-add-recipient PUBKEY=age1... +``` + +That is not free — every extra recipient is another key that can leak, and +removing one later protects future values only, because the history stays +readable by it. ADR-0024 weighs both. ### If you do have it