Skip to content

v0.9 — The full PKI migration

Choose a tag to compare

@sethbergman sethbergman released this 25 Aug 20:20
· 78 commits to main since this release
ee2b3ca

issue-node-cert.sh swaps one node's certificate. Doing that to a cluster
is not a loop around it.

Nodes verify their peers with tls_client_ca_file, so a node whose bundle
lacks the CA that signed its peer's certificate refuses that peer — which
presents as a network fault and gets diagnosed as one. There is exactly
one safe order, and scripts/migrate-to-vault-pki.sh drives it:

Phase What happens
trust every node's bundle gains the PKI CA, keeping the bootstrap CA. Nothing swaps.
swap one node at a time moves onto a PKI certificate.
prune once every node is on PKI, the bootstrap CA comes out.

Guarded, not documented

Running prune early partitions the cluster, so the script refuses. It
reads what each node actually serves over TLS, not what is on its
disk — a certificate installed but never reloaded is not migrated, and
that is exactly the case that would otherwise slip through.

Standbys swap before the active node. Not to avoid an election; a swap
costs no leadership. It is about what is still true if the run dies
halfway: the leader is the node you least want in an unknown state, so it
goes last, once the procedure has already worked twice.

Every node is health-gated between steps and the voter count must still
match — continuing past a node that did not come back is how one bad
certificate becomes an outage.

Tested

tests/pki-migration covers the refusals, since that is where the damage
is. tests/integration runs the whole rollout against the real
three-node cluster, including the prune that had previously only ever met
a shim, then checks every node serves a PKI issuer read off the wire, the
bootstrap subject is gone from the bundle, three voters remain, and a
re-run no-ops.

Also in this release

Two bugs the tests found rather than review. A trust-bundle prune
invalidates every other copy of that bundle — Prometheus and the blackbox
exporter both mount it, and until restarted they validate against a CA
that signs nothing, so the TLS probe goes dark and certificate expiry
becomes unmonitored with no error anywhere. And a shim that hardcoded one
openssl's DN rendering, which matched the machine it was written on and
nothing else.

See docs/security.md.