v0.14 — Five cloud defects found and fixed without an apply

Neither cloud profile has been applied. That has not changed.

What changed is how much of an apply's job can be done before paying for
one — and the answer turned out to be five real defects, none of which
the twenty-seven checks running on every pull request could see.

What was broken

Nobody could have reached the cluster. The node security group
admitted port 8200 from the load balancer's security group. But the target
group uses target_type = "instance", where client IP preservation is
enabled and cannot be turned off — so the load balancer forwards each
packet with the client's address as its source. The reference matched
the health checks, which come from the load balancer's own interface, and
nothing else. Every target healthy, every client dropped.

No snapshot could have uploaded. The bucket sets SSE-KMS default
encryption with the auto-unseal key, and S3 has the caller mint the data
key. Nothing granted kms:GenerateDataKey. The bucket, the lifecycle rule
and the s3:PutObject grant were all correct, and every hourly upload
would have been denied by KMS rather than by S3.

No peer could have joined. retry_join verifies the leader against
leader_tls_servername, one fixed name whichever node holds leadership.
The templates asked for vault.<cluster>.internal; the PKI role issued
<host>.vault.internal. That name existed in two files in the repository
and no certificate anywhere carried it.

A bare apply never terminated. The autoscaling group used ELB health
while the profile deliberately does not issue certificates. Vault could
not start, the check could not pass, and every instance was terminated at
the end of its grace period and replaced by one that repeated it — billing
EC2, three NAT gateways and EBS throughout, and looking like a slow
bootstrap.

Teardown stranded the backups. The root volume was encrypted with the
seal key, so terraform destroy scheduled the one key every archived
snapshot depended on. storage.tf warned in plain English that the key
must survive a teardown; the code deleted it.

What now stops the class recurring

tests/preflight-static — the agreements no single layer can see.
Every value a template reads is one Terraform passes and vice versa; the
rendered cloud-init parses and is shellchecked (it was linted nowhere
before); every auto_join key is one go-discover accepts in a combination
it accepts; the leader name is one the PKI role can actually issue. No
credentials, nothing created, seconds to run.

tests/cloud-apply-emulated — a real terraform apply, through the
real AWS provider, against an implementation of the AWS API. It settles
what mocked providers structurally cannot: whether the configuration
applies at all in one pass, whether the AMI filter matches anything,
whether any value is refused, and whether destroy takes it back down.

The thing worth taking away

Three of the five defects were held in place by passing assertions — a
test requiring the exact value that made the apply loop, one forbidding
the rule the cluster needed to serve traffic, and one comparing arns a
mock had flattened to a single value. Fixing each required deleting
something green.

That is written up at
I had to delete a passing test to fix the bug.

Also in this release

The Makefile now does what CI does — make lint could previously pass
while CI failed — with suite discovery and a check-ci-coverage target
that fails when a suite exists that no CI job runs.
bootstrap-dev-cluster.sh checks its tools before generating anything.
Provider lock files cover five platforms, so an ARM checkout stays clean.

Also, found the same way

A shell script checked out with CRLF does not run — its interpreter
becomes bash , and the failure surfaces as exit 127 naming no file
and no reason. Git for Windows ships core.autocrlf=true in its system
config, so that is the default for a Windows contributor, and CI cannot
see it because the runners check out on Linux.

make test-all failed on two scripts and passed on fifty-eight, because
only files Git actually rewrites get converted. .gitattributes now pins
LF and the tree is normalised.

Still requires an account

The KMS triangle against a real key policy. A terminated leader replaced
unattended. Real health checks keeping standbys in the pool. Azure's
scale-set discovery against real instances. Everything reachable from the
files has now been reached.

See docs/cloud-apply.md.

v0.13 — Alert routing by severity, and MySQL as a second database engine

Alertmanager was configured with a single catch-all route and a receiver
declared as webhook_configs: [] — one that parses, validates, and
delivers nowhere. The database engine spoke only PostgreSQL. Both are
addressed here, and the same release found three assertions that passed
while the thing they described was broken.

Alert routing

There are still no vendor receivers, deliberately. A config full of fake
PagerDuty keys proves nothing a reader could reuse. What transfers is
everything above the vendor:

	Critical	Warning
Receiver	page	ticket
group_wait	0s	30s
repeat_interval	15m	12h

group_wait: 0s is the point of the split. At the default a page waits
to see whether a second alert joins its group — a fine trade for a
ticket, a bad one for a cluster that has lost quorum. And a critical that
notifies once then goes quiet is indistinguishable from one nobody sent.

Grouping is by alertname and cluster, deliberately not by
instance: three nodes sealing at once is one incident, and grouping per
instance would page three times for it.

Two inhibit rules, both encoding the same judgement — when one fact
explains another, only the explaining fact is worth waking someone for.
No scrape targets at all explains every node being down; lost quorum
explains there being no active node.

Proving delivery needed a receiving end. Alertmanager's API can say
an alert arrived; it cannot say which receiver it reached, and a
misrouted alert looks exactly like a correctly routed one. Each receiver
posts to a sink that records the delivery, and the integration suite
asserts a critical alert reached the pager path and was not also filed
as a ticket.

Which route matches a label set is put to amtool from the pinned
image rather than modelled in the test, because a test that reimplements
the thing it is testing agrees with itself and nothing else.

The check worth knowing about: every alert in the rule file must carry a
severity that has a route of its own. One with a misspelled severity
matches no route, falls to the catch-all, and is never paged for —
nothing else here would notice. The suite asserts that failure mode
directly by asking amtool where severity=crticial goes.

MySQL, and where it is weaker

Same interface, same mount, one script. Only the plugin, the connection
string and the SQL differ.

The reason to add a second engine was never breadth. It was to find out
where "Vault's database engine supports X" hides something, and it does:

	Postgres	MySQL
Credential expires because	the database enforces it, and Vault revokes it	Vault revokes it
Vault down at lease expiry	still dies on schedule	stays live
readwrite can create tables	yes	no
Username limit	63 bytes	32 characters

MySQL has no VALID UNTIL. CREATE USER takes no deadline, so a
credential issued against it lives until Vault revokes it, where a
Postgres credential dies on schedule even if Vault is unreachable at
lease end. The Postgres path degrades safely; the MySQL path depends on
Vault being there at the right moment.

That is asserted in both directions — one test checks the MySQL
statements do not contain VALID UNTIL, its neighbour checks the
Postgres ones still do — because the tempting fix is to paste it in so
both engines look alike, and it is a syntax error there.

The readwrite difference is not quietly fixable either. Postgres ties
DDL to ownership: a credential can alter what it created and nothing
else. MySQL grants per schema, so the same privilege would let every
readwrite credential drop the whole database.

Vault connects as a dedicated vaultadmin account rather than root.
Issuing credentials needs CREATE USER and GRANT OPTION, and root is
the obvious shortcut — but the bootstrap rotates the password of whatever
account it connects as, and root's password is what the container
healthcheck uses. That would leave the container permanently unhealthy
while MySQL itself was fine.

The integration suite runs all of it against a real MySQL server: the
generated username fits the 32-character cap (it comes out at exactly
32), the credential connects, readonly cannot write or read mysql.user,
revoking the lease drops the account from mysql.user, and the bootstrap
password stops working after rotation.

Two cleanup handlers that turned success into failure

A cleanup function whose last command is a false test returns 1, and bash
applies that to the script's exit status from an EXIT trap:

cleanup() { [[ -n "$D" && -d "$D" ]] && rm -rf "$D"; }
trap cleanup EXIT
exit 0            # exits 1 when $D is empty

Both trap handlers here had that shape. Probed directly, vault-upgrade.sh
and oidc-login-test.sh exited 1 on an early success path.

Latent rather than live — nothing currently exits 0 before the temporary
directory is created. It goes live the moment someone adds "already at
the target version, nothing to do", which is a natural addition to an
upgrade script and impossible to connect to a cleanup function three
screens away.

Worth being precise about, because the shape is idiomatic elsewhere.
Under set -e a trailing && whose test is false does not exit at
the top level, and does not exit as the last statement of an if/else
branch. It is specifically the function-return path, and the trap that
consumes it.

What this release learned about its own tests

docs/roadmap.md rests on one claim: shipped means there is a test that
fails if the feature breaks. Three assertions did not.

An exclusion assertion is only as good as the spelling its author thought
to forbid. "aws s3 cp" reads like it forbids uploading — an upload via
aws s3api put-object passed it, demonstrated by making a standby upload
and watching the guard stay green. Two more of the same shape: a token
revoked through auth/token/revoke-self passed "a supplied VAULT_TOKEN
is left alone", and a blob deleted with curl --request DELETE passed
"no blob DELETE".

The mutation testing meant to catch this had the same flaw one level up.
An assertion rejecting CREATE, ALTER, DROP, "verified" by a mutation
granting exactly CREATE, ALTER, DROP, proves only that grep works;
granting CREATE alone walked straight through. A useful mutation is one
the assertion does not name.

Both habits are now in CONTRIBUTING.md, and tests/lint enforces in CI
the invariants shellcheck has no opinion about.

None of this changes what the shipped table claims. It changes how much
the word "tested" in it is worth.

Still not covered

• Neither cloud profile has been applied. Unchanged, and still the
  first v1.0 blocker. docs/cloud-apply.md lists what a real apply would
  settle.
• No vendor integration. Whether your PagerDuty key is correct is not
  something this repository can tell you.
• The cloud profiles have no monitoring stack at all, so every
  alerting guarantee here applies to the local profile only.
• Inhibition is checked structurally, not by arranging two alerts to
  overlap on a live cluster.

v0.12 — Tamper-evident audit

v0.11 made the audit trail outlive the node it describes. It could still
be edited in place, and nothing would say so.

The collector appended bytes, so anything able to write to the volume
could remove the record of what it did — and a shorter log is
indistinguishable from a quieter day.

The chain

Every entry now also produces a link in a parallel chain file:

<seq> <sha256(entry)> <sha256(previous_chain_hash + entry_hash)>

Each link covers the one before it, so removing, altering or inserting an
entry breaks everything after it. scripts/verify-audit-chain.sh
recomputes the chain from the entries, not from the chain — reading
the chain and checking it is self-consistent would pass on any internally
consistent forgery — and reports the first sequence that diverges.

It separates the verdicts, because they call for different responses:

Verdict	What happened
chain mismatch at N	an entry at or before N was altered, removed or inserted
log longer than chain	the tail was never chained; a crash does this, so does appending by hand
chain longer than log	entries were deleted, and the chain still remembers them
anchor mismatch	the chain itself was rewritten

The log is written exactly as before — raw entries, one per line, no added
fields — so anything already consuming it is unaffected.

Why the chain alone would not have been enough

A hash chain catches whoever cannot recompute it and does nothing against
whoever can. Delete the entries recording what you did, recompute from
that point, and the result verifies perfectly: internally consistent and
completely false.

So audit-anchor runs as its own service, recording the chain head to
its own volume with the audit volume mounted read-only. Neither
container can write where the other reads. A head recorded for sequence N
still remembers what the trail said, and any later rewrite at or before N
disagrees with it.

The test suite asserts this directly rather than describing it: it builds
a trail, anchors it, deletes an entry, recomputes the whole chain, and
checks the result passes verification without anchors and fails
with them. The passing half is the point — it is the evidence that
anchoring is load-bearing rather than decoration.

What this is not

One host. Both volumes live on the same Docker daemon, so a sufficiently
privileged compromise reaches the anchors too. This is tamper evidence,
not tamper proofing, and only the first is claimed.

A production anchor belongs where the Vault host cannot write at all —
object-lock storage, a different account, or an external timestamping
service. The anchor format is three fields of text precisely so that
shipping it elsewhere is not a redesign.

Off-host audit collection remains a v1.0 blocker.

See docs/audit.md.

v0.11 — An audit trail that outlives the node

An audit log that exists only on the node it describes is one a
compromise can delete. Anything able to destroy the Vault node is able to
destroy the evidence of what it did there.

v0.8 enabled two audit devices and argued for putting the secondary on a
different failure domain. This is that argument carried through: the
socket device now points at a collector container writing to a Docker
named volume with a lifecycle independent of every container.

Proven by destruction, not by inspection

The integration suite destroys the vault-0 container outright and reads
the trail back afterwards. That is the property worth having, and reading
a file while everything is running would not have demonstrated it.

The pairing also stays deliberate. A socket device alone can block
Vault when its endpoint goes away; paired with a file device it cannot,
because the file keeps satisfying the at-least-one guarantee. The suite
stops the collector and shows Vault still accepting writes, still
healthy, still recording to the survivor.

What it is not

Not off-host. A Docker named volume outlives the container and does not
leave the machine — anything with access to the Docker daemon can still
reach it.

docs/audit.md states this plainly and describes what a real deployment
should point the socket at instead, along with the properties worth
insisting on: a different host, append-only or object-locked storage, and
credentials that are not Vault's own. The last two are the ones people
skip, and shipping a log to a place the same attacker can edit is a
change of address rather than of risk.

See docs/audit.md.

v0.10 — Vault Agent: a secret without a token

Everything before this showed Vault holding and issuing secrets. Nothing
showed an application actually consuming one.

That gap matters, because the usual answer — give the app a Vault token
and let it call the API — moves the problem rather than solving it. The
token is now a long-lived credential sitting in the application's
environment, which is the thing Vault was adopted to get rid of.

What this adds

A vault-agent service that deliberately has no VAULT_TOKEN. It
authenticates with an AppRole whose credentials are placed on disk by
bootstrap-agent.sh, and renders a dynamic Postgres credential into
/rendered/db.env.

The application reads a file. It never learns Vault exists, holds no
token, and needs no Vault client library.

Agent handles the parts that are tedious to get right by hand: renewing
the lease before it expires, re-authenticating when the token cannot be
renewed further, and re-rendering the template when the credential
changes.

Also in this release

Several fixes from running it for real rather than reading the docs —
starting Agent the way its base image expects, giving it somewhere it can
actually write, giving a restarted node its logs, and a curl fallback
that printed 000000 on connection failure because curl already prints
000 itself and the || echo 000 appended a second one.

See docs/vault-agent.md.

v0.9 — The full PKI migration

issue-node-cert.sh swaps one node's certificate. Doing that to a cluster
is not a loop around it.

Nodes verify their peers with tls_client_ca_file, so a node whose bundle
lacks the CA that signed its peer's certificate refuses that peer — which
presents as a network fault and gets diagnosed as one. There is exactly
one safe order, and scripts/migrate-to-vault-pki.sh drives it:

Phase	What happens
trust	every node's bundle gains the PKI CA, keeping the bootstrap CA. Nothing swaps.
swap	one node at a time moves onto a PKI certificate.
prune	once every node is on PKI, the bootstrap CA comes out.

Guarded, not documented

Running prune early partitions the cluster, so the script refuses. It
reads what each node actually serves over TLS, not what is on its
disk — a certificate installed but never reloaded is not migrated, and
that is exactly the case that would otherwise slip through.

Standbys swap before the active node. Not to avoid an election; a swap
costs no leadership. It is about what is still true if the run dies
halfway: the leader is the node you least want in an unknown state, so it
goes last, once the procedure has already worked twice.

Every node is health-gated between steps and the voter count must still
match — continuing past a node that did not come back is how one bad
certificate becomes an outage.

Tested

tests/pki-migration covers the refusals, since that is where the damage
is. tests/integration runs the whole rollout against the real
three-node cluster, including the prune that had previously only ever met
a shim, then checks every node serves a PKI issuer read off the wire, the
bootstrap subject is gone from the bundle, three voters remain, and a
re-run no-ops.

Also in this release

Two bugs the tests found rather than review. A trust-bundle prune
invalidates every other copy of that bundle — Prometheus and the blackbox
exporter both mount it, and until restarted they validate against a CA
that signs nothing, so the TLS probe goes dark and certificate expiry
becomes unmonitored with no error anywhere. And a shim that hardcoded one
openssl's DN rendering, which matched the machine it was written on and
nothing else.

See docs/security.md.

v0.8 — Audit devices

The only thing in Vault that answers "who read that secret."

scripts/bootstrap-audit.sh enables audit devices on the cluster. It
enables two by default, and that is the part worth reading twice.

The tradeoff, stated plainly

Vault refuses to service requests when it cannot write to any enabled
audit device. That is the correct behaviour — an unlogged request to a
secrets manager is worse than a failed one — but it means a single audit
device turns a full disk into a total outage.

Two devices is the mitigation. It is also why --force will not let you
disable the last remaining one.

Entries are HMAC'd rather than recorded in clear, which is what makes the
logs safe to ship somewhere central.

Tested

CI covers the failure modes, not just the happy path:

• that two devices are enabled by default
• that --force cannot disable the only remaining device
• that an enable which succeeds without actually enabling anything is
  treated as a failure, not a pass

What this does not do

Nothing forwards the logs anywhere. Both devices in the local profile
write to the same filesystem — which proves entries reach both, and
proves nothing about surviving a full disk. A second device on an
independent failure domain is what a real deployment needs, and it is
now the third of three blockers on the road to v1.0.

See docs/audit.md.

---

This is the first cut release. Tags v0.1 through v0.7 were applied
retroactively at the commit where each version's scope was complete —
a fair reading of the history, but not the same as having released at
the time.