Neither cloud profile has been applied. That has not changed.
What changed is how much of an apply's job can be done before paying for
one — and the answer turned out to be five real defects, none of which
the twenty-seven checks running on every pull request could see.
What was broken
Nobody could have reached the cluster. The node security group
admitted port 8200 from the load balancer's security group. But the target
group uses target_type = "instance", where client IP preservation is
enabled and cannot be turned off — so the load balancer forwards each
packet with the client's address as its source. The reference matched
the health checks, which come from the load balancer's own interface, and
nothing else. Every target healthy, every client dropped.
No snapshot could have uploaded. The bucket sets SSE-KMS default
encryption with the auto-unseal key, and S3 has the caller mint the data
key. Nothing granted kms:GenerateDataKey. The bucket, the lifecycle rule
and the s3:PutObject grant were all correct, and every hourly upload
would have been denied by KMS rather than by S3.
No peer could have joined. retry_join verifies the leader against
leader_tls_servername, one fixed name whichever node holds leadership.
The templates asked for vault.<cluster>.internal; the PKI role issued
<host>.vault.internal. That name existed in two files in the repository
and no certificate anywhere carried it.
A bare apply never terminated. The autoscaling group used ELB health
while the profile deliberately does not issue certificates. Vault could
not start, the check could not pass, and every instance was terminated at
the end of its grace period and replaced by one that repeated it — billing
EC2, three NAT gateways and EBS throughout, and looking like a slow
bootstrap.
Teardown stranded the backups. The root volume was encrypted with the
seal key, so terraform destroy scheduled the one key every archived
snapshot depended on. storage.tf warned in plain English that the key
must survive a teardown; the code deleted it.
What now stops the class recurring
tests/preflight-static — the agreements no single layer can see.
Every value a template reads is one Terraform passes and vice versa; the
rendered cloud-init parses and is shellchecked (it was linted nowhere
before); every auto_join key is one go-discover accepts in a combination
it accepts; the leader name is one the PKI role can actually issue. No
credentials, nothing created, seconds to run.
tests/cloud-apply-emulated — a real terraform apply, through the
real AWS provider, against an implementation of the AWS API. It settles
what mocked providers structurally cannot: whether the configuration
applies at all in one pass, whether the AMI filter matches anything,
whether any value is refused, and whether destroy takes it back down.
The thing worth taking away
Three of the five defects were held in place by passing assertions — a
test requiring the exact value that made the apply loop, one forbidding
the rule the cluster needed to serve traffic, and one comparing arns a
mock had flattened to a single value. Fixing each required deleting
something green.
That is written up at
I had to delete a passing test to fix the bug.
Also in this release
The Makefile now does what CI does — make lint could previously pass
while CI failed — with suite discovery and a check-ci-coverage target
that fails when a suite exists that no CI job runs.
bootstrap-dev-cluster.sh checks its tools before generating anything.
Provider lock files cover five platforms, so an ARM checkout stays clean.
Also, found the same way
A shell script checked out with CRLF does not run — its interpreter
becomes bash , and the failure surfaces as exit 127 naming no file
and no reason. Git for Windows ships core.autocrlf=true in its system
config, so that is the default for a Windows contributor, and CI cannot
see it because the runners check out on Linux.
make test-all failed on two scripts and passed on fifty-eight, because
only files Git actually rewrites get converted. .gitattributes now pins
LF and the tree is normalised.
Still requires an account
The KMS triangle against a real key policy. A terminated leader replaced
unattended. Real health checks keeping standbys in the pool. Azure's
scale-set discovery against real instances. Everything reachable from the
files has now been reached.
See docs/cloud-apply.md.