v0.13 — Alert routing by severity, and MySQL as a second database engine

Alertmanager was configured with a single catch-all route and a receiver
declared as webhook_configs: [] — one that parses, validates, and
delivers nowhere. The database engine spoke only PostgreSQL. Both are
addressed here, and the same release found three assertions that passed
while the thing they described was broken.

Alert routing

There are still no vendor receivers, deliberately. A config full of fake
PagerDuty keys proves nothing a reader could reuse. What transfers is
everything above the vendor:

	Critical	Warning
Receiver	page	ticket
group_wait	0s	30s
repeat_interval	15m	12h

group_wait: 0s is the point of the split. At the default a page waits
to see whether a second alert joins its group — a fine trade for a
ticket, a bad one for a cluster that has lost quorum. And a critical that
notifies once then goes quiet is indistinguishable from one nobody sent.

Grouping is by alertname and cluster, deliberately not by
instance: three nodes sealing at once is one incident, and grouping per
instance would page three times for it.

Two inhibit rules, both encoding the same judgement — when one fact
explains another, only the explaining fact is worth waking someone for.
No scrape targets at all explains every node being down; lost quorum
explains there being no active node.

Proving delivery needed a receiving end. Alertmanager's API can say
an alert arrived; it cannot say which receiver it reached, and a
misrouted alert looks exactly like a correctly routed one. Each receiver
posts to a sink that records the delivery, and the integration suite
asserts a critical alert reached the pager path and was not also filed
as a ticket.

Which route matches a label set is put to amtool from the pinned
image rather than modelled in the test, because a test that reimplements
the thing it is testing agrees with itself and nothing else.

The check worth knowing about: every alert in the rule file must carry a
severity that has a route of its own. One with a misspelled severity
matches no route, falls to the catch-all, and is never paged for —
nothing else here would notice. The suite asserts that failure mode
directly by asking amtool where severity=crticial goes.

MySQL, and where it is weaker

Same interface, same mount, one script. Only the plugin, the connection
string and the SQL differ.

The reason to add a second engine was never breadth. It was to find out
where "Vault's database engine supports X" hides something, and it does:

	Postgres	MySQL
Credential expires because	the database enforces it, and Vault revokes it	Vault revokes it
Vault down at lease expiry	still dies on schedule	stays live
readwrite can create tables	yes	no
Username limit	63 bytes	32 characters

MySQL has no VALID UNTIL. CREATE USER takes no deadline, so a
credential issued against it lives until Vault revokes it, where a
Postgres credential dies on schedule even if Vault is unreachable at
lease end. The Postgres path degrades safely; the MySQL path depends on
Vault being there at the right moment.

That is asserted in both directions — one test checks the MySQL
statements do not contain VALID UNTIL, its neighbour checks the
Postgres ones still do — because the tempting fix is to paste it in so
both engines look alike, and it is a syntax error there.

The readwrite difference is not quietly fixable either. Postgres ties
DDL to ownership: a credential can alter what it created and nothing
else. MySQL grants per schema, so the same privilege would let every
readwrite credential drop the whole database.

Vault connects as a dedicated vaultadmin account rather than root.
Issuing credentials needs CREATE USER and GRANT OPTION, and root is
the obvious shortcut — but the bootstrap rotates the password of whatever
account it connects as, and root's password is what the container
healthcheck uses. That would leave the container permanently unhealthy
while MySQL itself was fine.

The integration suite runs all of it against a real MySQL server: the
generated username fits the 32-character cap (it comes out at exactly
32), the credential connects, readonly cannot write or read mysql.user,
revoking the lease drops the account from mysql.user, and the bootstrap
password stops working after rotation.

Two cleanup handlers that turned success into failure

A cleanup function whose last command is a false test returns 1, and bash
applies that to the script's exit status from an EXIT trap:

cleanup() { [[ -n "$D" && -d "$D" ]] && rm -rf "$D"; }
trap cleanup EXIT
exit 0            # exits 1 when $D is empty

Both trap handlers here had that shape. Probed directly, vault-upgrade.sh
and oidc-login-test.sh exited 1 on an early success path.

Latent rather than live — nothing currently exits 0 before the temporary
directory is created. It goes live the moment someone adds "already at
the target version, nothing to do", which is a natural addition to an
upgrade script and impossible to connect to a cleanup function three
screens away.

Worth being precise about, because the shape is idiomatic elsewhere.
Under set -e a trailing && whose test is false does not exit at
the top level, and does not exit as the last statement of an if/else
branch. It is specifically the function-return path, and the trap that
consumes it.

What this release learned about its own tests

docs/roadmap.md rests on one claim: shipped means there is a test that
fails if the feature breaks. Three assertions did not.

An exclusion assertion is only as good as the spelling its author thought
to forbid. "aws s3 cp" reads like it forbids uploading — an upload via
aws s3api put-object passed it, demonstrated by making a standby upload
and watching the guard stay green. Two more of the same shape: a token
revoked through auth/token/revoke-self passed "a supplied VAULT_TOKEN
is left alone", and a blob deleted with curl --request DELETE passed
"no blob DELETE".

The mutation testing meant to catch this had the same flaw one level up.
An assertion rejecting CREATE, ALTER, DROP, "verified" by a mutation
granting exactly CREATE, ALTER, DROP, proves only that grep works;
granting CREATE alone walked straight through. A useful mutation is one
the assertion does not name.

Both habits are now in CONTRIBUTING.md, and tests/lint enforces in CI
the invariants shellcheck has no opinion about.

None of this changes what the shipped table claims. It changes how much
the word "tested" in it is worth.

Still not covered

• Neither cloud profile has been applied. Unchanged, and still the
  first v1.0 blocker. docs/cloud-apply.md lists what a real apply would
  settle.
• No vendor integration. Whether your PagerDuty key is correct is not
  something this repository can tell you.
• The cloud profiles have no monitoring stack at all, so every
  alerting guarantee here applies to the local profile only.
• Inhibition is checked structurally, not by arranging two alerts to
  overlap on a live cluster.