Skip to content

QUIETZ v0.1.0

Latest

Choose a tag to compare

@PNX89 PNX89 released this 29 Aug 10:34
· 2 commits to main since this release

Monitors declared as data, with the alert rules generated from them and unit tested in both directions.

The indicator is freshness and completeness, not an HTTP status code. A feed that returns 200 and yesterday's numbers is healthy by every status check ever written, and that is the failure this exists to catch.

Four monitors, and only one of them wakes anybody. The registry refuses a monitor that pages on a window longer than two trading days, because waking somebody for a feed allowed to be three days late is how a rota learns to ignore the pager. It also refuses a monitor with no stated reason: one nobody can weigh is one nobody ever removes.

The windows are in trading days, so a feed that publishes on business days is not late on a Sunday. No holiday calendar ships here, and the consequence is stated beside the code rather than in a document nobody opens.

The rules are unit tested in both directions. promtool check rules says a file parses. These say an alert fires when it should, stays quiet when the same lateness falls inside another owner's tolerance, does not fire on a three minute dip, and produces nothing at all on a healthy feed. That last one matters most: a rule set nobody has watched staying quiet is one nobody can trust to stay quiet.

One incident: 24 alert firings become 2 notifications, measured against a real Alertmanager with the notifications counted at a receiver rather than reasoned about from the configuration. Deduplication, grouping and inhibition are three different mechanisms and the third is the one deduplication could never have caught.

This one fails open, and every sibling is right to fail closed. A notification path that fails closed suppresses, and a monitoring system that goes quiet when it is confused is most silent exactly when something is most wrong.

Verification. 6 required status checks, three of which re-run a measurement against a real Prometheus or Alertmanager and fail if a byte of the result changed.