monitoring: record what was built, and close the freshness gap for free (C-335 resolved) - #410
Merged
Merged
Conversation
…ap for free C-335 is resolved. A Better Stack monitor now polls the public status page every 3 minutes: verified Up at ~27ms from Europe, with a test alert delivered and read. The unbounded failure — Caddy stops while the host stays up, pipeline keeps succeeding and pinging, nothing ever notices — is closed. But ADR-051 specified TWO checks and only one was buildable. Better Stack gates keyword matching behind a paid plan. Rather than pay, or quietly ship an ADR describing a monitor that does not exist, the content half moved to .github/workflows/serving-freshness.yml — daily, running on GitHub, so genuinely external: it does not share fate with the host it watches, which is the objection that ruled out an on-host cron. THE DRILL FOUND THE SPEC WAS WRONG, NOT JUST THE IMPLEMENTATION. My first version searched the page text for "stale" and "missing". Against the live, healthy server it reported one of each: CASE 1: live healthy page (expect degraded=false) degraded=true - 1 stage(s) report **stale** - 1 stage(s) report **missing** The page carries a legend — "● OK ● Stale ● Missing" — explaining the dot colours, so those words appear on every healthy page. An earlier count said zero because it was case-sensitive. Shipped, this would have opened an issue every day until somebody muted it: a monitor that cries wolf on day one and is ignored by day three. ADR-051 specified the PAID check the same way — "alerts when the body does not contain the healthy marker". Identical bug. Buying the feature would not have saved us; we would have configured a keyword monitor with the same false positive and trusted it MORE, because it came from a vendor. The workflow parses per-cell title="<status>" attributes instead (ok x38, not_applicable x10 on a healthy page). Four paths drilled: healthy, aged past 40 days, one stage genuinely stale, and the page reshaped so the check would inspect nothing — the last treated as a problem, because a check that silently inspects nothing is worse than no check. ALSO - ADR-051 gains a dated amendment: what was built, what was not, why, and that its own specification was wrong. Not a silent rewrite — the gap between what was decided and what was buildable is the useful record. - docs/guides/monitoring.md is the setup of record: the monitor exactly as configured, the three-way division of labour (heartbeat = did it run, Better Stack = is it reachable, workflow = is it current), what we deliberately do not use, how to leave, and — at the operator's explicit request — §5 listing what to configure if the plan is ever upgraded. - C-338 registered for the residual: freshness detection is daily and by issue rather than 3-minute and by e-mail, and GitHub may delay cron. Tier 4 — late notice of stale data, not wrong data. - The workflow stays silent when the page is unreachable. That is Better Stack's job and it has already alerted; two alarms for one event is how people learn to ignore both. Verified: full suite exit 0, ruff clean, validate_docs.sh exit 0, all register guards green (index 1741 < 8000, header <= 3500, 338 IDs / 292 resolved / 43 open = 0+2+10+25+6, 108 struck-through). Live serving path re-checked HTTP 200 before closing C-335 — the claim being closed is a claim about the world, which is C-330's lesson. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves C-335. Registers C-338. Closes #401.
C-335 is closed
Better Stack monitor live on the public
status.html, 3-minute interval. Verified rather than assumed: Up at ~27 ms from Europe, test alert delivered and read. The unbounded failure — Caddy stops while the host stays up, pipeline keeps succeeding and pinging, nobody ever notices — is shut.Two qualifications kept in the closure rather than smoothed away: detection is by e-mail, not phone, and the freshness half is not in the vendor at all.
The drill found the specification was wrong
ADR-051 promised two checks. Better Stack gates keyword matching behind a paid plan, so the content half became
.github/workflows/serving-freshness.yml— daily, on GitHub, genuinely external.My first version searched the page text for
staleandmissing. Against the live, healthy server:The page has a legend —
● OK ● Stale ● Missing— explaining the dot colours. Those words are on every healthy page. An earlier count said zero because it was case-sensitive.Shipped, that opens an issue every day until someone mutes it.
ADR-051 specified the paid check identically — "alerts when the body does not contain the healthy marker." Same bug. Buying the feature would not have saved us; we'd have configured a vendor keyword monitor with the same false positive and trusted it more. The workflow parses per-cell
title="<status>"attributes instead (ok×38,not_applicable×10 when healthy).Four paths drilled
degraded=false✅Changes
serving-freshness.yml— daily + dispatch, one reusable issue, closes itself on recovery. Stays silent when the page is unreachable — that's Better Stack's job and it has already alerted; two alarms for one event is how people learn to ignore both.docs/guides/monitoring.md— setup of record. The monitor as configured, the three-way split (heartbeat = did it run, Better Stack = is it reachable, workflow = is it current), what we deliberately don't use, how to leave, and §5: what to configure if the plan is ever upgraded — recorded at the operator's explicit request.Verification
Full suite exit 0 ·
ruffclean ·validate_docs.shexit 0 · all register guards green (index 1741 < 8000, header ≤ 3500, 338 IDs / 292 resolved / 43 open = 0+2+10+25+6, 108 struck-through)Live serving path re-checked HTTP 200 immediately before closing C-335 — the claim being closed is a claim about the world, not the repo. That's C-330's lesson.