Skip to content

monitoring: record what was built, and close the freshness gap for free (C-335 resolved) - #410

Merged
Polichinel merged 1 commit into
developmentfrom
docs/monitoring-followthrough
Aug 2, 2026
Merged

monitoring: record what was built, and close the freshness gap for free (C-335 resolved)#410
Polichinel merged 1 commit into
developmentfrom
docs/monitoring-followthrough

Conversation

@Polichinel

Copy link
Copy Markdown
Contributor

Resolves C-335. Registers C-338. Closes #401.

C-335 is closed

Better Stack monitor live on the public status.html, 3-minute interval. Verified rather than assumed: Up at ~27 ms from Europe, test alert delivered and read. The unbounded failure — Caddy stops while the host stays up, pipeline keeps succeeding and pinging, nobody ever notices — is shut.

Two qualifications kept in the closure rather than smoothed away: detection is by e-mail, not phone, and the freshness half is not in the vendor at all.

The drill found the specification was wrong

ADR-051 promised two checks. Better Stack gates keyword matching behind a paid plan, so the content half became .github/workflows/serving-freshness.yml — daily, on GitHub, genuinely external.

My first version searched the page text for stale and missing. Against the live, healthy server:

CASE 1: live healthy page (expect degraded=false)
degraded=true
- 1 stage(s) report **stale**
- 1 stage(s) report **missing**

The page has a legend● OK ● Stale ● Missing — explaining the dot colours. Those words are on every healthy page. An earlier count said zero because it was case-sensitive.

Shipped, that opens an issue every day until someone mutes it.

ADR-051 specified the paid check identically"alerts when the body does not contain the healthy marker." Same bug. Buying the feature would not have saved us; we'd have configured a vendor keyword monitor with the same false positive and trusted it more. The workflow parses per-cell title="<status>" attributes instead (ok ×38, not_applicable ×10 when healthy).

Four paths drilled

Case Result
Live healthy page degraded=false
Aged past 40 days fires, age reason only ✅
One stage genuinely stale fires ✅
Page reshaped, no status cells fires — a check that silently inspects nothing is worse than no check

Changes

  • ADR-051 — dated amendment recording what was built, what wasn't, why, and that its own spec was wrong. Not a silent rewrite; the gap between decided and buildable is the record.
  • serving-freshness.yml — daily + dispatch, one reusable issue, closes itself on recovery. Stays silent when the page is unreachable — that's Better Stack's job and it has already alerted; two alarms for one event is how people learn to ignore both.
  • docs/guides/monitoring.md — setup of record. The monitor as configured, the three-way split (heartbeat = did it run, Better Stack = is it reachable, workflow = is it current), what we deliberately don't use, how to leave, and §5: what to configure if the plan is ever upgraded — recorded at the operator's explicit request.
  • C-338 — the residual, Tier 4. Freshness is daily-and-by-issue, and GitHub may delay cron. Late notice of stale data, not wrong data.

Verification

Full suite exit 0 · ruff clean · validate_docs.sh exit 0 · all register guards green (index 1741 < 8000, header ≤ 3500, 338 IDs / 292 resolved / 43 open = 0+2+10+25+6, 108 struck-through)

Live serving path re-checked HTTP 200 immediately before closing C-335 — the claim being closed is a claim about the world, not the repo. That's C-330's lesson.

…ap for free

C-335 is resolved. A Better Stack monitor now polls the public status page
every 3 minutes: verified Up at ~27ms from Europe, with a test alert
delivered and read. The unbounded failure — Caddy stops while the host
stays up, pipeline keeps succeeding and pinging, nothing ever notices —
is closed.

But ADR-051 specified TWO checks and only one was buildable. Better Stack
gates keyword matching behind a paid plan. Rather than pay, or quietly
ship an ADR describing a monitor that does not exist, the content half
moved to .github/workflows/serving-freshness.yml — daily, running on
GitHub, so genuinely external: it does not share fate with the host it
watches, which is the objection that ruled out an on-host cron.

THE DRILL FOUND THE SPEC WAS WRONG, NOT JUST THE IMPLEMENTATION.

My first version searched the page text for "stale" and "missing".
Against the live, healthy server it reported one of each:

  CASE 1: live healthy page (expect degraded=false)
  degraded=true
  - 1 stage(s) report **stale**
  - 1 stage(s) report **missing**

The page carries a legend — "● OK ● Stale ● Missing" — explaining the dot
colours, so those words appear on every healthy page. An earlier count
said zero because it was case-sensitive. Shipped, this would have opened
an issue every day until somebody muted it: a monitor that cries wolf on
day one and is ignored by day three.

ADR-051 specified the PAID check the same way — "alerts when the body
does not contain the healthy marker". Identical bug. Buying the feature
would not have saved us; we would have configured a keyword monitor with
the same false positive and trusted it MORE, because it came from a
vendor. The workflow parses per-cell title="<status>" attributes instead
(ok x38, not_applicable x10 on a healthy page).

Four paths drilled: healthy, aged past 40 days, one stage genuinely
stale, and the page reshaped so the check would inspect nothing — the
last treated as a problem, because a check that silently inspects nothing
is worse than no check.

ALSO
- ADR-051 gains a dated amendment: what was built, what was not, why, and
  that its own specification was wrong. Not a silent rewrite — the gap
  between what was decided and what was buildable is the useful record.
- docs/guides/monitoring.md is the setup of record: the monitor exactly as
  configured, the three-way division of labour (heartbeat = did it run,
  Better Stack = is it reachable, workflow = is it current), what we
  deliberately do not use, how to leave, and — at the operator's explicit
  request — §5 listing what to configure if the plan is ever upgraded.
- C-338 registered for the residual: freshness detection is daily and by
  issue rather than 3-minute and by e-mail, and GitHub may delay cron.
  Tier 4 — late notice of stale data, not wrong data.
- The workflow stays silent when the page is unreachable. That is Better
  Stack's job and it has already alerted; two alarms for one event is how
  people learn to ignore both.

Verified: full suite exit 0, ruff clean, validate_docs.sh exit 0, all
register guards green (index 1741 < 8000, header <= 3500, 338 IDs / 292
resolved / 43 open = 0+2+10+25+6, 108 struck-through). Live serving path
re-checked HTTP 200 before closing C-335 — the claim being closed is a
claim about the world, which is C-330's lesson.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Polichinel
Polichinel enabled auto-merge (squash) August 2, 2026 23:29
@Polichinel
Polichinel merged commit 282ffd2 into development Aug 2, 2026
5 checks passed
@Polichinel
Polichinel deleted the docs/monitoring-followthrough branch August 2, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant