v0.161.3: Monitoring that can interrupt you, instead of waiting to be checked

The problem

An audit of the monitoring stack found six observability channels and effectively zero automated alerting. Sentry, the system_logs viewer, /api/health, the debug dashboard, PostHog and the home-ops ingest are all pull-based — someone has to go and look before anything is noticed.

That's how a dead client-side Sentry SDK stayed dead for weeks (#573). Worse: lib/env.ts did detect the missing DSN and warned — into system_logs, which nobody reads. A monitoring system whose failure notice is delivered through an unread channel conceals its own failure.

This closes the two holes that made that unsafe.

1. The cron fleet could stop entirely, silently

Every scheduled job in Strummy rides one Vercel cron (Hobby allows a single definition): /api/cron/dispatcher fans out to 15 jobs in-process — lesson reminders, notification delivery, calendar sync, the GDPR cleanup.

A job that throws was already reported (logger.error → captureException). But a cron that never fires produces no error to capture, so that failure mode was pure silence. Your students would have noticed before you did.

The dispatcher now checks in to a Sentry cron monitor (cron-dispatcher, crontab 0 6 * * *, 30-minute margin, UTC): in_progress on entry, ok/error on exit. Sentry alerts on a missed check-in — which also catches "Sentry is misconfigured", because from Sentry's side both look identical: nothing arrived.

Two details worth knowing:

• No check-in is recorded when verifyCronSecret fails, so a stranger hitting the URL can't suppress the alert by faking a run.
• automaticVercelMonitors: true cannot do this — it doesn't instrument App Router route handlers, which is every route here. next.config.ts already said so in a comment; nothing had acted on it.

2. The dispatcher answered 200 even when jobs failed

Individual /api/cron/* routes keep returning 200-with-error-payload (no paging on known-degraded states). But the dispatcher is the fleet's single status signal, and an always-200 answer makes a broken run indistinguishable from a healthy one to any external prober.

It now returns 500 when any job failed. Deliberate no-ops are still excluded via skipped. The existing test that asserted 200 on a failed job now asserts 500 — that assertion change is the behavior change.

3. Nothing outside the app could check whether it was alive

/api/health is admin-only (401 without a session), so no uptime monitor can poll it. New /api/health/live is unauthenticated and returns a single reachability bit:

• 200 {"status":"ok","database":"up"} — app serves and database answers
• 503 — otherwise (unconfigured counts as down; in production a missing URL/key is an outage)

It leaks nothing — a test asserts the body has exactly status/database/checkedAt, because the underlying checker's message carries raw error text including LAN hosts and ports. And a plain "is the URL up" check is no substitute: Next happily returns 200 for a shell page with a dead database behind it.

Verified

• lint 0 errors (297 warnings — unchanged baseline; check-in logic moved to lib/health/cron-monitor.ts because keeping it inline pushed the route past the 200-line limit)
• tsc 0 errors
• 291 suites / 3805 tests green — +1 suite, +8 tests

Manual test steps: docs/manual-tests/2026-07-30-observability-alerting.html

⚠️ Two manual steps make this actually useful

A monitor with no alert rule is still pull-based. Both need credentials I don't have:

1. Sentry → Alerts: rule on the cron-dispatcher monitor for missed and error check-ins → email/phone. The monitor is created on first check-in, so trigger the dispatcher once first.
2. Uptime Kuma (on the Pi): HTTP monitor on https://strummy.vercel.app/api/health/live, 60s interval, accepted status 200-299 only so the 503 trips it. This is the only monitor that survives the whole Vercel deployment being down, because it runs outside it.

Deliberately not included

• Dropping the dormant audit_log + 12 partitions. Destructive production DDL — it belongs in its own migration with a rollback path, with explicit approval, not bundled into a monitoring change.
• Removing lib/observability/home-ops-log.mjs — I'd flagged this as dead code and was wrong. It's imported by app/api/cron/lesson-reminders/route.ts and has test coverage; it's simply unconfigured. Enabling it is just setting INGEST_URL/INGEST_TOKEN.

Docs

10-admin-observability.md gains the check-in contract and the two-health-endpoint rationale. Its UI-surfaces table is also corrected: it listed /dashboard/logs and /dashboard/admin/debug as "Coming soon" placeholders while its own Gaps section said both shipped 2026-07-19. Both are mounted.

Obsidian vault not yet updated — follow-up from the Mac.

---

Full Changelog: v0.161.2...v0.161.3