Add feed-health monitor: alert when a segment's feed is empty/stale

[mircealungu]

Why this exists (the incident)

We already monitor the supply/download side: crawler_liveness_check.py ("is the crawler alive at all?") and feed_health_check.py ("which individual RSS feeds went quiet?"). Both watch whether articles get downloaded.

But recently a real user's feed was silently EMPTY for ~5 days while downloads were perfectly fine. The simplification pipeline had broken, and that user's segment (Danish, B2+) is filtered toward simplified content — of which there was suddenly none. A supply/download check cannot see this: articles were arriving, they just never survived the per-user CEFR + feature-flag filtering to reach the feed.

This adds a delivery-side check: does a real user in each segment actually see a fresh feed?

Approach

The segment dimension that matters is (learned_language, CEFR_level) — feed inventory and filters vary along it, and the incident hit one specific corner. For each segment the tool computes two signals:

1. INVENTORY (primary alert trigger). Re-runs the app's own query builder build_elastic_recommender_query parameterized by the segment's language + CEFR level, with no per-user seen-exclusion and no topic narrowing. This is "does fresh content this segment could be shown even exist in the index?" It replicates the exact available_cefr_levels filtering that starved the incident's segment, without being confounded by any one user's reading history. Uses include_lower=False on purpose — the incident hit users who see only their exact level, so the strict same-level query is the worst-case signal.

2. PER-USER CANARY (corroboration). Picks a real, recently-active user in the segment (last_seen within --active-days, CEFR via cefr_level_for_learned_language) and calls the real feed function article_recommendations_for_user(user, 10). This exercises the full delivery path: CEFR filtering, feature flags (simplified-only vs originals-first, "show easier"), disturbing filter, ES index, seen-exclusion, teacher-upload filtering. No synthetic users are created — only existing active users are read.

Why inventory is the primary signal (the seen-history caveat)

A single power-user can legitimately have an empty personal feed — they've already opened everything fresh. That's a false positive for "the segment is broken." Segment-wide inventory has no per-user history, so an empty/stale inventory is the trustworthy alarm; the per-user canary is reported alongside as corroboration and as a reality check on the flags. When the canary feed is empty but inventory is healthy, the report explicitly says so rather than alerting.

Per segment, the status is:

• EMPTY — fewer than min-articles recommendable items at all
• NO_FRESH — enough items exist but fewer than min-fresh are fresh (the incident shape)
• STALE — enough fresh items, but even the freshest is older than stale-days
• OK otherwise

On any failing segment it emails a per-segment summary via ZeeguuMailer (the app's own SMTP — cron stdout goes to an unread log, same rationale as the other monitors), prints a structured report, and exits non-zero so cron surfaces it.

Thresholds (starting guesses — tune with real data)

All module constants, each env-overridable so cron can adjust without a code change:

Threshold	Default	Env var
Active-user window	30 days	FEED_HEALTH_ACTIVE_DAYS
Fresh window	7 days	FEED_HEALTH_FRESH_DAYS
Min recommendable items	3	FEED_HEALTH_MIN_ARTICLES
Min fresh items	3	FEED_HEALTH_MIN_FRESH
Freshest-item stale cutoff	4 days	FEED_HEALTH_STALE_DAYS
Items requested	10	FEED_HEALTH_COUNT

Files

• tools/feed_delivery_health_check.py — the monitor
• tools/test_feed_delivery_health_check.py — 11 unit tests for the pure freshness/staleness classification and segment-status decision (python -m pytest tools/test_feed_delivery_health_check.py, all green). The full ES+DB path needs a live index and is not unit-tested; the app context is created lazily inside main() so the module imports DB-free for tests.

Proposed crontab line (for the ops repo — NOT wired here)

Once daily, after the morning crawl has had time to run:

30 6 * * *  cd $ZEEGUU_API && run_task.sh feed_delivery_health_check

(Adjust to match the run_task.sh invocation convention the other monitors use.)

Open questions

• Which segments are "must-never-be-empty"? Right now every active segment is checked equally. We may want an allow-list of core (language, CEFR) segments that page vs. merely log, and to ignore tiny long-tail segments (one user at C2 in a thin-inventory language) that are empty by nature, not by breakage.
• Threshold tuning. min-fresh=3 / stale-days=4 are guesses. A few weeks of per-segment numbers (especially overnight/weekend low-water marks) should calibrate these; the strict include_lower=False inventory query may be too aggressive for genuinely thin-inventory languages.
• Primary signal choice. I made inventory the trigger and the canary corroboration. Alternative: alert only when both inventory is thin and the canary is empty (fewer false positives, but would have caught the incident a bit later). Worth revisiting once we see real alert volume.

🤖 Generated with Claude Code

[github-actions]

ArchLens - No architecturally relevant changes to the existing views