Proposal: Watch health states + rules (replacing ad-hoc failure counters)

[dgtlmoon]

Related: #4584, #4695

Problem

Watch "health" today is mostly last_error, a free-text string that every except branch in worker.py overwrites, plus a single counter (consecutive_filter_failures) that filter-missing and browser-step failures both share.

That means:

• No error type: code can't tell a 403 from a missing filter except by string matching.
• No "since": "failing for 3 days" isn't answerable, and can't be worked out afterwards.
• No counts for anything except filters.
• Recovery is implicit: whichever branch happens to set the counter back to 0.
• "Unchanged checksum" counts as success even though no filters ran.

Every new request (#4584 "notify only once", "remind me daily", "tell me when it's fixed") turns into another set of flags and counters spread across the worker. #4695 needed three new watch fields plus changes to both processors' checksum shortcut just to get "notify once".

I'd like to replace this with a small, reusable health state per watch, with rules that act on it.

Proposal

1. Health conditions

Each watch tracks named conditions. Each except branch in the worker maps to one:

Condition	From
http_status	Non200ErrorCodeReceived
filter_missing	FilterNotFoundInResponse
browser_step	BrowserStepsStepException
fetch	connect error, timeout, empty reply, page unloadable
processor	processor exceptions

Each condition has state (ok / pending / failing), since, count, detail, last_action_at.

The worker reports each check result to a condition with observe():

health.observe(watch, 'filter_missing', Outcome.FAIL, detail={'filter': '#price'})

There are three outcomes: OK, FAIL, INCONCLUSIVE. An unchanged checksum is inconclusive: it neither confirms recovery nor counts as a failure. That removes the need for the processor-side hacks in #4695.

Transitions (ok→failing, failing→ok, "still failing after X") fire actions.

2. Rules: reuse what we already have

No new dependencies:

• panzi-json-logic (already used by Conditions) evaluates the rules, so the rule builder is the same one users already know.
• pluggy provides a health_transition(watch, condition, old, new) hookspec. Built-in actions and third-party plugins both attach to it.
• blinker pushes realtime UI updates.

The state machine itself is ~3 states per condition, so a library like python-statemachine / transitions doesn't add much. The hard parts are JSON persistence, "for longer than X" timing and actions that change watch config, and those libraries don't solve any of them.

Rules live with the rest of the config: global settings, then tag, then watch (the existing inheritance).

{"name": "Pause dead pages",
 "when": {"and": [{"==": [{"var": "condition"}, "http_status"]},
                  {"in": [{"var": "detail.status"}, [404, 410]]},
                  {">=": [{"var": "failing_for_seconds"}, 604800]},
                  {">=": [{"var": "count"}, 5]}]},
 "do": [{"action": "pause", "reason": "Page not found for 7 days"},
        {"action": "notify", "template": "lifecycle_paused"}],
 "once_per_incident": true}

The rule requires both time and count. A check every minute would hit 5 failures in 5 minutes, and a weekly check might only see 1-2 checks in 7 days. Use time for "too long", because check intervals vary so much.

3. Storage: sparse, keeps the watch dict small

With ~100k watches in memory, adding always-present keys to every watch dict is expensive. So:

• watch['health']: a small summary key, only present while something is wrong (state, condition, since, summary). It's absent when healthy, so it costs nothing. It's written to watch.json only when the state changes, which is rare. The UI list reads it directly from memory, with no extra file reads.
• {datadir}/{uuid}/health.json: full per-condition detail, counters and action history (audit trail). Written with the existing save_json_atomic(). The worker reads it only when a check failed or the watch is already failing. It's deleted when all conditions recover. This follows the existing pattern of {processor_name}.json living beside watch.json.
• last_error becomes a property backed by watch['health']['summary'], so the UI and API keep working and the always-present 'last_error': False key goes away.
• health is read-only in the API, dropped on clone/export, and kept across "clear history".
• Migration: move any non-zero consecutive_filter_failures / last_error into the new structure on first load.

4. Presets first, raw rules later

Ship named presets rather than exposing raw JSON straight away:

Preset	Default
Notify once on filter missing (until it matches again)	ON	closes #4584
Pause dead pages (404/410 for 7 days and ≥5 checks → pause + "we paused it" notification)	OFF	it's a destructive action
Escalate on blocks (403/challenge ×3 → browser fetcher → stealth → other proxy)	OFF
Recovery notification ("filter found again")	OFF

The JSON rule builder becomes the advanced option.

Ideas this enables

Self-healing fetches

• Escalate fetcher/proxy on 403 or a Cloudflare challenge. Try cheap first, pay for browsers and residential proxies only when blocked.
• Step back down: after N healthy days on the expensive setup, try the cheap one again.
• 429: honour Retry-After and back off exponentially, then restore the interval on recovery.

Domain and system scope

• 20 watches on one domain all failing is one incident: back off the domain, apply the working fix to all, send one notification.
• BrowserConnectError across all watches means the browser container is down. That's one system alert, and the watches are marked inconclusive rather than failing. No more 500 last_errors.
• Before auto-pausing on 404, check whether every watch on that domain is 404ing too. If so the site is broken, not the page removed.

Missing filters

• Have the LLM suggest a replacement selector from the current page and include it in the alert.
• Fall back to a whole-page diff while the filter is missing, so real changes aren't lost.

Lifecycle and noise

• 404 for 7 days → pause and tag "gone" (restock: treat as delisted).
• Flapping detection: a condition toggling >N times a day → mark "flaky", suppress alerts.
• Daily digest ("5 watches broken") instead of a stream of individual alerts.

Visibility

• A health column/filter in the watch list ("broken for 3 days"), and tag-level health.
• Health exposed via the API/MCP ("which of my watches are broken and why?").
• Health fields usable in Conditions ("only notify on change if the fetch was healthy").

Guardrails for rules that change settings

• Audit trail: "fetcher changed to playwright by rule Escalate on 403 on 2026-11-07", shown in the UI with one-click revert. Every notification names the rule that caused it.
• paused_reason shown in the list and on the edit page, so auto-paused watches don't look like a bug.
• Unpause resets the incident, so a still-404 page starts a fresh 7 days / 5 checks.
• Loop prevention: rules run once per incident, escalation steps are capped, and stepping back down needs N healthy checks first.
• Actions only count as done if they happened. A muted or failed notification isn't marked as sent. But a pause still applies even if the notification couldn't be sent, and the UI shows the reason.

Suggested order

1. Health record + observe() in the worker (incl. INCONCLUSIVE for unchanged checksum), last_error as a property, migration.
2. "Notify once on filter missing" preset, which closes [feature] Option to only send a notification once when "Send a notification when the filter can no longer be found on the page" is active #4584 and replaces Suppress repeated missing-filter notifications until recovery #4695. The tests in Suppress repeated missing-filter notifications until recovery #4695 are good scenarios and can mostly be ported.
3. "Pause dead pages" preset + paused_reason + lifecycle notification.
4. Escalation preset.
5. Domain/system scope.

Feedback welcome, especially on preset defaults, which conditions to start with, and any rules you'd want that this design wouldn't handle.