Backlog: Observability & ops
Problem
When something goes wrong (systemic job failure, delivery failure, panic, worker-loop death), nobody is notified — an unattended background service fails silently.
Proposed solution
Email operator alerts to support@keywind.cc. Full design already written in ERROR-ALERTING-PLAN.md. Key points:
- A DI
Reporter (no-op default, Mailgun-backed impl reusing the existing send path), following the Engine pattern.
- Alert only on systemic failures, not expected ones — this requires splitting the two kinds of "job failed" in
App.execute (a transformer that cleanly reported success:false on bad input is the sender's problem and is already emailed to them; a crash/timeout/missing-result is systemic). That split is good design on its own.
- Mandatory throttle/dedupe — the delivery loop ticks every second, so an unthrottled failure = one email/second.
- Honest gap: "Mailgun send is failing" can't be reported via Mailgun — that's what the heartbeat issue covers.
Phasing
Per the plan: Reporter interface + fake + tests → taxonomy split in execute → Mailgun reporter + config + throttle → wire remaining sites (delivery failure, panic recovery, worker-loop fatal).
Notes
- Config:
alert_recipient: support@keywind.cc in email.yaml (empty = disabled).
- See
ERROR-ALERTING-PLAN.md for the full taxonomy table and open decisions.
Backlog: Observability & ops
Problem
When something goes wrong (systemic job failure, delivery failure, panic, worker-loop death), nobody is notified — an unattended background service fails silently.
Proposed solution
Email operator alerts to
support@keywind.cc. Full design already written inERROR-ALERTING-PLAN.md. Key points:Reporter(no-op default, Mailgun-backed impl reusing the existing send path), following theEnginepattern.App.execute(a transformer that cleanly reportedsuccess:falseon bad input is the sender's problem and is already emailed to them; a crash/timeout/missing-result is systemic). That split is good design on its own.Phasing
Per the plan: Reporter interface + fake + tests → taxonomy split in
execute→ Mailgun reporter + config + throttle → wire remaining sites (delivery failure, panic recovery, worker-loop fatal).Notes
alert_recipient: support@keywind.ccinemail.yaml(empty = disabled).ERROR-ALERTING-PLAN.mdfor the full taxonomy table and open decisions.