Add HTTP webhook notifications for alert transitions (A3)

[LarsLaskowski]

Summary

Delivers alert fired/cleared events (from the A2 engine, #50) to
configurable HTTP webhooks, so alerts leave the box. A single generic JSON
POST covers Slack, Discord, Home Assistant, ntfy, etc. via their
incoming-webhook formats.

• internal/alert/notify.go — a new Notifier. Engine.Evaluate now
  returns the transition events it emits each tick; the collector forwards
  them to the notifier.
• Delivery is fully decoupled from collection. Notify only enqueues
  onto a bounded channel and a single background worker performs the POSTs,
  so a slow or failing webhook can never block metric sampling (the queue
  drops with a warning if it ever backs up).
• Per webhook: a severity filter (min_level: warn/crit), an
  optional Go text/template request body (defaulting to a JSON object
  describing the event), and a per-attempt timeout.
• Resilience: failed deliveries retry with exponential backoff and then
  give up without crashing; backoff waits respect context cancellation so
  shutdown stays prompt.
• Rate limiting: keyed per (url, metric, resource) so a fast-flapping
  alert can't flood a destination, while a distinct metric alerting in the
  same tick is still delivered.
• Config only (std-lib net/http, no new runtime dependency): webhooks
  and notifier tuning live under the alerts: block. Malformed templates and
  invalid webhook fields are rejected at startup.

Both acceptance criteria are covered by tests in notify_test.go against a
local httptest server: firing an alert delivers the expected JSON payload,
and a failing delivery retries with backoff then gives up without crashing or
blocking. Verified end-to-end by running the binary against a real local
webhook sink. Per the issue, email/SMTP (A4) is deliberately left out.

Related Issue

Closes #4

Checklist

• Tests added/updated for the change (go test ./... passes locally)
• go vet ./... and golangci-lint run are clean
• Documentation updated if this changes the REST API (docs/API.md),
  configuration (README.md, packaging/pimonitor.example.yaml), or
  installation/packaging (packaging/install.sh, systemd units)
• No breaking change to /api/v1/... response shapes, or a new API
  version was introduced instead

[LarsLaskowski]

Review: Alert webhook notifications (A3)

Reviewed against the project's Go/security/API conventions. On claude/elegant-feynman-aur2l5:

• go build ./... — clean
• go vet ./... — clean
• go test ./... — all packages pass

This is a well-structured change. The decoupling of delivery from collection is exactly right: Notify only enqueues onto a bounded channel and never blocks the fast tick, per-attempt timeouts are enforced via context.WithTimeout, resp.Body is always closed, backoff respects context cancellation, and deliver can't panic the worker. No exec.Command, no new dependencies (std-lib net/http + text/template), no /api/v1 shape change (the message field lives only in the webhook payload). Config validation runs regardless of Enabled so typos fail fast at startup. Test coverage is thorough — default payload, template, bad-template rejection, retries-then-give-up, min_level filter, per-metric rate limit, and the eventReaches table.

Nothing here is blocking. A few observations:

Worth considering: rate limiter can drop a cleared event

rateLimited is keyed per (url, metric, resource) regardless of Kind, and dispatch applies it to every event. If a cleared arrives within notify_min_interval_seconds of the preceding fired delivery, the clear is dropped — leaving a state-based downstream consumer (e.g. a Home Assistant binary_sensor) stuck "on" with no correction until the next transition.

With the defaults this can't happen: the debounce (for_seconds: 30) applies symmetrically, so any two transitions for one metric are ≥30s apart, comfortably outside the 5s window. It only becomes reachable when a user sets for_seconds below notify_min_interval_seconds and the metric flaps. Given that, it's a latent edge rather than a live bug — but since the rate limit exists to coalesce repeated firings, it may be worth exempting cleared events (or keying the limit on Kind) so a clear is never suppressed. A small test pinning the intended behavior would document the decision either way.

Minor

• Notifier.wg is never awaited. Start does wg.Add(1)/defer wg.Done(), but nothing calls wg.Wait(), so the WaitGroup currently has no effect — on ctx cancel the worker just returns and queued events are dropped (which the doc comment does state). Either drop the field or add a Stop()/drain that waits on it, so the intent is unambiguous.
• Content-Type: application/json is sent unconditionally, including for a custom template that may render non-JSON. Fine for Slack/Discord/ntfy's JSON payloads, but a user templating a plain-text body would get a mislabeled request. Low impact; noting for completeness.
• lastSent grows unbounded, keyed per (url, metric, resource). Bounded in practice by the small set of metric/mountpoint combinations, so not a concern on a Pi — just flagging that pruned disk mountpoints leave stale entries behind, unlike the engine's pruneDisks.

Overall this looks ready to merge; the cleared-vs-rate-limit point is the only one I'd ask you to make a deliberate call on.

[LarsLaskowski]

Thanks for the thorough review. Addressed in 1192a85:

Rate limiter dropping a cleared (the deliberate call): made cleared events bypass the rate limiter entirely. A recovery signal must always land, so a state-based consumer can never get stuck "on"; the limiter now purely coalesces repeated firings. I considered keying the limiter on Kind instead, but exempting cleared outright is the stronger guarantee (the dangerous failure is a stuck alarm, not a missed re-fire) and reads more clearly. New test TestNotifier_ClearedBypassesRateLimit pins it, and the existing rate-limit test now uses two firings since a fired/cleared pair is no longer a valid rate-limit case.

wg never awaited: added Notifier.Stop() which joins the worker, wired into the collector via defer c.notifier.Stop() after Start. By the time Run returns the context is already canceled, so Stop unblocks promptly (post uses a ctx-derived timeout and backoff waits are ctx-aware) — the WaitGroup now guarantees no POST is in flight past exit rather than being dead code.

Content-Type unconditional: added an optional per-webhook content_type (default application/json) so a plain-text template body isn't mislabeled. Covered by TestNotifier_CustomContentType.

lastSent unbounded growth: left as-is deliberately — it's keyed by the small, near-fixed set of (url, metric, mountpoint) combos, so on a Pi it's effectively bounded, and adding pruning would mean threading disk-prune notifications into the notifier for a few stale map entries. Happy to revisit if you'd rather have it symmetric with the engine's pruneDisks.

go build / vet / test ./... (incl. -race) and golangci-lint all clean.