P4 — Alerting & incident response (full set)

[gangster]

Part of the observability epic #102 (Track B). Full design: docs/plans/102-observability-stack.md.

Scope. Promote P1’s minimal critical route to the full design: Mimir ruler (multi-tenant rules) + curated PrometheusRules (ArgoCD / Kyverno / Cilium / cert-manager / ESO / stores) + severity routing + inhibition + runbook_url links; Alertmanager → SNS / Slack / PagerDuty (escalation via oncall_provider); K8s events → Loki (event-exporter / Alloy); deploy/change annotations on dashboards.

Verify. A test alert fires end-to-end at each severity; inhibition + Watchdog confirmed; a deploy shows as a dashboard annotation.

Dependency / ordering. Needs P1 (done); the events→Loki piece needs P3.

[gangster]

P4 done + live. Promoted alerting to the full design:

• Routing: severity tree + inhibition — critical → SNS + Slack + PagerDuty (page), warning → Slack, info/Watchdog → dashboard-only. Verified end-to-end (Slack message + PagerDuty text/phone-call on a critical test alert). PRs feat(observability): promote Alertmanager routing — severity tree + inhibition (P4) #604 (routing), feat(observability): Alertmanager Slack receiver (P4) #610 (Slack), feat(observability): Alertmanager PagerDuty receiver — critical paging (P4) #612 (PagerDuty).
• 28 curated PrometheusRules across 7 components (cert-manager, kyverno, ArgoCD, Loki, Tempo, External Secrets, Cilium+Hubble), each grounded in live-verified metrics with runbook_urls. PRs feat(observability): curated Tier-2 PrometheusRules — cert-manager/kyverno/argocd (P4) #605, feat(observability): store self-monitoring — Loki + Tempo metrics + alerts (P4) #606 (store self-monitoring), feat(observability): External Secrets monitoring — ServiceMonitor + rules (P4) #607 (ESO), feat(observability): Cilium network-plane monitoring — scrape + rules (P4) #609 (Cilium). Metrics scraped via additionalServiceMonitors/PodMonitors where chart SMs are off/capability-gated.
• K8s events → Loki (feat(observability): K8s events → Loki (P4 incident-response glue) #608) and deploy/change dashboard annotations (feat(observability): deploy/change dashboard annotations (P4) #611).
• Runbooks + the as-built doc refreshed (docs(observability): refresh as-built for P3/P4 + notification runbook #613).

Acceptance met: test alerts fire end-to-end at each severity; inhibition confirmed; deploys show as dashboard annotations.

Deferred (optional/marginal): a scoped Cilium drop-rate alert (drops are partly policy-normal — needs per-env tuning); the Mimir ruler for multi-tenant rules activates when Mimir is enabled (off in the dev cost_profile). Neither blocks P4.

Closing.