Observability stack (epic) — metrics, logs, traces, profiles, cost

[gangster]

Observability stack — epic

Single pane of glass (metrics, logs, traces, profiles, cost, cloud resources) across all environments. Closes CNCF capability gap #7 (today only Hubble + ArgoCD). Delivered as the full stack in small, independently-verifiable phases — one Terragrunt unit + gated plan→apply + a runtime check each, never batched.

Full design: docs/plans/102-observability-stack.md

Progress: 2 / 14 phases live (P1, P2). Each remaining phase is now its own tracked sub-issue (links below).

Topology — hub & spoke

• Hub = platform cluster: central Grafana + SSO + multi-tenant stores Mimir (metrics) / Loki (logs) / Tempo (traces). One UI, one set of stores.
• Spokes = each workload cluster: lightweight collectors (Alloy / OTel) remote-write/push to the hub over Tailscale/TGW.
• Tenancy: tenant = team = teams.hcl key, stamped as X-Scope-OrgID from the namespace label on the write path, enforced server-side (auth_enabled: true), pinned per-team on the read path. Grafana is Tailscale-only.

Locked decisions

• Terragrunt units; hub on platform; isolation-ready from day one.
• Absolute latest stable (GA) chart + provider pins, re-resolved per phase.
• HA is a per-unit high_availability toggle (mirrors the ArgoCD module): off on reference clusters, on for prod/capacity-rich.
• Cost-conscious: attack data volume (retention, trace tail-sampling, cardinality limits, S3 lifecycle) + compute (Graviton/arm64, spot for stateless tiers), not replica counts. OpenCost (P11) is the evidence.
• OSS-default, commercial opt-in: every capability ships OSS as default; commercial (Grafana Enterprise/Cloud, AMP, SLO/Synthetics/OnCall) is a per-unit flag, never a fork.

Security invariants (load-bearing)

X-Scope-OrgID is a trust header, not auth → cross-cluster ingest is authenticated + the hub overwrites tenant from the authenticated source identity + the collector derives tenant from the verified pod namespace. Raw store endpoints are network-isolated from team-* pods. Per-tenant limits are a security control. Grafana datasource-proxy locked (SSRF). Secrets via External Secrets.

Phases

Track A — Core stores & UI

• P1 — Hub: Grafana + metrics (dogfood platform). kube-prometheus-stack; observability ns (+default-deny) + Kyverno exclude; dashboards-as-code + Grafana hardening; Tailscale-only; minimal critical SNS route + Watchdog. DONE / live. (SSO is a fast-follow — see below.)
• P2 — Hub: Mimir. Monolithic + S3 + IRSA; auth_enabled + per-tenant limits + store isolation; hub Prometheus remote_write → Mimir. DONE / live (PR feat(observability): Prometheus remote_write → Mimir + durable PVCs (#102 P2) #147).
• P3 — Hub: Loki + Tempo → P3 — Hub: Loki + Tempo (logs + traces stores) #582. S3 + IRSA; auth_enabled + limits + collector-side redaction; Alloy + OTel write platform telemetry; trace→logs. (Splittable P3a/P3b.) ← next up.
• Grafana SSO (P1 fast-follow) → Grafana SSO (Dex/Keycloak) — P1 fast-follow #592. Replace admin-password login with Keycloak/IC OIDC.

Track B — Make it actionable

• P4 — Alerting & incident response (full) → P4 — Alerting & incident response (full set) #583. Mimir ruler + curated PrometheusRules + severity routing + inhibition + runbooks; Alertmanager → SNS/Slack/PagerDuty; K8s events→Loki; deploy annotations.

Track C — Cloud-resource observability (AWS)

• P5 — Cloud resources → P5 — Cloud-resource observability: remaining dashboards + alert (ADR-079) #584. Grafana CloudWatch datasource + YACE/cloudwatch-exporter→Mimir + CloudWatch Logs→Loki (RDS/ALB/VPC-flow/CloudTrail).

Track D — Developer experience / APM

• P6 — APM correlation → P6 — APM correlation (RED + service graph + exemplars) #585. Tempo metrics-generator → RED + service graph into Mimir; exemplars; metrics↔traces↔logs.
• P7 — Zero-code auto-instrumentation → P7 — Zero-code auto-instrumentation (OTel Operator + Beyla) #586. OTel Operator (auto-inject) + Beyla (eBPF).
• P8 — Continuous profiling (Pyroscope) → P8 — Continuous profiling (Pyroscope) #587. S3 + IRSA + limits + HA toggle; traces↔profiles.

Track E — Reliability

• P9 — SLOs & synthetics → P9 — SLOs & synthetics (blackbox + k6 + Sloth/Pyrra) #588. blackbox-exporter + k6 → Mimir; Sloth/Pyrra burn-rate error-budget alerts. (Unblocks P2.4 (ADR-056): progressive delivery for prod — Argo Rollouts #500 — Argo Rollouts auto-rollback.)

Track F — Multi-cluster reach & economics

• P10 — Onboard preprod spoke (all signals) → Onboard preprod as an observability spoke (hub-and-spoke, P10) #150. Authenticated cross-cluster ingest; collectors stamp X-Scope-OrgID=<team> + cluster=preprod; instrument app-alpha.
• P11 — Cost → P11 — Cost (OpenCost + CUR→Athena) #589. OpenCost per cluster → Mimir + AWS CUR → Athena → Grafana.
• P12 — Policy reporting → Kyverno PolicyReport observability (Prometheus/Grafana + policy-reporter) #93. policy-reporter per Kyverno cluster → Mimir; violation/severity/verifyImages dashboards. Supersedes / closes Kyverno PolicyReport observability (Prometheus/Grafana + policy-reporter) #93 on completion.

Track G — Tenancy & self-service

• P13 — Full per-team isolation → P13 — Full per-team tenant isolation (all signals) #590. Per-team Grafana access; datasources pinned to X-Scope-OrgID; IC group-GUID gating; teams.hcl observability block.
• P14 — Self-service & golden path → P14 — Self-service & golden path (+ Backstage tie-in) #591. Per-team dashboards/alerts/SLOs auto-provisioned from teams.hcl; documented app conventions; Backstage tie-in (Developer portal / service catalog (Backstage) #103/Golden-path scaffolding (create-new-app/tenant templates) #104/BACK stack P6: Backstage plugins (ArgoCD, Crossplane, Kyverno, Grafana) #177).

Ordering

Hard ordering: A → (B,C,D,E depend on the P2/P3 stores) → P10 spoke → P11–P14. D/E can slip after P10 if capacity is tight. Next up: P3 (#582).

[gangster]

P1 (Grafana + metrics hub) — applied + verified on the platform cluster (PR #140 merged).

• kube-prometheus-stack 86.1.0 up: 8 pods Running (Prometheus, Alertmanager, Grafana, operator, kube-state-metrics, 3× node-exporter).
• 29 Prometheus targets, all up — EKS-disabled control-plane jobs correctly absent (no false 'target down'); Kyverno metrics scraped via ServiceMonitor.
• 24 bundled dashboards loaded; Watchdog firing (pipeline heartbeat).
• Alertmanager → SNS → email proven end-to-end (IRSA sigv4 publish; subscription confirmed).
• Grafana reachable Tailscale-only at grafana.aws.refplat.org (admin via Secrets Manager; fixed a Cilium-gateway ingress-identity NetworkPolicy gotcha).

Remaining P1 fast-follows: SSO (Dex + Identity Center SAML, pending the manual IC SAML app) and curated dashboards (Platform Health overview + vendored ArgoCD/Cilium/cert-manager/ESO). Then P2 — Mimir.