P3 — Hub: Loki + Tempo (logs + traces stores)

[gangster]

Part of the observability epic #102 (Track A — core stores). Full design: docs/plans/102-observability-stack.md.

Scope. Deploy Loki (logs) + Tempo (traces) on the platform hub: S3 + IRSA, auth_enabled + per-tenant limits + store-endpoint network isolation, collector-side redaction; hub Alloy + OTel Collector write platform telemetry (tenant _platform); Grafana datasources + trace→logs correlation. Mirrors the proven Mimir module pattern (ADR-044). Splittable into P3a (Loki) / P3b (Tempo).

Verify. Platform logs + traces queryable in Grafana; trace→logs correlation works; a forged X-Scope-OrgID from a tenant pod is network-denied.

Dependency / ordering. Needs P1+P2 (done). This is the prerequisite for P5/P6/P7/P8 and the agent-observability layer (ADR-076) — Tracks C–E all depend on these stores. It is the next phase up.

[gangster]

P3 build scope (finalized 2026-06-19)

Decisions locked (supersede the skeleton where they differ):

• Identity = EKS Pod Identity, not IRSA. P3 leads the ADR-047 standard (the existing IRSA add-ons migrate separately — Migrate platform add-on identity: IRSA → EKS Pod Identity (ADR-047) #594). Use the eks-pod-identity module (aws_eks_pod_identity_association): association maps (namespace, chart SA) → role (trust pods.eks.amazonaws.com); no OIDC sub trust condition, no eks.amazonaws.com/role-arn annotation.
• Collectors configurable per signal: log_pipeline (default Alloy) + trace_pipeline (default OTel Collector), each independently enableable (Alloy-for-both / OTel-for-both / either alone).
• Tempo topology = the standard high_availability toggle (single-replica ↔ distributed RF3 via the tempo-distributed chart) — same knob as Mimir/Loki; no separate chart-choice variable.
• Charts — latest stable (re-resolve at authoring): loki 7.0.0 · tempo-distributed 1.61.3 · alloy 1.10.0 · opentelemetry-collector 0.158.2. (Mimir 6.0.6 already latest; kube-prometheus-stack bump tracked in chore(observability): bump kube-prometheus-stack 86.1.0 → 86.3.2 #595.)

Modules (mirror observability-mimir, minus IRSA → Pod Identity)

1. observability-loki — grafana/loki 7.0.0, SingleBinary mode (→ SimpleScalable RF3 on HA via deploymentMode). In-module S3 chunks bucket <cluster>-loki-chunks- (SSE-S3, versioned, lifecycle, force_destroy). Pod-Identity role scoped to that bucket (List + Get/Put/Delete). auth_enabled + per-tenant limits + retention ~14d. Grafana Loki datasource ConfigMap (grafana_datasource: "1", X-Scope-OrgID header + derivedFields trace_id → Tempo).
2. observability-tempo — grafana/tempo-distributed 1.61.3 run single-replica (→ RF3 on HA). S3 traces bucket <cluster>-tempo-traces-. Pod-Identity role → bucket. auth_enabled + limits + retention ~7d. Grafana Tempo datasource ConfigMap (tracesToLogs → Loki).
3. Collector tier (greenfield):
   • Alloy DaemonSet (log_pipeline=alloy) → tail pod logs → Loki; stamps X-Scope-OrgID=_platform; best-effort redaction processor (low stakes at P3 — hub telemetry only).
   • OTel Collector gateway (trace_pipeline=otel) → OTLP receiver → Tempo; X-Scope-OrgID=_platform.
   • Collectors write to in-cluster Loki/Tempo Services (no S3) → typically no AWS role needed.

Live units (mirror the mimir unit)

.../platform/{loki,tempo,alloy,otel-collector}/terragrunt.hcl — include _base, deps on eks + node_groups + observability (collectors also dep loki/tempo), cost toggles (enable_loki/enable_tempo/enable_log_pipeline/enable_trace_pipeline), version pins from _versions.hcl, Pod-Identity association inputs (cluster, namespace, chart SA name).

_versions.hcl

Add helm_versions: loki=7.0.0, tempo=1.61.3, alloy=1.10.0, otel_collector=0.158.2; module_source: observability_loki, observability_tempo (+ collector module[s]).

Security

auth_enabled (multitenancy); per-tenant limits as a control; retention (logs 14d / traces 7d); store endpoints ClusterIP-only (no Gateway); observability ns already Kyverno-excluded + default-deny netpol; each store's Pod-Identity role scoped to its own bucket.

Testing & verify

Plan-only Terratest per module (convention). Acceptance: platform logs + traces queryable in Grafana; click a trace → its logs (trace→logs correlation); a forged X-Scope-OrgID from a tenant pod is network-denied. apply needs the cluster up (platctl up --env platform).

PR sequencing (the P3a/P3b split)

• P3a — observability-loki module + unit + Loki datasource + Alloy log pipeline + version pins. Gated apply → verify platform logs in Grafana.
• P3b — observability-tempo module + unit + Tempo datasource + trace→logs correlation + OTel trace pipeline. Gated apply → verify traces + trace→logs jump.

Not in P3 (deferred)

Per-team tenant derivation + cross-cluster ingest = P10 (#150). Metrics↔traces exemplars / RED / service graph = P6 (#585). Rigorous redaction once tenant data flows = later. P3 stays single-tenant _platform.

[gangster]

P3a Loki store — DEPLOYED + verified live (2026-06-19). loki-0 2/2 Running on platform (cost-effective single-binary), S3 chunks bucket + Pod-Identity (role assumed → S3 writes succeed), Grafana Loki datasource registered. PRs: #596 (module), #597 (cost_profile switch), #598 (enable on platform), #599 (S3 SSE fix).

Surfaced a stack-wide gotcha: the org SCP enforce-encryption (DenyUnencryptedS3Uploads) denies s3:PutObject without the x-amz-server-side-encryption header — bucket default encryption alone doesn't send it. Fixed for Loki via loki.storage.s3.sse.type=SSE-S3. P3b Tempo (and Mimir whenever it's enabled) must set the equivalent SSE config.

Remaining for P3a: the Alloy log pipeline (collector → Loki) — the store is up but no logs flow until the collector ships (next PR). Then P3b: Tempo + trace pipeline.