Onboard preprod as an observability spoke (hub-and-spoke, P10)

[gangster]

Gap

The preprod cluster is not monitored by the observability hub. Per the hub-and-spoke design (ADR-043 / ADR-044, and the #102 plan P10), preprod should ship its metrics to the platform hub's Mimir, not run its own stack.

Explicitly NOT

• Do not install a full kube-prometheus-stack + Grafana + Mimir on preprod. A second hub defeats the single-pane-of-glass and the entire reason ADR-044 chose Mimir (multi-tenancy). Conform to hub-and-spoke.

Scope (the spoke)

• A lightweight collector on preprod — prometheus-agent (or Grafana Alloy) — that scrapes preprod and remote_writes to the platform hub's Mimir under its own tenant (X-Scope-OrgID: preprod).
• Cross-cluster path: reach the hub Mimir gateway over Transit Gateway / Tailscale (private).
• Security (the ADR-044 caveat): ingest from spokes must be authenticated, and X-Scope-OrgID must be overwritten at the hub edge so a spoke cannot spoof another tenant (e.g. platform). The hub Mimir gateway / an auth proxy enforces this.
• Per-tenant limits for the preprod tenant.
• Grafana: preprod metrics queryable via the existing Mimir datasource (tenant selector).

Acceptance

• Preprod metrics visible in the platform Grafana under tenant preprod; remote_write healthy (0 failures); a spoke cannot read/write tenant platform (negative test).

Refs

ADR-043, ADR-044 (security model), #102 plan P10, observability-current-state.

Top priority — next up.