Common Prometheus DB for Network Observer and Console

[c-kruse]

Background: how the pieces fit today

The observer, the console, and Prometheus are deployed as one pod. The pod is the containment boundary: the only data that the console can query comes from that network-observer instance.

flowchart TB
  subgraph podA["network-observer pod"]
    direction LR
    conA["Console"] -- "raw PromQL" --> obsA["network-observer<br/>/metrics<br/>/api/v2alpha1/internal/prom"]
    obsA -- "reverse proxy, unmodified" --> promA[("Prometheus<br/>sidecar")]
    promA -. "scrapes /metrics" .-> obsA
  end
  podA

Loading

What breaks with a common Prometheus

Point multiple observers at one shared Prometheus and that containment is broken. Several things go wrong.

flowchart TB
  obsA1["observer A1<br/>network A"] -- "skupper_*" --> prom
  obsA2["observer A2<br/>network A (HA replica)"] -- "skupper_*" --> prom
  obsB["observer B<br/>network B"] -- "skupper_*" --> prom
  other["kube-state-metrics,<br/>node-exporter, …"] -- "kube_*, node_*, up" --> prom

  prom[("Common Prometheus<br/>one TSDB, every tenant")]

  conB["Console for network B"] -- "1. sum(rate(skupper_sent_bytes_total[2m]))<br/>matches A1 + A2 + B" --> obsB
  obsB -- "2. proxied unmodified,<br/>any metric, any tenant" --> prom

  linkStyle 4,5 stroke:#d33,stroke-width:3px
  style prom fill:#fff3cd,stroke:#d9a400

Loading

1. No network scoping. skupper_sent_bytes_total now holds series from A1, A2, and B. The console's sum(...) adds network A's traffic into network B's dashboard, and counts network A twice (A1 and A2 each report the whole network).
2. No authorization on the query. The proxy forwards whatever the console sends. Any user who can log in to network B's console can run up, kube_pod_info, or network A's skupper_* series against the shared database.

Proposal

Two labels on the write side; one proxy rule on the read side.

Write side: label every series

flowchart LR
  obsA1["observer A1"] -- "network_id=A<br/>observer_id=A1" --> prom
  obsA2["observer A2"] -- "network_id=A<br/>observer_id=A2" --> prom
  obsB["observer B"] -- "network_id=B<br/>observer_id=B" --> prom
  prom[("Common Prometheus")]

Loading

label	question it answers	how queries use it
network_id	Whose data is this?	Filter on it. Partition key between networks. Defaults to default; system sites derive it from the namespace.
observer_id	Who reported it?	Aggregate it away. Every observer reports the whole network, so HA replicas produce duplicate series. max without(observer_id) (...) collapses them; sum would double-count.

Read side: the proxy parses, allow-lists, and scopes

sequenceDiagram
  participant C as Console (network A)
  participant P as network-observer<br/>prom proxy
  participant DB as Common Prometheus

  C->>P: sum(rate(skupper_sent_bytes_total[2m]))
  activate P
  Note over P: parse PromQL
  Note over P: every metric name in skupper_* allow-list? yes
  Note over P: inject or assert network_id="A" on every selector
  P->>DB: sum(rate(skupper_sent_bytes_total{network_id="A"}[2m]))
  DB-->>P: result
  deactivate P
  P-->>C: result

  C->>P: up{job="kube-state-metrics"}
  activate P
  Note over P: parse PromQL
  Note over P: metric "up" not in allow-list
  P-->>C: 403 Forbidden
  deactivate P

  C->>P: skupper_sent_bytes_total{network_id="B"}
  activate P
  Note over P: metric allowed, but network_id does not match this observer
  P-->>C: 403 Forbidden
  deactivate P

Loading

Console changes (optional)

The console mostly works unchanged: the proxy injects network_id. In order to correctly handle multiple observer instances on the same network (HA), the console will need to be updated to aggregate observer_ids away.

flowchart LR
  a1["observer_id=A1<br/>rate = 100"] --> q
  a2["observer_id=A2<br/>rate = 100"] --> q
  q{{"max without(observer_id) (sum)"}} --> r["rate = 100"]
  q2{{"sum"}} --> r2["rate = 200 ✗"]
  a1 -.-> q2
  a2 -.-> q2
  style r2 fill:#f8d7da,stroke:#d33
  style r fill:#d4edda,stroke:#2a7

Loading