feat(obs): expose Prometheus /metrics on a managed DP — decouple to a dedicated metrics listener + integration docs

[moonming]

Problem

ai-gateway#284 shipped the aisix_* Prometheus metrics, but the metrics egress is broken in the managed topology — which is the real root cause behind AISIX-Cloud#555 ("the dashboard lacks a Prometheus integration UI/guide"):

• GET /metrics is registered only on the admin listener (crates/aisix-admin/src/lib.rs, build_router; it is unauthenticated — the module doc notes the scalar UI is "unauthenticated like /metrics"). But managed mode never binds the admin listener: config.managed.yaml pins admin to the non-bindable 127.0.0.1:0 (config.managed.yaml:40-46, with the comment "the admin listener is never [brought up]"), corroborated by AdminConfig::default_addr in crates/aisix-core/src/config.rs ("Intentionally non-routable. Managed-mode configs never bind this"). Yet that same config.managed.yaml still sets observability.metrics.prometheus.enabled + path: /metrics (:48-55). ⇒ the metrics endpoint is configured but stranded on an unbound listener — a managed DP's /metrics cannot be opened on any port.
• The DP also has no metrics push pipeline as a fallback: crates/aisix-obs/src/otlp.rs's install_otlp_tracer is "a no-op" / deferred (see its own module doc), crates/aisix-obs/src/otlp_http_sink.rs emits traces only (resourceSpans / scopeSpans), and crates/aisix-obs/src/lib.rs exports only the tracer — there is no MeterProvider / PeriodicReader / MetricExporter anywhere. MetricsConfig.otlp (OtlpConfig) is dead config consumed by nothing.
• Consequently the claim in docs/operations/metrics-and-logs.md that a managed DP exports metrics via OTLP does not match the implementation, and contradicts PRD prd-09 §9.6.4 (which envisions a managed DP exposing /metrics on localhost:9090 for the customer's own monitoring).

Bottom line: a managed DP — the mainstream product topology — has no usable Prometheus egress today: it can be neither scraped nor pushed.

Topology premise: the DP runs inside the customer's own network, outbound-only to the CP; the CP cannot reach the DP. So the correct direction is the customer scrapes the DP's /metrics locally, inside their own network — not a CP-mediated relay. Each DP is single-customer, so the scrape path has no multi-tenant metric-mixing problem.

Goal / acceptance

Make a managed DP scrapeable by the customer's own Prometheus, with no CP / dashboard change required:

• In managed mode, GET {path} is served on a dedicated, bindable metrics listener, enabled by default.
• That listener is decoupled from the admin listener — admin stays unbound in managed mode (security posture unchanged).
• Configurable addr / path / enabled / optional auth (bearer or mTLS) / optional TLS; all overridable via AISIX_OBSERVABILITY__METRICS__PROMETHEUS__* env vars (consistent with the existing AISIX_MANAGED__* override mechanism).
• Self-host behaviour stays compatible (the existing e2e tests/e2e/src/cases/prometheus-metrics-e2e.test.ts stays green, or is adjusted to the new listener).
• New e2e: in managed mode, the dedicated listener serves aisix_* (the current e2e only covers the admin listener, not managed).
• Docs: a new customer integration guide + fix the inaccurate OTLP-metrics claim in metrics-and-logs.md (see below).

Proposed design (details left to implementation; keep it simple)

Add a listener dimension to PrometheusConfig (crates/aisix-core/src/config.rs, currently just { enabled, path }), illustrative:

observability:
  metrics:
    prometheus:
      enabled: true
      path: /metrics
      addr: "0.0.0.0:9090"     # new: dedicated metrics listener; bound by default in managed mode
      auth_keys: []            # new (optional): if non-empty, require Bearer / x-api-key
      tls: null                # new (optional)

• When addr is set: stand up a dedicated metrics listener; managed mode supplies a default bind address (e.g. 0.0.0.0:9090).
• When addr is absent + self-host: keep current behaviour (mounted on the admin router) for back-compat.
• Update Config::validate: when managed + metrics enabled, addr must be bindable.
• Reuse the existing cardinality guards (fix(metrics): bound endpoint/model labels against cardinality DoS (#451) #482: normalize_endpoint_label, passthrough sentinel) — no new work needed.

Why a dedicated port rather than reusing the proxy listener (:3000): the proxy is the data-path traffic port; metrics-and-logs.md itself says to "treat /metrics as infrastructure-facing, not a public diagnostics surface"; comparable gateways (APISIX's dedicated export server :9091, Envoy :19000) all use a dedicated metrics port.

Docs (part of this issue, customer-facing)

Add an "Integrate Prometheus with the AISIX data plane" guide covering:

• Managed DP: expose the metrics port in your own deployment (docker -p 9090:9090 / k8s containerPort + Service / ServiceMonitor), and point your own Prometheus at http://<dp-host>:9090{path}.
• Copy-pasteable Prometheus scrape_config and k8s ServiceMonitor snippets; auth (if enabled).
• Metric & label catalog: aisix_*, label set endpoint / inbound_protocol / provider / model / upstream_model / provider_key_id / api_key_id / team_id / user_id / status / outcome.
• Self-host still uses the existing local /metrics.
• Fix metrics-and-logs.md: drop / rewrite "managed exports metrics via the OTLP exporter" (not implemented); mark OTLP-metrics push as optional future work.
• Convention: customer docs must not cite private CP-repo issue numbers — reference roadmap items + link roadmap.md instead.

Key: what does the CP / dashboard need to do? — nothing

• A managed DP's bootstrap config comes from the CP-generated install snippet (dashboard/lib/install-snippets.ts in AISIX-Cloud): an image-baked config.managed.yaml + injected AISIX_MANAGED__CP_* env vars. The listener is bootstrap config, not a CP-delivered dynamic resource.
• Once the metrics listener is bound by default in managed mode, the customer only: upgrades the DP image + (in their own deployment) exposes the metrics port + points their own Prometheus at it. No CP config, no dashboard form, no CP-side resource anywhere in the loop.
• (Optional, not in this issue) a future nicety could have the install snippet also expose the metrics port — that's a dashboard change, out of scope under the current "no dashboard change" direction.

Out of scope

• Any dashboard UI (AISIX-Cloud#555 converges to "DP docs are enough").
• OTLP /v1/metrics metrics push (separate, optional, later phase).
• A CP-mediated relay (the CP can't reach the DP; not applicable).
• In-dashboard metric visualization (belongs to the usage-event analytics track, AISIX-Cloud#560).

References

• DP source: crates/aisix-obs/src/metrics.rs, crates/aisix-admin/src/lib.rs, crates/aisix-server/src/main.rs, crates/aisix-core/src/config.rs, config.managed.yaml, docs/operations/metrics-and-logs.md, tests/e2e/src/cases/prometheus-metrics-e2e.test.ts
• Related PRs: feat: add AISIX-native Prometheus metrics #331 (AISIX-native metrics), fix(metrics): bound endpoint/model labels against cardinality DoS (#451) #482 (cardinality DoS bound), docs(metrics): clarify empty /metrics before traffic and managed DP exposure (#478) #488 (managed / empty-metrics docs)
• Related: AISIX-Cloud#555 (this issue is its prerequisite / root cause)

[coderabbitai]

🔗 Related PRs

api7/a7#55 - docs: reconcile AGENTS.md with GA scope [merged]
api7/AISIX-Cloud#571 - test: Add DeepSeek observability CI coverage [merged]
api7/api7-ee-3-gateway#1643 - feat(prometheus): add MCP tool dimensions to HTTP metrics [merged]
api7/AISIX-Cloud#681 - feat(logs): surface client source IP, client type, and cache tokens in request logs (#492) [merged]
#536 - fix(managed): pin HTTP/1.1 on the dp-manager mTLS REST clients [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!