Instrument Prometheus metrics across the broker

[haasonsaas]

Summary

No metrics instrumentation exists. For a security-critical system, observability into request rates, error rates, grant decisions, and connector health is non-negotiable.

Required work

• Add Prometheus client dependency and /metrics endpoint
• Request metrics: counter and histogram by endpoint, method, status code
• Session metrics: gauge of active sessions by tenant, counter of sessions created/revoked/expired
• Grant metrics: counter by outcome (issued, denied, pending, revoked, expired), histogram of grant TTLs
• Approval metrics: counter by outcome (approved, denied, expired), histogram of approval wait time
• Artifact metrics: gauge of active artifacts by connector kind, counter of unwrap operations
• Connector metrics: counter and latency histogram per connector per operation (GitHub API calls, Vault credential fetches, lease revocations)
• Policy metrics: counter of policy evaluation outcomes (allowed, denied) by capability
• Budget metrics: counter of budget exhaustion events per handle
• Cleanup worker metrics: counter of items processed per type per pass, histogram of pass duration
• Storage metrics: Postgres pool stats (active, idle, wait), Redis command latency

Files

• New package: internal/metrics/ with metric registration
• internal/api/httpapi/server.go — middleware for request metrics
• internal/app/service.go — instrument decision points
• internal/connectors/*/ — instrument external calls
• internal/worker/runner.go — instrument cleanup passes
• cmd/asb-api/main.go — register /metrics handler

Priority

High — required for operational visibility.

🤖 Generated with Claude Code

[haasonsaas]

Request-level Prometheus instrumentation landed in #34: ASB now exposes /metrics through service-runtime/observability, records HTTP request counters and latency histograms, and wraps the API with shared request IDs plus request logging. Remaining scope on this issue is the domain-specific instrumentation: sessions, grants, approvals, artifacts, connectors, policy, budgets, cleanup worker, and storage stats.

[haasonsaas]

Another metrics slice landed in #36: ASB now registers Postgres pool gauges on /metrics through service-runtime/observability.RegisterDBStats whenever ASB_POSTGRES_DSN is configured. Remaining scope here is still the domain and dependency metrics: sessions, grants, approvals, artifacts, connectors, policy, budgets, cleanup worker, and Redis/runtime storage stats.

[haasonsaas]

Storage metrics are now fully covered across #36, #37, and #38:

• Expose ASB Postgres pool metrics #36 added Postgres pool gauges on /metrics
• Expose Redis pool metrics #37 added Redis pool gauges on /metrics
• Record Redis command latency metrics #38 added Redis command counters and latency histograms via the shared service-runtime Redis hook

Remaining scope on this issue is the domain-side instrumentation only: sessions, grants, approvals, artifacts, connectors, policy evaluations, budget exhaustion, and cleanup worker passes.

[haasonsaas]

Another metrics slice landed in #39 and #40:

• Instrument session and grant metrics #39 added session, grant, and approval Prometheus metrics in the app layer
• Instrument policy and budget metrics #40 added policy evaluation outcome counters by capability plus proxy budget exhaustion counters by handle

Remaining scope on this issue is now narrowed to the remaining domain instrumentation only: artifacts, connector operations/latency, and cleanup worker pass metrics.

[haasonsaas]

Another metrics slice landed in #41:

• active artifact gauges are now emitted by connector kind
• artifact unwrap counters are now emitted by connector kind
• grant issue / unwrap / revoke / expire paths now keep artifact gauges accurate

Remaining scope on this issue is now just connector operation metrics and cleanup worker pass metrics.

[haasonsaas]

The remaining cleanup worker metrics seam landed in #43:

• cleanup item counters by type are now recorded in the worker
• cleanup pass duration histograms are now recorded in the worker
• cmd/asb-worker now exposes an optional metrics endpoint via -metrics-addr or ASB_WORKER_METRICS_ADDR

That completes the Prometheus scope tracked in this issue, so I’m closing it.