feat: supports integrating Prometheus for metrics statistics

[gxthrj]

background

Communicate with customers and collect AI gateway requirements, which need to support the integration of Prometheus.
Metrics only need to be provided in DP, and prometheus on the client side will actively capture DP metrics.

Competitive Research

LiteLLM supports integrating Prometheus and supports various metrics: https://docs.litellm.ai/docs/proxy/prometheus

Metrics list

Usage & Billing

Metric Name	Type	Description	Labels
litellm_spend_metric	Counter	Total cost	end_user, hashed_api_key, api_key_alias, model, team, team_alias, user
litellm_total_tokens_metric	Counter	input + output tokens	end_user, hashed_api_key, api_key_alias, requested_model, team, team_alias, user, model
litellm_input_tokens_metric	Counter	input tokens	Same as above
litellm_output_tokens_metric	Counter	output tokens	Same as above
litellm_requests_metric	Counter	Total requests by endpoint	end_user, hashed_api_key, api_key_alias, model, team, team_alias, user, user_email

Budget & Limits

Metric name	Description
litellm_team_max_budget_metric	Team maximum budget
litellm_remaining_team_budget_metric	Team remaining budget
litellm_team_budget_remaining_hours_metric	Team budget reset remaining hours
litellm_api_key_max_budget_metric	API Key maximum budget
litellm_remaining_api_key_budget_metric	API Key remaining budget
litellm_api_key_budget_remaining_hours_metric	API Key budget reset remaining hours
litellm_remaining_api_key_requests_for_model	The remaining number of API Key requests for a specific model (requires setting rpm limit)
litellm_remaining_api_key_tokens_for_model	API Key number of remaining tokens for a specific model (tpm limit needs to be set)

LLM Provider (deployment level)

Metric name	Description
litellm_deployment_success_responses	Number of successful LLM API calls
litellm_deployment_failure_responses	Number of failed LLM API calls
litellm_deployment_total_requests	Total LLM API calls
litellm_remaining_requests_metric	x-ratelimit-remaining-requests
litellm_remaining_tokens_metric	x-ratelimit-remaining-tokens
litellm_deployment_state	Deployment status (0=healthy, 1=partial failure, 2=complete failure)
litellm_deployment_latency_per_output_token	Latency per output token
litellm_deployment_cooled_down	Number of times cooled (fuse)
litellm_deployment_successful_fallbacks	Number of successful fallbacks
litellm_deployment_failed_fallbacks	Number of failed fallbacks

Delay

Metric name	Type	Description
litellm_request_total_latency_metric	Histogram	Request total latency (to Proxy)
litellm_overhead_latency_metric	Histogram	LiteLLM's own processing overhead
litellm_llm_api_latency_metric	Histogram	Pure LLM API call latency
litellm_llm_api_time_to_first_token_metric	Histogram	TTFT (streaming only)

Proxy Level

Metric name	Description
litellm_proxy_total_requests_metric	Total number of client requests
litellm_proxy_failed_requests_metric	Number of client failed requests
litellm_in_flight_requests	Current number of in-flight requests (Gauge)
litellm_callback_logging_failures_metric	Number of write failures to downstream callbacks (such as S3, Langfuse)

System health (need to enable prometheus_system)

Metric name	Description
litellm_redis_latency	Redis call latency (Histogram)
litellm_redis_fails	Number of Redis failures
litellm_self_latency	LiteLLM API successful call latency (Histogram)
litellm_pod_lock_manager_size	Obtain DB write lock Pod
litellm_in_memory_daily_spend_update_queue_size	Daily consumption update queue size in memory
litellm_redis_daily_spend_update_queue_size	Redis daily consumption update queue size
litellm_in_memory_spend_update_queue_size	Memory aggregate consumption queue
litellm_redis_spend_update_queue_size	Redis aggregate consumption queue

[coderabbitai]

🔗 Related PRs

api7/api7-ee-3-gateway#1656 - feat(prometheus): add LLM token distribution histograms and TTFT via llm_latency type label [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[jarvis9443]

我看了一下 LiteLLM 的 Prometheus 指标，建议这里不要直接照搬 litellm_* 名字，也不要伪造 AISIX 目前没有的数据。更合适的方向是暴露 AISIX-native 的 aisix_* 指标，覆盖 LiteLLM 里能对应上的类别。

目前 DP 侧可以支持的部分包括：请求总数、成功/失败数、HTTP status/outcome、请求总延迟、可测量到的 LLM API 延迟、streaming TTFT、input/output/total tokens、模型价格存在时计算出的 spend、proxy in-flight requests、rate-limit rejection、request/token remaining gauge、deployment health/cooldown、routing fallback success/failure，以及 telemetry/exporter drop/failure 这类导出链路指标。

部分支持的指标主要有两类：预算指标只有在 CP budget-check 响应返回 limit/spent/reset 这类字段时才能上报；team/member 相关 label 现在只能稳定使用 team_id、owner_id，不能提供 team_alias、email 这类展示字段。

暂时不建议支持的是：end_user、user、user_email、team_alias 等 DP 当前没有的 label；DP 内部持久化 per-team budget 状态；LiteLLM 自己的内部队列指标；AISIX 没有的 callback 系统指标；以及未在 AISIX 中配置的外部组件 system metrics。

我准备先做一组 aisix_* 指标来覆盖上面“可以支持”和“部分支持”的子集，同时保留现有指标兼容性。