bug: can not see the label apisix_llm_prompt_tokens and apisix_llm_completion_tokens from the metrics endpoint

[huanghuang0913]

Current Behavior

When I open http://127.0.0.1:9080/apisix/prometheus/metrics, I can only find the information of Labels for apisix_llm_active_connections, but can not find apisix_llm_prompt_tokens and apisix_llm_completion_tokens.
for example: apisix_llm_active_connections{route="JiTuan",route_id="ai-proxy-route",matched_uri="/v1/chat/completions",matched_host="",service="",service_id="",consumer="",node="ai-proxy-openai-compatible",request_type="ai_stream",request_llm_model="Qwen/Qwen32B",llm_model="Qwen/Qwen32B"} 0
but I can see the information about llm_completion_tokens and llm_prompt_tokens from access.log.

Expected Behavior

I can find the new labels releated to ai added by the newest version. https://apisix.apache.org/docs/apisix/plugins/prometheus/#labels-for-apisix_llm_completion_tokens
https://apisix.apache.org/docs/apisix/plugins/prometheus/#labels-for-apisix_llm_prompt_tokens

Error Logs

No response

Steps to Reproduce

Run APISIX via source code
Run Prometheus via docker
send the ai request via apisix successully
check the endpoint and find the label

Environment

• APISIX version (run apisix version): 3.14
• Operating system (run uname -a): x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V):1.21.4.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):bitnami/etcd:3.5.11
• APISIX Dashboard version, if relevant:apache/apisix-dashboard:latest
• Plugin runner version, for issues related to plugin runners:prom/prometheus:v2.25.0
• LuaRocks version, for installation issues (run luarocks --version): 3.12.0

[Baoyuantop]

Hi @huanghuang0913, how is your Prometheus plugin configured?

[huanghuang0913]

I found the discussion: #12710, then add "prometheus": {} into plugins in the route, now it works, but if this can be added into our document, it would be great!