bug: apisix prometheus metrics lost, must restart apisix pods

[cataglyphis]

Current Behavior

• Using Prometheus agent to scrap metrics from apisix pods, and metrics lost about 3 days (no apisix metrics any more)
• After restart apisix pods, everything works well
• This issues occurs after we upgrade apisix from 3.2 to 3.8

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1. Run apisix 3.8.1 in AWS EKS cluster
2. Use Prometheus agent to scrape metrics from apisix pods

Environment

• APISIX version (run apisix version): 3.8.1
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

Hi @cataglyphis, are there any error logs?

[cataglyphis]

Hi @cataglyphis, are there any error logs?

is this error logs related? @Baoyuantop

2025/04/01 20:09:48 [error] 74#74: *127311366 [lua] prometheus_resty_counter.lua:39: increasing counter in shdict: lru eviction: key=http_latency_bucket{type="request",route="env-prod-api_api",service="",consumer="",node="10.170.104.3",le="0050.0"}, context: ngx.timer
2025/04/01 20:09:48 [error] 74#74: *127311366 [lua] prometheus_resty_counter.lua:39: increasing counter in shdict: lru eviction: key=bandwidth{type="egress",route="ing_xxx_mall-portal_7202e927",service="",consumer="",node="10.170.159.243"}, context: ngx.timer

[Baoyuantop]

From the error logs, these two errors are related to the http_latency and bandwidth metrics respectively, indicating that these metrics generate a large number of key values under a large number of routes and nodes, resulting in insufficient space in shared memory.

You can try increasing the size of the prometheus-metrics shared dictionary, or using the metrics' expire configuration to reduce the number of metrics stored for the long term, and restarting APISIX to take effect

[cataglyphis]

From the error logs, these two errors are related to the http_latency and bandwidth metrics respectively, indicating that these metrics generate a large number of key values under a large number of routes and nodes, resulting in insufficient space in shared memory.

You can try increasing the size of the prometheus-metrics shared dictionary, or using the metrics' expire configuration to reduce the number of metrics stored for the long term, and restarting APISIX to take effect

the metrics' expire configuration: apisix 3.8 has this configuration yet?

[cataglyphis]

From the error logs, these two errors are related to the http_latency and bandwidth metrics respectively, indicating that these metrics generate a large number of key values under a large number of routes and nodes, resulting in insufficient space in shared memory.

You can try increasing the size of the prometheus-metrics shared dictionary, or using the metrics' expire configuration to reduce the number of metrics stored for the long term, and restarting APISIX to take effect

@Baoyuantop
I have another question: the value of apisix_shared_dict_free_space_bytes for Prometheus metrics is currently almost 0, while the default capacity is set to 10 MiB. What is the recommended value for the shared dictionary space for Prometheus?

[cataglyphis]

After reviewed the helm chart of Apisix, there is no way to set shared dict space for prometheus-metrics in values, we must add below config in apisix configmap manually.

nginx_config:
  ...
  http:
    ...
    lua_shared_dict:
      prometheus-metrics: 100m
    ...

[Baoyuantop]

@Baoyuantop I have another question: the value of apisix_shared_dict_free_space_bytes for Prometheus metrics is currently almost 0, while the default capacity is set to 10 MiB. What is the recommended value for the shared dictionary space for Prometheus?

This needs to be set according to the number of requests you have, see the #11948 (comment)