Request per Second Metric Does Not Sync with Total Request Count in Mimir Visualization

[azlan001]

Describe the bug

I am using OpenTelemetry for sending metrics to Mimir, and visualizing them on Grafana. The metric name is http_server_duration_milliseconds_count. When querying the request per second, the results do not sync with the total request count. For example, if the request per second rate is 4 requests per second for 10 minutes, the total requests should be 2400, but this is not reflected

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir: Version 5.1.0.
2. Perform Operations:
   • Send metrics using OpenTelemetry collector version 0.74.1.
   • Use the following query to check the total number of requests:

     sum(http_server_duration_milliseconds_count{job="${namespace}/${service_name}", cluster="$cluster", http_route!="<endpoint path>", http_route!="<path>*", http_route!="<path>**", http_route!="<path>", http_route!="<path>", http_route!="/ready", net_host_name=~".<hostname>"})

   • Use the following query to check requests per second:

     sum by(http_route)(irate(http_server_duration_milliseconds_count{job="${namespace}/${service_name}", cluster="$cluster", http_route!="<endpoint path>", http_route!="<path>*", http_route!="<path>**", http_route!="<path>", http_route!="<path>", http_route!="/ready", net_host_name=~".<hostname>"}[1m]))

Expected behavior

The total number of requests should continuously increase and not show any downfall. The requests per second should sync with the total request count, such that if the request rate is 4 requests per second for 10 minutes, the total requests should be 2400.

Actual behavior

The total number of requests shows a downfall in the visualization, which is not expected. The requests per second do not sync with the total request count.

Environment

• Infrastructure: Kubernetes
• Deployment tool: Helm
• Operating System: [Amazon Linux 2]
• Mimir Version: 5.1.0
• OpenTelemetry Collector Version: 0.74.1