Prometheus Queries are very slow

[jroshanfdo]

What is the issue?
Prometheus queries are very slow. We have a retention time of 72hours and the queries we pretty much fast for the first two months. Now we see that the same queries are taking so long up to 50 seconds. When i keep executing the same query again and again , i see that the response time varying every time. The number of metrics has also not increased.
Does this has something to do with Rule evaluation.
Because i see that 'prometheus_evaluator_duration_seconds' has gone upto 100 seconds.

What did you expect to see?
Prometheus Queries to be fast.

• Prometheus version:
  1.6.1

[fabxc]

If your retention is 72h, there should be now slowdown creeping up over the course of 2 months. Is it possible that you are putting more querying load onto it than in the past, e.g. via dashboards and more rules?

[jroshanfdo]

Yes we re doing a lot with queries and alerting. We did a restart for one of the instance and it worked fine after which the 'prometheus_evaluator_duration_seconds' has come down to ~20 seconds. Is the rule evaluation causing the slowness or is it just a symptom of some other issue.
Would be great if you can explain us whats causing the slowness in few lines.

[lueliu]

I also have problems with slow queries.
But my problem is a little bit more strange.
There is only one metric that is very slow to query. There is no function used in the query. The metric name and tag are used only in the query.
It takes 30 seconds to query.
(Load time: 32445ms
Resolution: 14s ;
Total time series: 19;)

Then I used other metric name (metrics size same)to query very quickly, all in milliseconds.
(Load time: 77ms
Resolution: 14s
Total time series: 16)

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.