Promethues increase functions shows difference in 100s during heavy load

[thiyagu19841]

Hi,

I have been using prometheus for our app metrics to get the total count of transactions for given time range. I just played around in grafana and thought single stat panel will fit for our requirements; I just used increase/rate function to find difference between current scrape and last scrape for the given time range. I noticed that it is working fine and getting value near by accuracy ( i.e, + or - 2) when ever I tried with small amount of load; bu i am seeing big difference (in 100's) if test with big load.

I agree that increase/rate function will calculate the difference with extrapolation; hence we are seeing differences some time; but not good for 100s or 300s count difference. I thought of using max_over_time and min_over_time for my requirement and it is working fine; but it will also have problem if counter resets the value 0. I just verified all the community sites and everyone is proposing to go for metrics logs for accuracy result. Hence I tried to use irate function on metrics logs in graph; but it shows difference only for specific interval and not for the total difference for complete range

As I am exploring prometheus and grafana recently, I am looking for some guidance to come up with some query which will give approximate (but not big difference) transactions for the given range. Please note that our metrics will get scrapped for every 1 minutes by prometheus and we are expecting last one hour transactions counts from the scrapped metrics. Kindly help.

Thanks and regards,
Thiyagarajan K.

[roidelapluie]

Thanks for your report. It looks as if this is actually a question about usage and not development.

To make your question, and all replies, easier to find, we suggest you move this over to our user mailing list, which you can also search. If you prefer more interactive help, join or our IRC channel, #prometheus on irc.freenode.net. Please be aware that our IRC channel has no logs, is not searchable, and that people might not answer quickly if they are busy or asleep. If in doubt, you should choose the mailing list.

If you think this is not purely a support question, feel free to comment in here or take the underlying issues to our developer mailing list.

Once your questions have been answered, please add a short line pointing to relevant replies in case anyone stumbles here via a search engine in the future.

In your specific use case, we will need a bit more information to help you, for exemple, the console result of your_metric[2h] and increase(your_metric[1h]).