Generic rules causes incorrect graphs in grafana

[sergeyshevch]

Generic rules (pyrra_requests/errors_total) cause incorrect calculation in Grafana sum(rate(sum(metric)) instead of sum(rate(metric) that lead to spikes when one of counter resetting.

Article with explanation: https://www.robustperception.io/rate-then-sum-never-sum-then-rate/

Panel from SLO Detail dashboard:

Graph with comparison to correct metric calculation

Is it possible to overtake such issues in any way? Maybe store sum(rate() instead of sum and call metric as pyrra_requests_rate?

[sepulworld]

I see the same issue while using Grafana. Seem to correlate to counter resets

[sepulworld]

@sergeyshevch are you able overcome this by changing the expr in the panel? I tried to use quantile_over_time but it doesn't work with vector data. I think this has to be fixed in Pyrra generic rules code

[sergeyshevch]

Pyrra should store rate/increase data instead of sum than it will be fixed.
I've done query for workaround on similar case but it will only work on VictoriaMetrics

I will share it later on next week