Support auto step for annotations and $__interval & $__interval_ms variable for query field

[wernight]

Grafana 5.2.0 (but not a regression) + Prometheus.

If I put a query like: rate(my_backups_total[24h])*3600*24, I get a graph like:

Note that it drops right around the time there is a backup. However in Prometheus this graph looks fine:

There are various related issues, and the only way I get a kind of smooth graph is via a hack:
rate(my_backups_total[1441m])*3600*24 which is less readable and means that for 1 minute I may count backups twice.

The same issue happens for example for annotations, which disappear when the current time is just around one of these boundaries.

I tried fiddling with Min step, Min time interval, and Resolution in Graphana, but none fixed this issue completely. Looks to me as if there was an issue near time boundaries.

[torkelo]

can you show what the query is to Prometheus (inkl step) and the response? (Open Query Inspector in metrics tab), this shows all params sent to Prometheus and Prometheus response.

rate does have some issues, have you tried irate?

[wernight]

With irate it gives the opposite graph:

The counter increases every day by one, to mark a backup. Here is the query inspector for:

362:Array[1529895450,1.0001118064807732]
363:Array[1529895465,1.0004592269963362]
364:Array[1529895480,1.0001118064807732]
365:Array[1529895495,1.0004592269963362]
366:Array[1529895510,1.0001118064807732]
367:Array[1529895525,1.0004592269963362]
368:Array[1529895540,1.0001118064807732]
369:Array[1529895555,1.0004592269963362]
370:Array[1529895570,1.0001118064807732]
371:Array[1529895585,0]
372:Array[1529895600,0]
373:Array[1529895615,0]
374:Array[1529895630,1.0001118064807732]
375:Array[1529895645,1.000459238581043]
376:Array[1529895660,1.0001118180574355]
377:Array[1529895675,1.0004592269963362]
378:Array[1529895690,1.0001118064807732]
379:Array[1529895705,1.0004592269963362]
380:Array[1529895720,1.0001118064807732]
381:Array[1529895735,1.0004592269963362]
382:Array[1529895750,1.0001118064807732]

Yes for some reason the data goes to zero near that 24h mark.

Could be related to prometheus/prometheus#3746.

If I put a 60s frame in Prometheus I see the same spikes. I think it's a bug in prometheus then:

By the way I've a similar issue with annotations, where I try to annotate automatically when this counter increases but I could only get it to work with a delay (meaning quite after the event) and also has some rounding issues but those should be Prometheus:

I used annotation Query increase(my_backups_total[4m]) != bool 0 with step of 2m, because increase(pbex_backups_total[4m]) > 0 didn't work (showed no annotations).

[wernight]

I found a somewhat nicer version but still delayed by 5m:

• Query: my_backups_total - my_backups_total offset 5m > 0
• Step: 5m

Looks like Grafana is showing the line when the value is NOT anymore > 0.

[torkelo]

this looks more and more like a Prometheus issue, not sure what Grafana can do here.

[wernight]

What about annotations? I'm grouping them here because the same issues with rate apply to them, but possibly there is something else that can be done to show a single line when a value > 0.

[torkelo]

Well the annotation query is also executed by Prometheus, and seems to return data that you think is incorrect?

[wernight]

I'd say I need an annotation step size of 10m in Grafana to show a single line, so it can only be 10 min accurate. If I choose a value too small, it may miss some, or complain that there are too many results, and if it's too large, it's not accurate.

It has exactly 10m delay in this case.

I see now that it's really because of the mechanic of how annotations work in Grafana. There is concept of start and end time marking of a Prometheus query.

[torkelo]

I don't know if there is anything Grafana can do here, or is this an issue in Prometheus?

[wernight]

If I could set in Grafana to show a line when the lower graph goes up or has some values, yes. At the moment I can do that accurately only if the step size is relatively small, but if it's smaller than my range interval, then Grafana would start showing multiple annotations lines (as long as there are values > 0 and for each step size).

Also given that when you zoom out you want bigger step sizes, the step size should be like $__interval, i.e. scales with the time range. But $__interval doesn't seem to be supported.

[torkelo]

Think we need to add support for auto step for annotations and $__interval & $__interval_ms variable for query field

[davkal]

Is the problem just a head vs tail rendering issue on the 10m bucket? Anyway, not really a Prometheus issue.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!

[github-actions]

This issue has been automatically closed because it has not had any further activity in the last 30 days. Thank you for your contributions!