Prometheus data source: Add $__rate_interval

[beorn7]

What would you like to be added:

A variable $__xinterval (“extended interval”) similar to the already existing $__interval. It's value is the value of $__interval plus a configurable duration, which should default to the Scrape interval configured in the Prometheus data source not multiplied by anything like the resolution factor!!!

To configure the additional duration explicitly, we need a new per-query input field in the query editor. It's tempting to call it Scrape interval, too, because that's what it will usually be (see below), but perhaps it's better to be more explicit about what it's doing and call it Interval extension or something. I'll leave that to the UX experts.

Why is this needed:

Wider context: this RP blog post.

Key paragraph: “Another consideration is that if you're using query_range, such as in graphing, then the range should be at least the size of the step. Otherwise you'll skip over some data. Grafana's $__interval can be useful here.”

To actually not skip over data, the range needs to be slightly larger than $__interval, ideally to create an overlap of one sample. (This is not true if there are already recording rules in the game, and then avg_over_time is used to extend the result to longer time spans, which is also described in the RP blog post. In this case, the current behavior works just fine.)

This “not skipping over data at all” was such a deep desire for some, that they proposed to change how PromQL handles range selectors specifically for rate expressions. What came out of that is a fork with something called xrate. This approach has issues of very different kinds. It would be great if we could solve it in Grafana, which then separates concerns quite nicely.

That's where this suggestion is coming from: Add a new variable just for the Prometheus data source $__xinterval which is (usually) one scrape interval longer than $__interval. Since the scrape interval for a given time series cannot be inquired by Grafana, we need to make it configurable per query (with the default being the scrape interval configured in the data source). This also opens the door for some other, more fringe use cases, where people wanted to use range selectors with a somewhat modified $__xinterval. (We are only offering adding a duration here, but should legitimate use cases come up where a factor is required, we could introduce that without a breaking change.)

Note: Other solutions to this problem were discussed, but Grafana has an obligation to not break existing dashboards. That's why the introduction of a separate variable was inevitable.

[beorn7]

In the spirit of conference driven development, I have submitted a talk to Grafanacon to explain how rate queries should be done in Grafana. It would be great if we could inaugurate $__xinterval during that talk.

[torkelo]

Not thrilled by this idea, seems to add a lot of complexity and confusion to a situation that is already too complex and confusing (step & interval & resolution). Need to rethink this a bit more ,maybe there are ways we can make the default behavior and concepts more widely usable and understandable. Or if there are ways to solve this in Prometheus?

Another option would be to always make $__interval = interval set by grafana (taking into account resolution, ds scrape interval, query step, etc) + ds scrape interval.
I guess there is problem with doing that as scrape intervals can be different for different metrics. But should not Prometheus know what the scrape interval is for each metric and can then add this amount to the interval automatically or some syntax? Then the user does not have to always know scrape interval for each metric and specify that per query.

[torkelo]

I agree that this is a problem we need to solve, but worried that we are making an already complex and messy solution worse.

[beorn7]

I had a long whiteboard session with @davkal and @SuperQ. To reduce the existing complexity and messiness, we needed changes that would make it hard to keep existing dashboards working. (One could do things like keep supporting the old settings but hide them in newly created panels, but @davkal explained to me that those kind of changes are highly problematic in the Grafana context.)

Another option would be to always make $__interval = interval set by grafana (taking into account resolution, ds scrape interval, query step, etc) + ds scrape interval.

We thought about that, but the problem is that there are a lot of situations where you want $__interval to be exactly what it is right now. Also, changing how $__interval is calculated would break existing dashboards. We also thought about an additional per-query setting to add a specified value to $__interval. But that would not provide a sane default. Thus, we settled on a new variable to make the difference explicit and also provide a usable default value.

I guess there is problem with doing that as scrape intervals can be different for different metrics.

Indeed. That's why the duration added to $__interval to get to $__xinterval needs to be configurable by query (and only defaults to the scrape interval configured in the data source).

But should not Prometheus know what the scrape interval is for each metric and can then add this amount to the interval automatically or some syntax?

Short answer is: For better or worse, Prometheus doesn't know the scrape interval (and neither the evaluation interval, if we talk about rules).

Longer answer: While the scrape and evaluation intervals are configured in Prometheus, they may change over time, and that change over time is not persisted in the TSDB (and it wouldn't be an easy change, neither in terms of implementation nor in terms of concepts, to add it to TSDB). On top of that, both scrapes and evaluations can be skipped or get delayed. Thus, PromQL expects the user to be explicit about it. The user needs to know the expected scrape interval and craft range durations accordingly (see the linked RP blog post for best practices). A syntax to "make something one scrape interval longer" would thus boil down to let the user explicitly fill in the expected scrape interval, which then degenerates to just providing a longer duration in the range selector. And that's what this issue suggests: Let's facilitate that in Grafana. Or looking at it from a slightly different angle: The "problem" really only appears because we have a dynamic duration in the range selector. But that dynamic range is a Grafana feature, provided via $__interval. Thus, adjusting this dynamic interval in the right way is also something that has to live in Grafana. (It's a different story why Prometheus doesn't have a feature for dynamic range durations built in. It has to do with the execution model, in which the /api/v1/query_range endpoint is essentially just syntactic sugar for a series of independant instant queries via /api/v1/query. And that's just the HTTP API. Pure PromQL expressions have no notion whatsoever that they would be executed for different regular time intervals to arrange the results in a graph. One might be of the opinion that this is a mistake, and the language should take into account the ways it might be used for graphing, but PromQL was designed differently. Changing basic ideas now will be hard.)

Another long answer: It has been viciously discussed before if the way Prometheus calculates a rate over a range should be changed (at least optionally) to include one data point before the range starts. If you google for "prometheus xrate", you'll get the gist of it. The idea met resistance because it would introduce a fundamentally different way of evaluating range selectors into the language for a use case that is mostly centered on graphing. It also introduces new emerging behaviors for corner cases (like irregular scrape/evaluation intervals), which are hard to understand and to reason with. (The existing range evaluation has those problems, too, but over many years, an understanding how to cope with them has evolved.) From there, we circle back to the idea of solving graphing concerns in Grafana. It replaces making PromQL more complicated (and arguable messier) in a very invasive way by a fairly light-weight addition to Grafana that merely is a variation on an already existing theme ($__interval and the various config levers to control it).

[beorn7]

For reference: The discussion leading to this issue was sparked by #19885 . The meandering discussion there makes it quite clear that there were way more involved changes on the table than merely introducing the $__xinterval. The idea of the latter is to address most use cases with a relatively minor addition.

[torkelo]

Ok, thanks for more context, that was very helpful.

But what would be nice was more info on how step fits into this (something many prometheus users fail to understand). Today $__interval is & step are the same, and in your first comment you said:

not multiplied by anything like the resolution factor!!!

But the resolution will impact step, which will impact $__interval, so having a different behavior for $__xInterval seems confusing/inconsistent. Why would $__interval not be the same as $__interval + scrape with exact same logic on how $__interval is calculated.

Maybe we can add support for math in range vectors so one could write vectors like[$__interval + 15s] or [$__interval + $__scrape] , would be a bit more difficult to implement but would not add a new input field and variable with a inconsisten evaluation logic.

Just brainstorming trying to see if all there is a more user friendly & intutive for new users way to solve this.

[beorn7]

To clarify: $__xinterval will still be (indirectly) affected by the resolution factor, as it is derived from $__interval. However, the difference between $__interval and $__xinterval must not be affected by the resolution factor.

[beorn7]

Maybe we can add support for math in range vectors

I'm fairly sure that adding this kind of math on the PromQL level would be very controversial. Math in variable expansions (on the Grafana level) would be an option, as discussed in #19885, but it would be a quite involved change compared to the $__xinterval suggestion here.

[dmgtn]

Hi! About two years ago I've opened #11451. Now we live with patched Grafana, which has two extra variables for Prometheus:

• $__interval_sx3 – it's $__interval, but not less than scrape interval x 3. We use it for all queries requiring at least one point (like avg_over_time), and the number 3 goes from — 1 point required, 1 point for "metrics in fly", and 1 point extra as an allowance for missing one scrape.
• $__interval_sx4 – it's $__interval, but not less than scrape interval x 4. We use it for all queries requiring at least two points (like rate and increase).

Here is the patch we use now (it has some legacy things, but you will get the idea):

diff --git a/public/app/plugins/datasource/prometheus/datasource.ts b/public/app/plugins/datasource/prometheus/datasource.ts
index e6c79faa42..55668ac4f7 100644
--- a/public/app/plugins/datasource/prometheus/datasource.ts
+++ b/public/app/plugins/datasource/prometheus/datasource.ts
@@ -356,6 +356,15 @@ export class PrometheusDatasource extends DataSourceApi<PromQuery, PromOptions>
         ...this.getRangeScopedVars(options.range),
       });
     }
+    const intervalScrapeX3ForOverTimeVectors = Math.max(kbn.interval_to_seconds(this.interval) * 3, interval);
+    const intervalScrapeX4ForRangeVectors = Math.max(kbn.interval_to_seconds(this.interval) * 4, interval);
+    const intervalForRangeVectors = Math.max(kbn.interval_to_seconds(this.interval) * 2, interval);
+    scopedVars = Object.assign({}, scopedVars, {
+      __interval_rv: { text: intervalForRangeVectors + 's', value: intervalForRangeVectors + 's'},
+      __interval_sx3: { text: intervalScrapeX3ForOverTimeVectors + 's', value: intervalScrapeX3ForOverTimeVectors + 's'},
+      __interval_sx4: { text: intervalScrapeX4ForRangeVectors + 's', value: intervalScrapeX4ForRangeVectors + 's'},
+    });
+
     query.step = interval;
 
     let expr = target.expr;
diff --git a/public/app/plugins/datasource/prometheus/promql.ts b/public/app/plugins/datasource/prometheus/promql.ts
index 0bde2da9c2..a63a0ac707 100644
--- a/public/app/plugins/datasource/prometheus/promql.ts
+++ b/public/app/plugins/datasource/prometheus/promql.ts
@@ -2,6 +2,8 @@ import { CompletionItem } from '@grafana/ui';
 
 export const RATE_RANGES: CompletionItem[] = [
   { label: '$__interval', sortText: '$__interval' },
+  { label: '$__interval_sx3', sortText: '$__interval_sx3' },
+  { label: '$__interval_sx4', sortText: '$__interval_sx4' },
   { label: '1m', sortText: '00:01:00' },
   { label: '5m', sortText: '00:05:00' },
   { label: '10m', sortText: '00:10:00' },

[roidelapluie]

Thank you for your input @distol . Sounds like something worth investigating.

[beorn7]

Note that @distol 's suggestion addresses a different problem. It's about not having a duration in the range selector that doesn't get you enough points.

This issue is about the problem of having "full coverage" of data points, for which you need a duration in the range selector that is equal to the interval plus one scrape interval.

Having said that, @distol 's patch also has to face the problem that the scrape interval configured in the data source might not be the scrape interval of the concrete series displayed. A generally applicable solution needed a separate setting for the scrape interval per query (which is somewhat similar to the separate setting we need for the xinterval suggestion). However, once you set the scrape interval separately, it is not much less convenient to set the min-interval separately (which solves @distol 's problem, but not the problem this issue is about).

Having some support for arithmetic in the variables would help in both cases.

[dmgtn]

Having hundreds Prometheus in operation we end up strongly forcing one scrape interval across all metrics in one Prometheus. This approach simplifies a lot of things and it helped us a lot.

[beorn7]

Yes, that's definitely a good practice and also recommended, but it's not generally true. (Just saying that because we need to find a solution that's easy in the common case but also still works in the special case.)

[dmgtn]

Okay, then maybe the general solution would be:

1. Introduce variables __interval_sx3 and __interval_sx4 with the behavior described in above mentioned post (but with some more meaningful names, of course).
2. Introduce variable __interval_plus_scrape, if the first two don't solve all the cases.
3. Introduce the ability to override the scrape interval (specified in the datasource) on a per-query basis (it might be convenient to allow doing it also on a per-dashboard basis, but I'm not sure how it fits).

IMHO it solves all the problems. What do you think?

This way, if you follow the practice of one-scrape-interval-per-prometheus — you can just use special vars for range vectors, and if you don't — you have to specify scrape interval on a per-query basis.

[beorn7]

So… what I propose in this issue is essentially (2) and (3) from your list.

The good thing about it is that it doesn't prevent us from later doing (1). (That's why I'd like to not overload this issue. It might already appear pretty invasive to the core Grafana folks.)

(2) and (3) are of higher immediate importance, because you can essentially emulate (1) by setting an appropriate min step (which is per query). The only restriction is that you cannot have higher resolution than that then. But that would lead to "overlapping" ranges, which is not desirable anyway (i.e. in practice you don't want a step in the actual API query that is smaller than the duration in your range selectors).

My gut feeling is that it will be easier/better to establish some kind of arithmetic in the Grafana variable expansion part than introducing more and more variables. Thus, I see two courses of action:

1. Go directly to "arithmetic in variable expansions".
2. Implement what's suggested here with the option of doing "arithmetic in variable expansions" later anyway.

I prefer (2) here because it is relatively easy to implement and hopefully not very controversial. (1) will take longer to get it right.

[beorn7]

Another point in favor of "arithmetic in variable expansions" is if subqueries enter the game. @gouthamve just had a case where he would have liked to write min by (instance)(min_over_time(metric[$__interval:${__interval/10}]))

[beorn7]

After digesting all the discussion, additional thoughts during preparation for my GrafanaCon talk and some feedback received, here a snapshot of my brain state:

The "proper" solution that actually nails all use cases is to allow arithmetic of some kind. This could be arithmetic in Grafana variable expansion. It could also be arithmetic in PromQL range durations. The former would also solve problems Grafana users have with non-Prometheus data sources, while the latter would also solve problems Prometheus users have without using Grafana (e.g. with other templating systems that might also lack support for arithmetic). The problem is that both solutions are quite invasive for their respective project (Grafana or Prometheus) and need a lot of thought to get right (and of course buy-in from the respective project itself – this might be a case where each project points fingers at each other as being in charge). Even if things work out really well, it might take a while.

Thus, we need an interim solution. However, for an interim solution, we probably don't want to introduce yet another setting, as suggested above (called Scrape interval or Interval extension above). Also, the arithmetic will clearly be a power user feature, and it would be good to have a "zero configuration" setting for the more naïve user. Therefore, I came up with the following idea:

Let's still introduce an additional variable, but call it $__rate_interval. It is meant to be a good enough heuristic that we can generally recommend to be used in rate queries. It is defined as max( $__interval + scrapeinterval, 4*scrapeinterval), where scrapeinterval is the Min step setting (AKA query_interval, a setting per PromQL query), if any is set, and otherwise the scrape interval as set in the Prometheus datasource (but ignoring any Min interval setting in the panel, because the latter is modified by the resolution setting).

I believe this should "just work" in 99+% of cases. It should be easy to add. It does not bloat the UI (no additional setting). It's just that one more variable name that we have to teach to users, but it can be used in most cases without much thinking (in contrast to the current situation, where you have to juggle with Min interval or Min step for rate queries, where the default behavior will not work, as demonstrated in my talk).

(Minor note: This will also allow the use case where the range used in the rate query is much larger than the actually used step size AKA $__interval, e.g. scrape interval is 15s, graph is for a short time range, so step size is also 15s, but the range in the rate query will be 1m, which results in something you could call a running average of a running average, but I have now seen a few cases where that makes some sense, in contrast to what we came up with in the whiteboard session that I had a while ago with @davkal and @SuperQ.)

[valyala]

To actually not skip over data, the range needs to be slightly larger than $__interval, ideally to create an overlap of one sample

This is exactly how rate works in MetricsQL.

What came out of that is a fork with something called xrate. This approach has issues of very different kinds.

It would be great to have a list of issues that xrate introduced in prometheus/prometheus#3746 .

Another point in favor of "arithmetic in variable expansions" is if subqueries enter the game. @gouthamve just had a case where he would have liked to write min by (instance)(min_over_time(metric[$__interval:${__interval/10}]))

FYI, this is already possible with MetricsQL:

min by (instance)(min_over_time(metric[1i:0.1i]))

[F30]

It should be easy to add. It does not bloat the UI (no additional setting). It's just that one more variable name that we have to teach to users, but it can be used in most cases without much thinking (in contrast to the current situation, where you have to juggle with Min interval or Min step for rate queries, where the default behavior will not work, as demonstrated in my talk).

I would like to add it appears much easier to just use a variable that "magically" does a reasonable thing, instead of having to deal with arithmetic for every range query.

Moreover so since the proposed definition of $__rate_interval seems reasonable, but it would be somewhat tiresome to manually replicate it repeatedly. So I think $__rate_interval would still be useful even with arithmetic implemented by Grafana or Prometheus.

[beorn7]

Playing with this right now.

Detailed result below. tl;dr: It works most of the time as intended, but setting a lower resolution (setting specific to Prometheus queries) throws things mostly off. Also, raising min step above the scrape interval in the data source gives surprising results.

I'm using a setup with a default data source, i.e. scrape interval is assumed to be 15s.

First test run: Don't set anything else, just use different time ranges to trigger different intervals:

• For a 7d graph, $__interval is 5m, while $__rate_interval is 315s (5m + 15s). GOOD
• For a 1d graph, $__interval is 30s, while $__rate_interval is 60s (4*15s). GOOD
• For a 5m graph, $__interval is 15s (hitting scrape interval as a minimum), while $__rate_interval is 60s (4*15s). GOOD

Second test run: Set a min interval of 1m for the panel. This should not modify the assumed scrape interval.

• For a 7d graph, $__interval is 5m, while $__rate_interval is 315s (5m + 15s). GOOD
• For a 1d graph, $__interval is 1m (hitting min interval), while $__rate_interval is 75s (1m + 15s). GOOD
• For a 5m graph, $__interval is 1m (hitting min interval), while $__rate_interval is 75s (1m + 15s). GOOD

Third test run: Set a min step of 1m in the query settings. This should change the assumed scrape interval to 1m.

• For a 7d graph, $__interval is 5m, while $__rate_interval is 360s (5m + 1m). GOOD
• For a 1d graph, $__interval is 1m (hitting min step), while $__rate_interval is 75s, but should be 4m (4*1m). BAD
• For a 5m graph, $__interval is 1m (hitting min step), while $__rate_interval is 75s, but should be 4m (4*1m). BAD

Forth test run: Set a min step of 6s in the query settings. This should change the assumed scrape interval to 6s.

• For a 7d graph, $__interval is 5m, while $__rate_interval is 306s (5m + 6s). GOOD
• For a 1d graph, $__interval is 30s, while $__rate_interval is 36s (30s + 6s). GOOD
• For a 5m graph, $__interval is 15s, while $__rate_interval is 24s (4*6s). GOOD

Fifth test run: Set a resolution of 1/2.

• For a 7d graph, $__interval is 10m, while $__rate_interval is 20m, but should be 615s (10m + 15s). BAD
• For a 1d graph, $__interval is 1m, while $__rate_interval is 2m, but should be 75s (1m + 15s). BAD
• For a 5m graph, $__interval is 30s, while $__rate_interval is 60s (4*15s). GOOD

@zoltanbedi perhaps we can pair next week to got to the bottom of this and create suitable test cases?

[powerman]

The latest docs already mention this new variable, but it looks like just released 7.1.4 doesn't provide it.

[trallnag]

Indeed https://grafana.com/docs/grafana/latest/features/datasources/prometheus/#using-__rate_interval-variable

[davkal]

The latest docs already mention this new variable, but it looks like just released 7.1.4 doesn't provide it.

Good point. This is new functionality and will be released with the next minor release (7.2) in September. Until then it's only available in the master builds. We'll update the docs to reflect this.

CC @oddlittlebird

[beorn7]

Also note that the variable is it is in head is still buggy. @zoltanbedi is back from vacation next week, and we'll get things fixed up as described above. Hopefully, all will be good for 7.2.

[trallnag]

A bit off-topic, but I have just spent almost two hours (even though I have been following this issue for quite some time 🤦‍♂️) trying to figure out why I'm not able to graph the rate of response bytes. Already considered that something is wrong with my instrumentation, but no, it comes down to the interval.

I have completely neglected the fact that this occurs on all time ranges and not just fairly big ones. Definitely looking forward to this getting released. I think I'll try out the master build. I could simply switch to a fixed interval like 5m and hope that the increase does not occur right at the beginning of a step, but then I would have to accept to ignore a lot of data at time ranges where Grafana's step exceeds 5m...

---

Edit: Nevermind, v7.2.0-e475443dpre (a2a6b94) is really buggy

[beorn7]

@trallnag our 2nd attempt is merged now. If you build from head now, at or after commit 86e44ee, everything should work as intended. If you still see weird behavior, we'd be grateful to hear about.

[trallnag]

@beorn7 I did and $__rate_interval seems to work fine, but I have quite a few other new issues that only occur with the grafana:master build. For example I cannot select values from the template variable dropdowns anymore. How stable is master supposed to be?

[beorn7]

Great to hear about $__rate_interval working as expected. I don't know a lot about the Grafana release process and how stable master is supposed to be. (Apparently not so stable… :o) Despite my affiliation, I don't really work on Grafana itself a lot. Others have to tell us more here.

[roidelapluie]

On 27 Aug 04:42, Björn Rabenstein wrote: Great to hear about `$__rate_interval` working as expected. I don't know a lot about the Grafana release process and how stable master is supposed to be. (Apparently not so stable… :o) Despite my affiliation, I don't really work on Grafana itself a lot. Other have to tell us more here.

Master is usually very stable.

[trallnag]

@roidelapluie Okay, thanks for the notice. I will try out the master again. Stuff like selecting template variables stopped working for me. And it seems like I'm not the only one (see this question on SO). I have also included a GIF recording that shows another weird behavior #21417 (comment)

[trallnag]

@beorn7, @zoltanbedi could the issue I have opened #27500 be related to this change? I cannot use Grafana interval variables in the min step field starting from Grafana versions > 7.1.5

[beorn7]

That definitely seems a regression. Not sure if it has to do with this change, @zoltanbedi would be much more qualified to say.

[zoltanbedi]

It is yes. Thanks for reporting it!