Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v10.2.x] Prometheus: Fix $__rate_interval calculation #84063

Merged
merged 1 commit into from
Mar 7, 2024

Conversation

tolzhabayev
Copy link
Contributor

@tolzhabayev tolzhabayev commented Mar 7, 2024

Manual Backport b607a4e from #77234


Background Information

$__rate_interval calculation is explained here in detail.
Shortly it is max($__interval + Scrape interval, 4 * Scrape interval) where Scrape interval is the “Min step” setting (also known as query*interval, a setting per PromQL query) if any is set. Otherwise, Grafana uses the Prometheus data source’s “Scrape interval” setting.
Why we need that in the first place is explained here in this beautiful blog post.
$__interval calculation is explained here.

For all data sources, we have a Min Interval value. The default value of this is 15s or you can set it differently in data source settings. To be able to interpolate $__interval we also send intervalMs. This is always the millisecond value of Interval in the panel below.

Last 2 Days time range
image

In the Prometheus query editor, we also have Min Step for queries.
image

Users might want to override Min Interval or Min Step.

Min Interval setting has precedence over the scrape interval as set in the data source.
Min Step setting has precedence over the Min Interval.

Backend receives that as interval. This can be seen in Query Inspector.

So let's use the values below as example.

Data source Scrape Interval 30s
Time Range Last 2 Days
Interval (calculated Automatically) 1m
Min Interval (Overriden) 100s
Min Step (Overriden) 150s

image

Based on the documentation let's put the values in the equation and find the $__rate_interval value.

max($__interval + query.interval, 4 * query.interval)
max(       60   +  150 ,          4 * 150)          

So this should give the value as 600s. And the step value we send to Prometheus should be 150s => 2m30s. Let's see it in the query inspector:
image

As you can see something is not quite right. So this PR is fixing it. Let's switch to this branch and run our query again.
image

Who is this feature for?

Prometheus users

Special notes for your reviewer:

  • I updated the variable names since they were quite confusing
  • I put some comments

Please check that:

  • It works as expected from a user's perspective.
  • If this is a pre-GA feature, it is behind a feature toggle.
  • The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

* Remove unused param

* simple unit test

* rename

* rename

* add some comments

* Update values

* refactor

* rename

* always calculate rate interval

* fix unit tests

* Fix indentation

* linter fix

* update test

* Fixing issues with the calculation

* new test

* fix $__interval interpolation

* fix test

* add comment

(cherry picked from commit b607a4e)
@tolzhabayev tolzhabayev added type/bug datasource/Prometheus area/backend add to changelog backport A backport PR product-approved Pull requests that are approved by product/managers and are allowed to be backported labels Mar 7, 2024
@tolzhabayev tolzhabayev added this to the 10.2.x milestone Mar 7, 2024
@tolzhabayev tolzhabayev changed the title [v10.2.x] Prometheus: Fix calculation [v10.2.x] Prometheus: Fix $__rate_interval calculation Mar 7, 2024
@tolzhabayev tolzhabayev marked this pull request as ready for review March 7, 2024 12:15
@tolzhabayev tolzhabayev requested a review from a team as a code owner March 7, 2024 12:15
@itsmylife itsmylife self-requested a review March 7, 2024 13:09
@tolzhabayev tolzhabayev merged commit e6f7a39 into v10.2.x Mar 7, 2024
28 checks passed
@tolzhabayev tolzhabayev deleted the backport-77234-to-v10.2.x branch March 7, 2024 17:01
@ifrost ifrost modified the milestones: 10.2.x, 10.2.6 Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
add to changelog area/backend backport A backport PR datasource/Prometheus product-approved Pull requests that are approved by product/managers and are allowed to be backported type/bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants