[v10.2.x] Prometheus: Fix $__rate_interval calculation #84063
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Manual Backport b607a4e from #77234
Background Information
$__rate_interval
calculation is explained here in detail.Shortly it is
max($__interval + Scrape interval, 4 * Scrape interval)
where Scrape interval is the “Min step” setting (also known as query*interval, a setting per PromQL query) if any is set. Otherwise, Grafana uses the Prometheus data source’s “Scrape interval” setting.Why we need that in the first place is explained here in this beautiful blog post.
$__interval
calculation is explained here.For all data sources, we have a
Min Interval
value. The default value of this is15s
or you can set it differently in data source settings. To be able to interpolate$__interval
we also sendintervalMs
. This is always the millisecond value ofInterval
in the panel below.Last 2 Days time range
![image](https://private-user-images.githubusercontent.com/820946/278413876-3d7278d5-c195-4efc-be4e-c72641e702e3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxODM2MzIsIm5iZiI6MTcyMjE4MzMzMiwicGF0aCI6Ii84MjA5NDYvMjc4NDEzODc2LTNkNzI3OGQ1LWMxOTUtNGVmYy1iZTRlLWM3MjY0MWU3MDJlMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyOFQxNjE1MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05NWVhYTQ3MmZjYmJlYTBmZTNmMzFkMTA2Njk2ZjNmNzdmY2YzNzU0NmY0ODM4MDM0ODMzMTJkMzY5OTNkZmY1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.loR5ckPNXS28bOD9OkTqdvBJLckzHTNfWCZf8RQXhaY)
In the Prometheus query editor, we also have
![image](https://private-user-images.githubusercontent.com/820946/278414146-4872ddd1-1e9f-4009-9587-ddd4575591d3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxODM2MzIsIm5iZiI6MTcyMjE4MzMzMiwicGF0aCI6Ii84MjA5NDYvMjc4NDE0MTQ2LTQ4NzJkZGQxLTFlOWYtNDAwOS05NTg3LWRkZDQ1NzU1OTFkMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyOFQxNjE1MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05N2VmNDY5YzhjZmEyMzliZDgwN2YzMmVlMzRiMjY4OTM4YmQzZDA2Y2MxNmZjMmUyYjQ5MjY2NjFlNzQ5ZTZiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.CqoFbQ-ZUNKtr9JIkq_Dg9n5Nfgbs9Ah1F0X-Q3PMTM)
Min Step
for queries.Users might want to override
Min Interval
orMin Step
.Min Interval
setting has precedence over the scrape interval as set in the data source.Min Step
setting has precedence over theMin Interval
.Backend receives that as
interval
. This can be seen inQuery Inspector
.So let's use the values below as example.
Based on the documentation let's put the values in the equation and find the
$__rate_interval
value.So this should give the value as 600s. And the
![image](https://private-user-images.githubusercontent.com/820946/278426630-91dc5388-7eec-4c90-a4c8-76427dcc6886.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxODM2MzIsIm5iZiI6MTcyMjE4MzMzMiwicGF0aCI6Ii84MjA5NDYvMjc4NDI2NjMwLTkxZGM1Mzg4LTdlZWMtNGM5MC1hNGM4LTc2NDI3ZGNjNjg4Ni5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyOFQxNjE1MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00ZDFmZjJjNzhiNTJiYTdhYmZkYjYzZDJlNzA3NjkxMTdiOGJmNWE5YzdmNTBiZTdiYWVjM2Y5NTc1MTBjZDFkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.RV571gVE6TfQPMtXHYf-1ZI-hoi6KqQZAfzFnfuvfCU)
step
value we send to Prometheus should be150s => 2m30s
. Let's see it in the query inspector:As you can see something is not quite right. So this PR is fixing it. Let's switch to this branch and run our query again.
![image](https://private-user-images.githubusercontent.com/820946/278426395-0d912b16-ef76-4723-acdc-0571b6df7953.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxODM2MzIsIm5iZiI6MTcyMjE4MzMzMiwicGF0aCI6Ii84MjA5NDYvMjc4NDI2Mzk1LTBkOTEyYjE2LWVmNzYtNDcyMy1hY2RjLTA1NzFiNmRmNzk1My5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzI4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyOFQxNjE1MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lNDdjYzA5YzY0YWVjY2Y4YjczNjIxYTBkMzBiMTBjMzVlYzVhNmRlODQ1Y2VjMzc1NjI3OWRlNmJmMGM2NWVmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.msDytsb7wDX7er1kboxdLqQ2qFs6Zx9E86Ty4Rzvnmc)
Who is this feature for?
Prometheus users
Special notes for your reviewer:
Please check that: