You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What happened:
A problem similar to one described here.
After joining the metric with kube_pod_labels to filter by an additional label sum and avg aggregations started making gaps. Other aggregations (max and min) still have no gaps though.
How it happens:
Have one pod under small but constant load (5rps) and emitting bucket counter metrics. The metrics are collected by prometheus. There's no gap in the aggregated metric.
Wind up a second, similar pod and start putting way more rps on it (100), so its bucket counters are growing 20 times faster.
Stop applying load on the second pod. Its bucket counters are not growing anymore. The first pod is still loaded with 5rps.
A new gap starts in the aggregated metric.
Delete the second pod.
The aggregated metric still has a gap before the time the second pod was deleted but no more gap after the time.
The PromQL expression used: avg(histogram_quantile(0.5, sum(rate(my_latency_bucket[$__rate_interval])) by (le,pod)) * on(pod) group_right(le) kube_pod_labels{my_pod_label="my_pod_label_value"})
Here is the unaggregated metric. There are a few pods coming and going:
Here is the result of min. No gaps, as expected:
And here is the result of avg. Notice the gaps:
What you expected to happen:
No gaps in aggregations if there are no gaps in the underlying metrics.
How to reproduce it (as minimally and precisely as possible):
I'm unsure.
Full logs to relevant components:
Cannot find the logs relevant to the expression evaluated.
Anything else we need to know:
The text was updated successfully, but these errors were encountered:
velavokr
changed the title
Gaps in sum and avg aggregations after joining the metric
Gaps in sum and avg aggregations after joining the metric with pod labels
Jun 17, 2024
velavokr
changed the title
Gaps in sum and avg aggregations after joining the metric with pod labels
Gaps in sum and avg aggregations after joining histogram with pod labels
Jun 17, 2024
velavokr
changed the title
Gaps in sum and avg aggregations after joining histogram with pod labels
Gaps in sum and avg aggregations after joining histogram quantile with pod labels
Jun 17, 2024
velavokr
changed the title
Gaps in sum and avg aggregations after joining histogram quantile with pod labels
[BUG] Gaps in sum and avg aggregations after joining histogram quantile with pod labels
Jun 17, 2024
velavokr
changed the title
[BUG] Gaps in sum and avg aggregations after joining histogram quantile with pod labels
[Bug] Gaps in sum and avg aggregations after joining histogram quantile with pod labels
Jun 17, 2024
velavokr
changed the title
[Bug] Gaps in sum and avg aggregations after joining histogram quantile with pod labels
[Bug] Gaps in sum and avg aggregations when joining histogram quantile with pod labels
Jun 17, 2024
#2736 (comment)
Thanos, Prometheus and Golang version used:
thanos helm chart: 15.0.5
kube-prometheus-stack helm chart: 57.2.0
Chart.yaml
andvalues.yaml
for them: https://gist.github.com/velavokr/e8410555385db7bc9b1a1c184fe99b72Object Storage Provider:
s3
What happened:
A problem similar to one described here.
After joining the metric with
kube_pod_labels
to filter by an additional labelsum
andavg
aggregations started making gaps. Other aggregations (max
andmin
) still have no gaps though.How it happens:
The PromQL expression used:
avg(histogram_quantile(0.5, sum(rate(my_latency_bucket[$__rate_interval])) by (le,pod)) * on(pod) group_right(le) kube_pod_labels{my_pod_label="my_pod_label_value"})
Here is the unaggregated metric. There are a few pods coming and going:
![Screenshot from 2024-06-16 21-50-06](https://private-user-images.githubusercontent.com/10815678/340180381-19b455bd-498c-4fc9-af32-3b9829e5f9e6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzMDg2NDksIm5iZiI6MTcyNDMwODM0OSwicGF0aCI6Ii8xMDgxNTY3OC8zNDAxODAzODEtMTliNDU1YmQtNDk4Yy00ZmM5LWFmMzItM2I5ODI5ZTVmOWU2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIyVDA2MzIyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdiNThjY2IzNTI2Y2U1ZGZhYzAzNDMwNjEzOGJlM2Q4YzVmODZiNDIzYWQ3NzQyNzM5NTI1ODg3NTMyN2Y3NGYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.vhHxiLrv7Wr-3zgItG1G3TGSQw5BVBPPKdgcxLGfnlc)
Here is the result of
![Screenshot from 2024-06-16 21-49-27](https://private-user-images.githubusercontent.com/10815678/340180736-023705f6-b35a-419a-9f6b-f4d94484d3a7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzMDg2NDksIm5iZiI6MTcyNDMwODM0OSwicGF0aCI6Ii8xMDgxNTY3OC8zNDAxODA3MzYtMDIzNzA1ZjYtYjM1YS00MTlhLTlmNmItZjRkOTQ0ODRkM2E3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIyVDA2MzIyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWIxMWJiNTgyYTljZjUzODFmMzE0ZmIzYThmMGMyN2M1Yjk5ZjdhYjBmNjRkNDc0MjY0ZjAwNDMzZTI2OTAyYzImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.vp3QGYmi-jmxOGqQi5NWoDsJsOAUE4KM6xg368c9w38)
min
. No gaps, as expected:And here is the result of
![Screenshot from 2024-06-16 21-49-43](https://private-user-images.githubusercontent.com/10815678/340180624-78434c47-0a00-4ef7-81d4-855fa5ee4488.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzMDg2NDksIm5iZiI6MTcyNDMwODM0OSwicGF0aCI6Ii8xMDgxNTY3OC8zNDAxODA2MjQtNzg0MzRjNDctMGEwMC00ZWY3LTgxZDQtODU1ZmE1ZWU0NDg4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIyVDA2MzIyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFhM2FiZDgxZDY4OGJkNDE1YWIxYzYwNzI0MjBlZjdjYTBmNmZlNzQ2MDYzOGViMWRhMjhmZTU1ZGViOTZmZjEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.LAKFGffLqM8qwuCxi9coShYj2mqvhmFKkv7hVWdjBOw)
avg
. Notice the gaps:What you expected to happen:
No gaps in aggregations if there are no gaps in the underlying metrics.
How to reproduce it (as minimally and precisely as possible):
I'm unsure.
Full logs to relevant components:
Cannot find the logs relevant to the expression evaluated.
Anything else we need to know:
The text was updated successfully, but these errors were encountered: