rate on federated non aggregated metrics is too low in last minute #10957
Replies: 4 comments
-
In case anyone encounter similar problem, I walk around the problem by splitting federation jobs and reduce the cost of each job from 45s to 15s, and the promblem disappeared. |
Beta Was this translation helpful? Give feedback.
-
Very generally, federation will introduce a delay: Metrics are ingested with their original timestamp, but at a somewhat later time. This effect gets worse if the federation scrape takes a long time (as you have noticed). You might want to point dashboards to a timerange that excludes the most recent minute or so. |
Beta Was this translation helpful? Give feedback.
-
Thanks for the explanation.
Is is possible to have those newly ingested metrics unvisible until they actually have values? This way, we can still use the old value last time ingested instead of zero. Query results would be normal, too. |
Beta Was this translation helpful? Give feedback.
-
You could do rate(metric[5m] offset 1m) |
Beta Was this translation helpful? Give feedback.
-
What did you do?
I have 3 prometheus servers A1, A2, B.
Server A1/A2 scrapes metrics every 3 seconds from disjoint targets.
B federates all metrics from A every 1 min, each federation scrape takes about 45s.
Let's say
my_total
is original metric scraped by A.When running
![企业微信截图_16524094357129](https://user-images.githubusercontent.com/22031595/168200358-236b5edc-c30a-48b5-ad87-97aa8aac9f2c.png)
sum( rate( my_total [2m] ) )
on B, I got results too low, like:I check
![企业微信截图_16524102046849](https://user-images.githubusercontent.com/22031595/168201729-c457024b-bcbc-4f9c-97ee-61620d39213e.png)
rate( my_total [2m] )
and found some series are normal.while others are not:
![企业微信截图_16524101424636](https://user-images.githubusercontent.com/22031595/168201656-e99b26dc-9680-416e-a8be-fd4660d03fd8.png)
What should I do to prevent it?
What did you expect to see?
normal last minute metrics
What did you see instead? Under which circumstances?
last minute's data is too low
Beta Was this translation helpful? Give feedback.
All reactions