You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Originally posted by JorritSalverda June 1, 2017
To distribute load more evenly on a queried system like ElasticSearch or Prometheus it will help to add a little bit of randomness / jitter to the refresh interval of each individual panel and every time it triggers pick a new 'random' interval until the next update.
With the current implementation of very rigid intervals our Prometheus server has huge spikes in cpu load corresponding with the refresh intervals of frequently used dashboards.
An implementation in Javascript might look like this for a jitter percentage of 25% resulting in a random number between 75% and 125% of the defined interval for the respective dashboard:
Most important is that every panel calculates it's own interval until the next refresh every time, to prevent it from slowly synchronizing after time and does so after each refresh.
The text was updated successfully, but these errors were encountered:
Discussed in #36991
Originally posted by JorritSalverda June 1, 2017
To distribute load more evenly on a queried system like ElasticSearch or Prometheus it will help to add a little bit of randomness / jitter to the refresh interval of each individual panel and every time it triggers pick a new 'random' interval until the next update.
With the current implementation of very rigid intervals our Prometheus server has huge spikes in cpu load corresponding with the refresh intervals of frequently used dashboards.
See article https://cloudplatform.googleblog.com/2016/11/how-to-avoid-a-self-inflicted-DDoS-Attack-CRE-life-lessons.html about why this is useful.
An implementation in Javascript might look like this for a jitter percentage of 25% resulting in a random number between 75% and 125% of the defined interval for the respective dashboard:
Most important is that every panel calculates it's own interval until the next refresh every time, to prevent it from slowly synchronizing after time and does so after each refresh.
The text was updated successfully, but these errors were encountered: