Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: add jitter to refresh interval of individual panels #83019

Open
gelicia opened this issue Feb 17, 2024 Discussed in #36991 · 0 comments
Open

Feature request: add jitter to refresh interval of individual panels #83019

gelicia opened this issue Feb 17, 2024 Discussed in #36991 · 0 comments

Comments

@gelicia
Copy link
Contributor

gelicia commented Feb 17, 2024

Discussed in #36991

Originally posted by JorritSalverda June 1, 2017
To distribute load more evenly on a queried system like ElasticSearch or Prometheus it will help to add a little bit of randomness / jitter to the refresh interval of each individual panel and every time it triggers pick a new 'random' interval until the next update.

With the current implementation of very rigid intervals our Prometheus server has huge spikes in cpu load corresponding with the refresh intervals of frequently used dashboards.

See article https://cloudplatform.googleblog.com/2016/11/how-to-avoid-a-self-inflicted-DDoS-Attack-CRE-life-lessons.html about why this is useful.

An implementation in Javascript might look like this for a jitter percentage of 25% resulting in a random number between 75% and 125% of the defined interval for the respective dashboard:

jitteredIntervalUntilNextRefresh = Math.random() * 0.5 * interval + 0.75 * interval  

Most important is that every panel calculates it's own interval until the next refresh every time, to prevent it from slowly synchronizing after time and does so after each refresh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant