Alerting: Enable group-level rule evaluation jittering by default, remove feature toggle #82212
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this feature?
This PR removes the
jitterAlertRules
feature toggle. The new default behavior is as if it's enabled.The jitter behavior was introduced in #80766
The effect is that the evaluation time of rule groups are distributed over their group interval. Rules within a group will still evaluate together on all replicas, but this time is smoothed out over the rule's evaluation interval rather than synchronized with all other rules on the entire system.
This feature has been globally active in Grafana Cloud for a while now, without issue. It heavily reduces load on datasources by decreasing query spikes. The lack of spiky evaluations improves alert evaluation time and notification p99 time by roughly 10% across the board.
The jitter is now on by default, but can still be disabled with
disable_jitter
under the[unified_alerting]
config section.Which issue(s) does this PR fix?:
Fixes #53744
Special notes for your reviewer:
Please check that: