Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alerting: Add a feature flag to periodically save states #80987

Merged
merged 3 commits into from
Jan 23, 2024

Conversation

JohnnyQQQQ
Copy link
Member

This PR introduces a new feature flag that enables grafana admins to make their grafana instance save the state of alert instances periodically instead of synchronously. This should remove a lot of load for grafana instances that are running a evaluating a lot of alert rules.

The name of the feature flag is alertingSaveStatePeriodic and the interval for the periodic save can be configured by setting state_periodic_save_interval, by default it is set to 5 minutes.

@JohnnyQQQQ JohnnyQQQQ requested a review from a team January 22, 2024 14:01
@JohnnyQQQQ JohnnyQQQQ requested a review from a team as a code owner January 22, 2024 14:01
@JohnnyQQQQ JohnnyQQQQ requested review from rwwiv, JacobsonMT, yuri-tceretian, grobinson-grafana, papagian, undef1nd and suntala and removed request for a team January 22, 2024 14:01
@grafana-delivery-bot grafana-delivery-bot bot added this to the 10.4.x milestone Jan 22, 2024
@JohnnyQQQQ JohnnyQQQQ changed the title feat(alerting): add feature flag to periodically save states feat(alerting): Add a feature flag to periodically save states Jan 22, 2024
@JohnnyQQQQ JohnnyQQQQ changed the title feat(alerting): Add a feature flag to periodically save states Alerting: Add a feature flag to periodically save states Jan 22, 2024
@JohnnyQQQQ JohnnyQQQQ requested review from alexweav and removed request for torkelo, a team, papagian, undef1nd and suntala January 22, 2024 14:03
@grafana-pr-automation grafana-pr-automation bot requested review from a team and removed request for a team January 22, 2024 14:05
Copy link
Contributor

@grobinson-grafana grobinson-grafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested both with and without the feature flag and it works! 👍 I left a question about metrics. In short, how would I as an operator/engineer on-call know that syncs are taking too long and need to increase the interval?

conf/sample.ini Show resolved Hide resolved
pkg/services/ngalert/state/manager.go Outdated Show resolved Hide resolved
pkg/services/ngalert/state/persister_async.go Show resolved Hide resolved
@grafana-pr-automation grafana-pr-automation bot requested review from a team and removed request for a team January 23, 2024 15:52
@JohnnyQQQQ JohnnyQQQQ merged commit aa25776 into main Jan 23, 2024
18 checks passed
@JohnnyQQQQ JohnnyQQQQ deleted the add-periodic-state-save-ff branch January 23, 2024 16:03
@aangelisc aangelisc modified the milestones: 10.4.x, 10.4.0 Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants