[v8.1.x] Alerting: Fix alert flapping in the internal alertmanager #38829
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Alerting: Fix alert flapping in the alertmanager fixes a bug that caused Alerts that are evaluated at low intervals (sub 1 minute), to flap in the Alertmanager. Mostly due to a combination of `EndsAt` and resend delay. The Alertmanager uses `EndsAt` as a heuristic to know whenever it should resolve a firing alert, in the case that it hasn't heard back from the alert generation system. Because grafana sent the alert with an `EndsAt` which is equal to the `For` of the alert itself, and we had a hard-coded 1 minute re-send delay (only applicable to firing alerts) this meant that a firing alert would resolve in the Alertmanager before we re-notify that it still firing. This commit, increases the `EndsAt` by 3x the the resend delay or alert interval (depending on which one is higher). The resendDelay has been decreased to 30 seconds. (cherry picked from commit dd502f2)
stevepostill
pushed a commit
to Reveal-International/grafana
that referenced
this pull request
Nov 3, 2021
…-github to revdev * commit '0d046c46b480ffc5fd602142a2b0e5dfc765b620': (63 commits) "Release: Updated versions in package to 8.1.3" (grafana#38965) Change grabpl version to 2.3.4 (grafana#38967) PieChart: Display "No data" when there is no data (grafana#38808) (grafana#38960) track signature files + add warn log (grafana#38938) (grafana#38958) Docs: Clarify delta value (grafana#38824) (grafana#38916) Postgres/MySQL/MSSQL: Fix region annotations not displayed correctly (grafana#38936) (grafana#38953) uPlot: Fix default value for plot legend visibility (grafana#36660) (grafana#38930) Prometheus: Fix validate selector in metrics browser (grafana#38921) (grafana#38926) Dashboard: Forces panel re-render when exiting panel edit (grafana#38913) (grafana#38919) [v8.1.x] Dashboard: Fix UIDs are not preserved when importing/creating dashboards thru importing .json file (grafana#38892) OAuth: add docs for disableAutoLogin param (grafana#38752) (grafana#38896) LibraryPanels: Prevents duplicate repeated panels from being created (grafana#38804) (grafana#38863) Adding missing information, more than just the manual backport. (grafana#38837) Elasticsearch: Prevent pipeline aggregations to show up in terms order by options (grafana#38448) (grafana#38830) Build: Upgrade grabpl to 2.4.2 (grafana#38820) (grafana#38828) Alerting: Fix alert flapping in the internal alertmanager (grafana#38648) (grafana#38829) [v8.1.x] Chore: Update to alpine:3.14.2 (grafana#38821) Update Dockerfile (grafana#38785) (grafana#38815) Deprecate browser access mode for the Graphite data source. (grafana#38783) (grafana#38809) Live: prepend orgId when publishing from HTTP (grafana#38775) (grafana#38793) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport dd502f2 from #38648