Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed flushes alert #1157

Merged
merged 8 commits into from
Dec 6, 2021
Merged

Failed flushes alert #1157

merged 8 commits into from
Dec 6, 2021

Conversation

joe-elliott
Copy link
Member

What this PR does:
Makes the TempoIngesterFlushesFailing alert less trigger-happy in situations where it can self-heal.
It also includes a variation on the alert which causes it to page immediately if self-healing is failing.

Which issue(s) this PR fixes:
This completes the follow-up I should have done after #1049

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Danny Kopping and others added 6 commits December 6, 2021 09:12
…re it can self heal

Include an alert to page immediately if self-healing is failing

Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Copy link
Contributor

@dannykopping dannykopping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

operations/tempo-mixin/yamls/alerts.yaml Outdated Show resolved Hide resolved
Signed-off-by: Joe Elliott <number101010@gmail.com>
Copy link
Member

@kvrhdn kvrhdn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@joe-elliott joe-elliott merged commit 1131a9b into main Dec 6, 2021
@joe-elliott joe-elliott deleted the failed-flushes-alert branch December 6, 2021 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants