Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new alerts for alertmanager sharding mode of operation. #377

Merged
merged 2 commits into from
Aug 25, 2021

Conversation

stevesg
Copy link
Contributor

@stevesg stevesg commented Aug 24, 2021

What this PR does:
Adds new alerts to cover aspects of the alertmanager pertinent to sharding operation.

Checklist

  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@stevesg stevesg force-pushed the alertmanager-sharding-alerts branch from 425e936 to 0e78e94 Compare August 24, 2021 11:31
@stevesg stevesg marked this pull request as ready for review August 24, 2021 11:34
@stevesg stevesg requested a review from a team as a code owner August 24, 2021 11:34
@stevesg
Copy link
Contributor Author

stevesg commented Aug 24, 2021

Question: Shall we to add the playbooks onto this PR or open a new PR for them?

Copy link
Contributor

@gotjosh gotjosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, let's get these in, for now, we can add the runbooks independently.

CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! All the alerts make sense to me 👍

I think it's a better practice to write the playbook along with the alert instead of postponing it. You can start very easy, explaining (1) when the alert fires and (2) how to investigate (see other playbooks as an example). The "how to investigate" could even just "check out the logs to find the specific error occurred", but at least with (1) we explain when the alert fires and how the system works.

@stevesg stevesg force-pushed the alertmanager-sharding-alerts branch from 8c94ddc to 6422e0b Compare August 25, 2021 10:44
@stevesg stevesg merged commit 79144dd into grafana:main Aug 25, 2021
simonswine pushed a commit to grafana/mimir that referenced this pull request Oct 18, 2021
…er-sharding-alerts

Add new alerts for alertmanager sharding mode of operation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants