Add mixin alert for ring member mismatch #2404
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pracucci
reviewed
Jul 14, 2022
pracucci
reviewed
Sep 12, 2022
pracucci
reviewed
Sep 19, 2022
pracucci
reviewed
Sep 19, 2022
jhesketh
force-pushed
the
active_member_alert
branch
from
4ef661f
to
dad6955
Compare
|
@pracucci bump for when you have time please :-) |
pracucci
reviewed
Oct 6, 2022
osg-grafana
approved these changes
Oct 7, 2022
|
|
||
| How to **investigate**: | ||
|
|
||
| - Check the [hash ring web page]({{< relref "../reference-http-api/index.md#ingesters-ring-status" >}}) for the component for which the alert has fired, and look for unexpected instances in the list. |
There was a problem hiding this comment.
Suggested change
| - Check the [hash ring web page]({{< relref "../reference-http-api/index.md#ingesters-ring-status" >}}) for the component for which the alert has fired, and look for unexpected instances in the list. | |
| - For the component for which the alert has fired, and to look for unexpected instances in the list, see [Ingesters ring status]({{< relref "../reference-http-api/index.md#ingesters-ring-status" >}}). |
There was a problem hiding this comment.
I'm not sure this suggestion makes sense. This is an action to take once you know a component that is mis-matched.
|
The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase |
jhesketh
force-pushed
the
active_member_alert
branch
from
66e5347
to
824e850
Compare
This alert fires when the expected number of ring members does not match the running number for each component (one of `compactor`, `distributor`, `ingester`, `ruler`, or `store-gateway`). Fixes: grafana#1960
Because we aren't building dashboards with $namespace/$cluster etc in the alerts, we can't use dashboard-utils.jobMatcher and thus need to optionally match on possible namespace prefix (.*/).
jhesketh
force-pushed
the
active_member_alert
branch
from
824e850
to
b855870
Compare
pracucci
approved these changes
Oct 11, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This alert fires when the expected number of ring members does not
match the running number for each component (one of
compactor,distributor,ingester,ruler, orstore-gateway).What this PR does
Which issue(s) this PR fixes or relates to
Fixes #1960
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]