Mixin: Remove low ingestion rate warning for receiver #5824
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Matej Gera <matejgera@gmail.com>
squat
approved these changes
Oct 26, 2022
|
Very good, i think the point about this alerting on cause vs symptom is completely right |
2 tasks
|
@GiedriusS small addition, this was not caught because of the failing CircleCI tests - #5831 |
ngraham20
pushed a commit
to ngraham20/thanos
that referenced
this pull request
May 18, 2023
Signed-off-by: Matej Gera <matejgera@gmail.com> Signed-off-by: Matej Gera <matejgera@gmail.com>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Matej Gera matejgera@gmail.com
Changes
This PR removes the
ThanosReceiveTrafficBelowThreshold. This alert is rather noisy, since it's takes into account every tenant separately, which increases the noisiness in environments with higher number of tenants with non-constant ingestion rate, which is not a cause for alarm. Besides, this alert tries to alert on cause and not the symptoms, making it more of an attempt to catch anomalies rather than actual issues with receiver - low ingestion rate does not necessarily signal a problem, as this could be expected behavior, depending on the characteristics of user's system, number of clients, their churn rate etc. Users are better off alerting on other issues (e.g. absent receivers, alerting on client's side with dead man's snitch etc.)Verification
N/A