Loki-Canary: Backoff retries on query failures, add histograms for query performance. #2413
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…unt of time else the nature of the async thread doing the testing can ask for them from Loki only a few ms after they are written and they would have no chance of being present. Adding two histograms for tracking the performance of the spot check and metric test queries
…n a 429 is received from Loki
Codecov Report
@@ Coverage Diff @@
## master #2413 +/- ##
==========================================
+ Coverage 61.67% 61.80% +0.13%
==========================================
Files 160 160
Lines 13577 13584 +7
==========================================
+ Hits 8373 8395 +22
+ Misses 4581 4565 -16
- Partials 623 624 +1
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current code is a bit aggressive when Loki responds with a failure, this puts a global rate limit inside the Reader object which will be updated if any query fails using an exponential backoff, all other queries will have to wait for the backoff before they can proceed.
Additionally 429's are special cased with an immediate 5min wait, this error can happen when the query frontend queue is full and we don't want canaries exacerbating this situation.