feat: instrument the query layer to track rate-limited queries #3894
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
pstibrany
reviewed
Mar 2, 2021
pkg/scheduler/queue/queue.go
Outdated
| @@ -91,6 +95,7 @@ func (q *RequestQueue) EnqueueRequest(userID string, req Request, maxQueriers in | |||
| } | |||
| return nil | |||
| default: | |||
| q.discardedQueries.WithLabelValues(userID, validation.RateLimited).Inc() | |||
There was a problem hiding this comment.
I think "rate limited" is misleading here. We only drop the request if queue is full, which can be for various reasons (slow queries, queriers crashed, no querier connected or all queriers are busy). I'd suggest to start without "reason" label.
Also I don't think we should use "validation" reasons (used for validating incoming samples) here.
pkg/scheduler/scheduler.go
Outdated
| @@ -100,7 +101,12 @@ func NewScheduler(cfg Config, limits Limits, log log.Logger, registerer promethe | |||
| Name: "cortex_query_scheduler_queue_length", | |||
| Help: "Number of queries in the queue.", | |||
| }, []string{"user"}) | |||
| s.requestQueue = queue.NewRequestQueue(cfg.MaxOutstandingPerTenant, s.queueLength) | |||
|
|
|||
| s.discardedQueries = promauto.With(registerer).NewCounterVec(prometheus.CounterOpts{ | |||
There was a problem hiding this comment.
Please cleanup this metric in cleanupMetricsForInactiveUser and add it to tests to verify that.
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
roystchiang
pushed a commit
to roystchiang/cortex
that referenced
this pull request
Apr 6, 2022
…xproject#3894) * feat: instrument the query layer to track rate-limited queries Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * chore: update changelog Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix: fix goimports linting error Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix per PR comments Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * rename discarded_queries --> discarded_requests Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix lint Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Instrument the query-frontend and query-scheduler to track the queries that were discarded because of the configured outstanding queries rate limiter. I pre-emptively added a reason label to allow for flexibility in adding other reasons for why a query may be discarded.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]