per user index query readiness with limits overrides #5484
Conversation
| Directory: t.TempDir(), | ||
| }, | ||
| }, metrics) | ||
| objectClient, err := local.NewFSObjectClient(local.FSConfig{ |
There was a problem hiding this comment.
All the test changes in usagestats package were done to avoid a circular dependency.
It was caused by the import fix I made at https://github.com/grafana/loki/pull/5484/files#diff-8aa5c09f2d873a1f7c7a48d253da4d5d1a434cf2fa32ef84e5c1a68e0617f82aR15
| "github.com/grafana/loki/pkg/util/validation" | ||
| "github.com/grafana/loki/pkg/validation" |
There was a problem hiding this comment.
The github.com/grafana/loki/pkg/util/validation points to a copy of the cortex validation package, which we should clean up sometime.
sandeepsukhani
force-pushed
the
query-readiness-with-tenant-overrides
branch
from
754a6cf
to
c8cfe6c
Compare
| // metrics for measuring performance of downloading of files per period initially i.e for the first time | ||
| tablesDownloadDurationSeconds *downloadTableDurationMetric | ||
|
|
||
| queryTimeTableDownloadDurationSeconds *prometheus.CounterVec |
There was a problem hiding this comment.
I wonder if an histogram wouldn't be better here ?
There was a problem hiding this comment.
Not sure if that would add much value. Would we be interested in looking at a distribution? I think it would be rare and we can use quantile_over_time from the logline I have added here. wdyt?
There was a problem hiding this comment.
What's the PromQL you'll use with this ?
There was a problem hiding this comment.
something like sum(quantile_over_time({container="index-gateway"} |= "downloaded index set at query time" | logfmt | unwrap duration(duration)[5m]) by (user_id). Makes sense?
…ile ensuring query readiness
What this PR does / why we need it:
The current query readiness support works only on table level. With a cluster building per user index and having query readiness configured, it would add a lot of overhead to keep the active index synced since we get a new index from ingesters every 15 mins.
This PR adds support for more fine-grained control over query readiness for per user index using limits overrides config.
I have also changed the metric to track the duration it takes to download the index at query time. It would be tracked per table. I have not added a user id to it to avoid adding too many series. I have added a log line which we can use for viewing per tenant download duration by both user id and table.
I have also dropped the older metric
loki_boltdb_shipper_initial_tables_download_duration_seconds, which was not very reliable and the name was not very clear too.Checklist
CHANGELOG.mdabout the changes.