*: remove unmaintained gzip library #6332
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
fpetkovski
approved these changes
May 4, 2023
bwplotka
approved these changes
May 4, 2023
squat
approved these changes
May 4, 2023
|
@GiedriusS Do we have any benchmark about the two libraries? Does it help with the allocation after switching to the new one? |
GiedriusS
added a commit
to vinted/thanos
that referenced
this pull request
May 17, 2023
Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
hczhu
pushed a commit
to databricks/thanos
that referenced
this pull request
Jun 27, 2023
Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
hczhu
pushed a commit
to databricks/thanos
that referenced
this pull request
Jun 27, 2023
Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
subbarao-meduri
added a commit
to stolostron/thanos
that referenced
this pull request
Sep 7, 2023
* Updates busybox SHA (#6283)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287)
* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346
Signed-off-by: Alex Le <leqiyue@amazon.com>
* Reverted test code
Signed-off-by: Alex Le <leqiyue@amazon.com>
* Updated comment
Signed-off-by: Alex Le <leqiyue@amazon.com>
* docs: mismatch in changelog
Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
* Updates busybox SHA (#6283)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
* trigger workflow
Signed-off-by: Alex Le <leqiyue@amazon.com>
* trigger workflow
Signed-off-by: Alex Le <leqiyue@amazon.com>
---------
Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com>
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Add CarTrade Tech as new adopter
Signed-off-by: naveadkazi <navead@carwale.com>
* sort ProxyResponseHeap comparing only diff labels
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* proxy heap: remove store labels before comparing response series
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* tests: Remove custom Between test matcher (#6310)
* Remove custom Between test matcher
The upstream PR to efficientgo/e2e has been merged, so we can use it from there.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Run go mod tidy
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Remove testingRespSet and update comparison func
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* Fix tests
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* query frontend, query UI: Native histogram support (#6071)
* Implemented native histogram support for qfe and query UI
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed marshalling for histograms in qfe
Started working on native histogram query ui
Copied histogram implementation for graph
Added query range support for native histograms in qfe
Use prom model (un-)marshal for native histograms in qfe
Use prom model (un-)marshal for native histograms in qfe
Fixed sample and sample stream marshal fn
Extended qfe native histogram e2e tests
Added copyright to qfe queryrange compat
Added query range test fo histograms and try to fix ui tests
Fixed DataTable test
Review feedback
Fixed native histogram e2e test
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Add histogram support for ApplyCounterResetsSeriesIterator
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Made assets
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Add chnagelog
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed changelog
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed qfe
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed PrometheusResponse minTime for histograms in qfe
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Updated prometheus common to v0.40.0 and queryrange.Sample fixes
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Updated Readme
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Addressed PR comments
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
trigger tests
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Made assets
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Made assets
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* fixed tsdbutil references
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* fixed imports
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Enabled pushdown for query native hist test and removed ToDo
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Refactored native histogram query UI
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
---------
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* store: add streamed snappy encoding for postings list (#6303)
* store: add streamed snappy encoding for postings list
We've noticed that decoding Snappy compressed postings list
takes a lot of RAM:
```
(pprof) top
Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total
Dropped 1069 nodes (cum <= 10.56GB)
Showing top 10 nodes out of 82
flat flat% sum% cum cum%
0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1
2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings
1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings
2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings
1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode
1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode
```
This is because we are creating a new []byte slice for the decoded data
each time. To avoid this RAM usage problem, let's stream the decoding
from a given buffer. Since Snappy block format doesn't support streamed
decoding, let's switch to Snappy stream format which is made for exactly
that.
Notice that our current `index.Postings` list does not
support going back through Seek() even if theoretically one could want
something like that. Fortunately, to search for posting intersection, we
need to only go forward.
Benchmark data:
```
name time/op
PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3%
PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4%
PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1%
PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4%
PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1%
PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13%
PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1%
PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4%
PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1%
PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4%
name alloc/op
PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0%
name allocs/op
PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0%
```
Compression ratios are still the same like previously:
```
$ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store
[snip]
=== RUN TestDiffVarintCodec/snappy/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44498 bytes
postings_codec_test.go:81: ratio: 0.012
=== RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44670 bytes
postings_codec_test.go:81: ratio: 0.012
```
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: clean up postings code
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: fix estimation
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: use buffer.Bytes()
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store/postings_codec: reuse extgrpc compressors/decompressors
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* CHANGELOG: add item
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* CHANGELOG: clean up whitespace
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
---------
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* compact: atomically replace no compact marked map (#6319)
With lots of blocks it could take some time to fill this no compact
marked map hence replace it atomically. I believe this leads to problems
in the compaction planner where it picks up no compact marked blocks
because meta syncer does synchronizations concurrently.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Reuse buffers for label comparison
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Use len instead of cap
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Use len instead of cap
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Fixed Jaeger exporter specification link (#6324)
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* receive: Make max backoff configurable (#6163)
Signed-off-by: Philip Gough <philip.p.gough@gmail.com>
* Replace summary in extprom metrics with histogram (#6327)
* Replaced summary in extprom metrics with histogram
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Added changelog
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Removed unused parameters from NewInstrumentationMiddleware
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Reverted NewInstrumentationMiddleware
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
---------
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Expose info for each TSDB
This commit exposes the label set alongside the min and max time
for each TSDB covered by a Store.
This information is used to scope the min time for a remote query
so that we do not produce partial aggregates in distriuted mode.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Add test case for proxy store
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Pass engine factory by reference
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Cleanup
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Use NewQueryEngineFactory in tests
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Avoid expensive log.Valuer evaluation for disallowed levels (#6322)
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
* Fix inconsistent error for series limits in Store API (#6330)
* store: fix inconsistent error for series limits
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* update changelog
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* Update pkg/store/bucket.go
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* Update pkg/store/bucket.go
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* rename labelValues serires liimiter test function
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
---------
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* *: remove unmaintained gzip library (#6332)
Switch from nytimes gzip library to the klaustpost's gzip code. The old
gzip HTTP handler shows up a lot in allocs so that's how I ended up
doing this change.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Traces sampler env var (#6306)
* Issue#5947 OTEL_TRACES_SAMPLER env var
Signed-off-by: shayyxi <shazi12384@gmail.com>
* Test correction
Signed-off-by: shayyxi <shazi12384@gmail.com>
* doc failure correction. parse float argument correction.
Signed-off-by: shayyxi <shazi12384@gmail.com>
* added the changelog.
Signed-off-by: shayyxi <shazi12384@gmail.com>
* ran make docs to fix the build failure.
Signed-off-by: shayyxi <shazi12384@gmail.com>
* corrected the incorrect change in tools.md
Signed-off-by: shayyxi <shazi12384@gmail.com>
* fixed review comments.
Signed-off-by: shayyxi <shazi12384@gmail.com>
---------
Signed-off-by: shayyxi <shazi12384@gmail.com>
Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com>
Co-authored-by: shayyxi <shazi12384@gmail.com>
* query: use storepb.SeriesServer (#6334)
Use storepb.SeriesServer instead of the concrete struct. This allows
implementing functionality on top of the proxy.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342)
* use github.com/onsi/gomega/gleak to detect goroutine leak with timeout
Signed-off-by: Rueian <rueiancsie@gmail.com>
* fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345)
Signed-off-by: aimuz <mr.imuz@gmail.com>
* Return grpc code resource exhausted for byte limit error (#6325)
* return grpc code resource exhausted for byte limit error
Signed-off-by: Ben Ye <benye@amazon.com>
* fix lint
Signed-off-by: Ben Ye <benye@amazon.com>
* update partial response strategy
Signed-off-by: Ben Ye <benye@amazon.com>
* fix limit
Signed-off-by: Ben Ye <benye@amazon.com>
* try to fix tests
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test error message
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349)
Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>
* Optimize generating store match reason (#6355)
Removes reason generation when debug logging is not enabled.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Updates busybox SHA (#6365)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Query: Add +Inf bucket to query duration metrics (#6358)
* Query: Add +Inf bucket to query duration metrics
For the query duration metrics
(`thanos_store_api_query_duration_seconds`), we record query respond
latency, based on the size of the query (samples/series), and save to a
histogram.
However, when a query is made which exceeds the biggest sample/serie
size, we would prior to this commit, put the request into the largest
bucket.
With this commit, we instead create an `+Inf` bucket, and put requests
which are larger than the biggest defined bucket into that. This gives
more accurate results, and also allow one to see if the bucket sizes are
incorrectly sized.
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Tests: Mutex around non-thread safe random source
When creating test blocks, we use a non-thread safe random source, in
multiple goroutines. Due to this, tests would sometime panic.
This commits puts a mutex around calls using the same source, in order
to avoid this.
This should hopefully improve reliability of e2e tests.
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
---------
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Check context when expanding postings (#6363)
* check context when expanding postings
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* ui: only keep name in store_matches param (#6371)
We are doing store matching on the `name` field hence only keep that
field in the URL because otherwise the URL could get quite lengthy with
external labelsets inside of it.
Besides unit tests, I have also tested locally:
- Enable store filtering;
- Select store(-s);
- Copy/paste URL into the new tab and see that the same stores are
loaded like expected;
- See that URL only has names in them.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* docs: replace --store with --endpoint
Replace deprecated `--store` with `--endpoint` in docs.
Signed-off-by: Paul Gier <paul.gier@datastax.com>
* Optimizing "grafana generated" regex matchers (#6376)
* Opmizing Group Regex
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* fixing native histogram tests
Signed-off-by: Alan Protasio <alanprot@gmail.com>
---------
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Cache: various index cache client improvements (#6374)
* e2e(query): Reproduce dedup issue from #6257
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add dedup e2e test for Receive
With internal and external labels support.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Simplify generated blocks for query test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve query dedup test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Write a query test for dedup with sidecar
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Refactor query dedup test with sidecar
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix Receive query test
Now it properly ensures the double dedup works (on internal and external labels).
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix receive drawing
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add one extra test caes for query dedup from store
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Complement test for Receive query with dedup
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Complement test for Sidecar query dedup
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Query Explanation (#6346)
* Return Query Explaination in QueryAPI
A param `explain` is added to QueryAPI, if true then explanation
returned by the `Explain()` method of the query having structure
`ExplainOutputNode` is returned in response.
Query Explanation is added under new field in response that is
`thanosInfo`.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add explain checkbox in thanos UI
A explain checkbox is added to Thanos Query UI, that requests for
query explanation from thanos query api.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add ExpandableNode Component
ExpandableNode component renders Query Explanation in the thanos
UI. Requires a new package `react-accessible-treeview`.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Disable Explain checkbox on prometheus engine
Prometheus engine sends out error if toggle explain button. To
provide better experience, the explain checkbox get disbaled on
switching to prometheus engine and enable back on switching to
thanos engine.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add alert box with horizontal scrolling for Explanation
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Remove ExpandableNode and Add ListTree
Updates the design for query explanation box, removes
`ExpandableNode` and the dependency. Builts a new `ListTree` that
does the same using reactstrap and custom css.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Minor refactor in Query API response
`thanosInfo` is removed from Query reponse and used `explanation`
directly. `disableCheckbox` is also renamed to
`disableExplainCheckbox` in thanos UI.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update UI tests to passing
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Minor UI changes and test fix
UI improvements and Panel test fix other way around, resetting
the results on panel construction.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update promql-engine to use Explain method
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Build UI assets
Build UI assets, that runs new thanos UI with explain button.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Revert proxy url change from package.json
`proxy` was accidently changed and committed with package.json
when removed dependency. Hence, reverting it back.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Minor changes in UI
Fix requested changes in UI.
- Rename `state` and `setState` to `mapping` and `setMapping`.
- Rename `NodeTree` to `QueryTree`.
- Use unicode characters instead of `-` and `+`.
- Fix blue box on explain button.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update UI assets
Signed-off-by: Pradyumna Krishna <git@onpy.in>
---------
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379)
* Implementing Regex optimization on the MatchNotRegexp matcher type
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Opmizing MatchNotEqual
Signed-off-by: Alan Protasio <alanprot@gmail.com>
---------
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Expected failure of block label query dedup tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Put back the correct makefile
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* add de-cix as adopter (#6386)
Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de>
* Remove extra line that broke untouched test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add back line break at end of makefile
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix Receive single ingestor test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Reproduce dedup issue in Receive
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add even more test cases for dedup on store gw
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Reproduce dedup bug in Sidecar
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Reuse nginx image name
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Let all users read the metrics file from static metrics server
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* [chore] Updating Query Engine and Prometheus (#6392)
* Updating Query Engine
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* fix prometheus breaking change
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Update prometheus with https://github.com/prometheus/prometheus/pull/12387
Signed-off-by: Alan Protasio <alanprot@gmail.com>
---------
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777)
Signed-off-by: haanhvu <haanh6594@gmail.com>
* check context cancel when doing posting batches (#6396)
Signed-off-by: Ben Ye <benye@amazon.com>
* Expose store gateway query stats in series response hints (#6352)
* expose query stats hints
Signed-off-by: Ben Ye <benye@amazon.com>
* update
Signed-off-by: Ben Ye <benye@amazon.com>
* add query stats hints in result
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* add merge method
Signed-off-by: Ben Ye <benye@amazon.com>
* fix unit test
Signed-off-by: Ben Ye <benye@amazon.com>
modify hints proto
Signed-off-by: Ben Ye <benye@amazon.com>
fix unit test
Signed-off-by: Ben Ye <benye@amazon.com>
update format
Signed-off-by: Ben Ye <benye@amazon.com>
* update comments
Signed-off-by: Ben Ye <benye@amazon.com>
* try again
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* receive: make az aware ketama hashring (#6369)
* receive: make az aware ketama hashring
Signed-off-by: Alexander Rickardsson <alxric@aiven.io>
* receive: pass endpoints in hashring config as object
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* receive: add some tests for consistent hashing in presence of AZs
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* receive,docs: add migration note for az aware hashring
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
---------
Signed-off-by: Alexander Rickardsson <alxric@aiven.io>
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* Reformat asciiflow chart
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Proposal: query path tenancy (#6320)
* Add 1st version of query path tenancy proposal
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update proposal after initial feedback
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add cool picture
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Include example in cross tenant query complications
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve reasoning for why not using the QFE
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve writing in "How" section
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix owner profile link
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Address few more PR review comments
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Address feedback on flag name text
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update diagram
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve non-goals text
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update diagram
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update docs/proposals-accepted/202304-query-path-tenancy.md
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Clarify scenario for pitfalls of current solution
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Clarify that Store doesn't care about tenant label
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add an action plan
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Mention alternative idea of modifying Store API
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix typo
Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Address lots of feedback on the proposal
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Format query path tenancy proposal doc
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add a "Tenancy Model" subsection to "Goals"
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Mention header semanthics in comparison with gRPC message field
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve action plan structure and writing
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>
* Fix double-counting bug in http_request_duration metric (#6399)
* fix double-counting bug in http_request_duration metric
Signed-off-by: 4orty <kwk5178@gmail.com>
* Update Changelog
Signed-off-by: 4orty <kwk5178@gmail.com>
---------
Signed-off-by: 4orty <kwk5178@gmail.com>
* Replace group with resolution in compact metrics.
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.
Fixes: https://github.com/thanos-io/thanos/issues/5841
Signed-off-by: SuperQ <superq@gmail.com>
* Reuse static metrics server from e2e framework
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Run go mod tidy
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Updates busybox SHA (#6403)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Fix series stats merge (#6408)
* fix series stats merge
Signed-off-by: Ben Ye <benye@amazon.com>
* update license header
Signed-off-by: Ben Ye <benye@amazon.com>
* use reflect
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Receive: allow unlimited head_series_limit tenants (#6406)
With this commit we now allow to configure tenants with unlimited active
series limit by setting the limit to `0`. Prior to this commit setting a
per tenant limit to `0` would cause the tenant to be unable to write any
metrics at all.
This fixes: https://github.com/thanos-io/thanos/issues/6393
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* expose downloaded data size in query hints (#6409)
Signed-off-by: Ben Ye <benye@amazon.com>
* maintainers: add myself to triagers (#6414)
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
* Add `@douglascamata` to triagers (#6418)
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add Blog (#6411)
* Add LFX blog
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add Headers to blog
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Lint blog
Signed-off-by: Pradyumna Krishna <git@onpy.in>
---------
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* blog: Fix images for LFX post (#6422)
* blog: Fix images for LFX post
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* fix lint
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
---------
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* Index Cache: Change cache key for postings (#6405)
* extend postings cache key with codec
Signed-off-by: Ben Ye <benye@amazon.com>
* add changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* update code back
Signed-off-by: Ben Ye <benye@amazon.com>
* add colon
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* fix another test
Signed-off-by: Ben Ye <benye@amazon.com>
* add compression scheme const to remote index cache
Signed-off-by: Ben Ye <benye@amazon.com>
* address required comments
Signed-off-by: Ben Ye <benye@amazon.com>
* fix compression scheme name
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Receive: upgrading logs for failed uploads to error (#6427)
* FIX: upgrading log for failed upload to error
Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>
* docs: added changelog entry
Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>
---------
Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>
* fix postings test
Signed-off-by: Ben Ye <benye@amazon.com>
* Add aiven as adopter... more soon!
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* First draft of the Aiven adoption story
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Add aiven as adopter... more soon! (#6430)
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Report gRPC connnection errors to the caller (#6428)
By default `grpc.DialContext()` is non-blocking so any connection issue
will not be surfaced to the user. This change makes it blocking and
configures the gRPC dialer to report the underlying error if any
happens.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432)
* chore: remove duplicated `gopkg.in/fsnotify.v1` dep
`github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same
dependency. We can keep `github.com/fsnotify/fsnotify` and remove
`gopkg.in/fsnotify.v1`.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* docs: add changelog
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
---------
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Expose estimated chunk and series size as configurable options (#6426)
* expose estimated chunk and series size as configurable options
Signed-off-by: Ben Ye <benye@amazon.com>
* fix lint
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Receive: make tsdb stats limit configurable (#6437)
* Receive: make tsdb stats limit configurable
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
* Receive: make tsdb stats limit configurable
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
---------
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
* *: wire new Engine/Explain fields in query-frontend (#6433)
- Pass Engine/Explain fields in query-frontend codecs
- Add Engine field to QFE cache key
- Add e2e tests for all cases
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* index cache: Cache expanded postings (#6420)
* cache expanded postings in index cache
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* fix
Signed-off-by: Ben Ye <benye@amazon.com>
* fix lint
Signed-off-by: Ben Ye <benye@amazon.com>
* rebase main and added compression name to key
Signed-off-by: Ben Ye <benye@amazon.com>
* update key
Signed-off-by: Ben Ye <benye@amazon.com>
* add e2e test for memcached
Signed-off-by: Ben Ye <benye@amazon.com>
* fix cache config
Signed-off-by: Ben Ye <benye@amazon.com>
* address review comments
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* add approximate series size to index stats (#6425)
Signed-off-by: Ben Ye <benye@amazon.com>
* index stats: fix chunk size calculation (#6424)
Signed-off-by: Ben Ye <benye@amazon.com>
* Remove some unused Cortex vendored code and metrics (#6440)
* Avoid direct cortex vendor dependency in promclient (#6443)
* use own explanation struct
Signed-off-by: Ben Ye <benye@amazon.com>
* omit empty
Signed-off-by: Ben Ye <benye@amazon.com>
* fix e2e test
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351)
Fixes: #6280
Signed-off-by: Jonathan Davies <jpds@protonmail.com>
* Updates busybox SHA (#6448)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* store: read postings directly into delta encoded format (#6442)
Instead of allocating bytes for raw postings, let's read them directly
into diff varint format to save memory.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* compact: Add Index Stats to block metadata (#6441)
* compactor set index stats on block
Signed-off-by: Ben Ye <benye@amazon.com>
* comment
Signed-off-by: Ben Ye <benye@amazon.com>
* change field
Signed-off-by: Ben Ye <benye@amazon.com>
* update downsample
Signed-off-by: Ben Ye <benye@amazon.com>
* add index stats into compaction
Signed-off-by: Ben Ye <benye@amazon.com>
* fix tests
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* estimate block chunk and series size from metadata (#6449)
Signed-off-by: Ben Ye <benye@amazon.com>
* Adding graphic for blog showing architecture
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing mermaid graph
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing mermaid graph again
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Remove image and fix mermaid for other flowchart
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing prod flowchart
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing prod flowchart title
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing md table
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing md table again
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Adding clarification on cost breakdowns
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Adding clarification on cost breakdowns
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Store: fix crash on empty regex matcher
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* Some corrections on text and links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing diagams
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing currencies
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing diagams
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing images
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Website: Fix blog images width (#6450)
* Set blog post images width to 100%
Blog Post images width isn't set and hence the images doesn't get
fit to the page. Setting width to 100% make sure images fit to
the web page and screen size.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Remove width constraint from blog images
Signed-off-by: Pradyumna Krishna <git@onpy.in>
---------
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Cache calculated mint and maxt for each remote engine (#6458)
The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times
on each remote engine. These calculations can get expensive when an engine
covers a large amount of TSDBs.
This commit introduces caching of the calculated values for a remote
engine's mint, maxt and label sets.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* cache empty expanded postings (#6464)
Signed-off-by: Ben Ye <benye@amazon.com>
* mixin/receive: add limits alerting (#6466)
* add some thanos receive limits alerts
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* update changelog
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* fix and generate examples
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* quick fix
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* quick fix
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* fix test
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* fix invalid changelog
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
---------
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* Update docs/blog/2023-06-08-thanos-at-aiven.md
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Add retention for M3
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Add image versus mermaid diagram. Couldn't comment it out unfortunately
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Adding text around link to fix linter
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* make docs
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* add alert for tenant reaching head series limit (#6467)
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* go.mod: bump rest of otel libs (#6447)
* go.mod: bump rest of otel libs
Bump these two OTEL libs to be like the rest of the OTEL libs. We had a
strange issue where `service.name` was not set even though it was in
environment variables. Bumping these libs fixes the problem.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* otlp: only set service.name if provided
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* e2e: add test case
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
---------
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: disable pooling for postings benchmarks (#6473)
Pooling hides the real allocations in postings benchmarks because the
allocation only happens once really. In reality this is different when
multiple readers are created at once. Disable pooling in benchmarks to
understand what is happening.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Fixing date and author for Aiven blog (#6476)
* Fixing authors
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* make docs and fixing date
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing authors
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
---------
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Check context when expanding postings 2nd attempt (#6471)
* check context when expanding postings
Signed-off-by: Ben Ye <benye@amazon.com>
* import
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* [FEAT] Add thanos ruler blog post (#6477)
Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com>
* blog: Small fixes to Aiven article (#6481)
* Correct some links and typos in 2023-06-08-thanos-at-aiven.md
The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links.
Also fixed a few other typos I spotted.
Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com>
* Update 2023-06-08-thanos-at-aiven.md
Clarify a couple of things in the final paragraph.
Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com>
---------
Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com>
* Rename series and postings fetch duration metrics (#6479)
* fix wrong metric names for series and postings fetch duration
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* note breaking change
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Update to prometheus/common v0.44.0 (#6483)
Signed-off-by: Dan Bason <colonel.dan@gmail.com>
* go.mod: update rueidis client (#6485)
The latest improvements have great perf improvements. Let's update it.
Thank you to [rueian](https://github.com/rueian) and all of the
contributors to that awesome Redis client!
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Introduce tenancy package (#6482)
This commit introduces a tenancy package in preparation for introduction
of tenancy in the query path. Tenancy related code which can be common
between components are moved out of the receive component into the
tenancy package.
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Ignore nginx.com links in mdox (#6489)
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* blog: Update Aiven blog post (#6487)
Wording changes and better formatting for the tables.
Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io>
* store: optimized snappy streamed reading (#6475)
Optimize snappy streamed reading by traversing through the byte slice in
advance to determine how big of a buffer we will need. I have opted to
rewrite snappy streamed format decoding because it is a straightforward
format. Complex parts were deferred to klauspost/compress.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Remove Exists call in meta fetcher (#6474)
* Remove Exists call in meta fetcher
The meta fetcher lists the bucket by iterating through
top level paths from the entire keyspace, and then for each key
calls Exists on the meta.json file. This leads to an N+1 amplification
against object storage, manifesting as a very high increase in costs
and occasional throttling by cloud providers.
This commit changes the iteration logic to recursively list all files instead,
which can be done with far fewer API calls to object storage. Meta files
are then identified from the keys returned by the list operation instead
of making one API request from each file.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Revert e2e test change
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
---------
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* e2e/store: use now instead of time.Now() each time (#6493)
Consistently use the same timestamp instead of the current time each
time during a test.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* [FIX] Documentation issue (#6494)
Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com>
* Updates busybox SHA (#6497)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* chore: pkg imported more than once (#6499)
* chore: pkg imported more than once
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
* fix: remove pkg alias
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
---------
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
* *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496)
* Fix and improve PathContentReloader tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Run tests in parallel
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add changelog entry
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix table tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix typo
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
* add context check when decoding cached postings (#6506)
Signed-off-by: Ben Ye <benye@amazon.com>
* *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503)
* Add fallback config reload for symlinks
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve pooling config reload
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Make tests more reliable
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve tests again
Assert on runutil.Repeat output
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Pass the debounce/reload time to each test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add comments and ensure interfaces are implemented
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Make the limit configuration reload time customizable
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* goimports file
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix lint warning
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve log for polling engine
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Make linter happy
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Extract symlink identification to a separate function
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update changelog
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix links to go-grpc-middleware after v2 merge into main
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add period to make linter happy.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update changelog entry
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Replace the fsnotify based engine from `PathContentReloader` with the polling one
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Remove check for empty filePath
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Unexpose pollingEngine.Start
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix pollingEngine doc-comment
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Log file path when config is reloaded
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* receive: add float histogram support (#6323)
* Added receive float histogram support
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed imports
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Added comments for fns copied from Prometheus
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Improved comment
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Removed unnecessary if
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Fixed native histogram proto conversion in remote engine
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Removed unused histogram conversion from remote engine
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Fix and renaming in native_histograms_test.go
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Trigger Build
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
---------
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* go.mod: bump e2e framework version (#6516)
Bump e2e framework version and add a note about E2E_DOCKER_CPUS.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Add changelog for #6474 (#6518)
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* cortex/redisclient: use rueidis client (#6520)
* cortex/redisclient: use rueidis client
Use the same rueidis client in query-frontend. Solves
https://github.com/thanos-io/thanos/issues/6094.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* cortex/cache: gofumpt + fix errors.Errorf
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* cacheutil/docs: clean up more old stuff
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
---------
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* index header: Remove memWriter from fileWriter (#6509)
* index header: remove memWriter from fileWriter
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* refactor
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
* update comment
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Wrap object store Prometheus registry (#6152)
* Wrap object store Prometheus registery
In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update objstore to the latest version
This version removes the `thanos_` prefix from metrics, which is the
reason for wrapping the objstore's metrics registry in the first place.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix modules
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Wrap metrics registerer for objstore bucket
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Remove prefix from Thanos Store metrics
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix goimports
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Put back upgraded objstore dep
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Move to more recent ref of thanos/objstore
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Ignore OBS in objstore tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix linting error on main test after objstore upgrade
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Skip OCS objstore test in circle ci
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix echo in makefile
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix typo
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* e2e/store: try to fix Series() limit test again (#6522)
I finally managed to reproduce this failure locally with
https://github.com/efficientgo/e2e/commit/c316eb95ae5b0fa65b0be110f8cfc3c48e8810bb.
The added t.Logf() showed that is the problem is that with a lower bytes
limit, it might hit the series or chunks part first. I have bumped the
bytes limit. I calculated the new bytes limit by checking how much bytes
are allocated before sending the last chunk.
I have also noticed that one block is created without a delay. Update it
so that it would be like the others.
Include objstore@main update with
https://github.com/thanos-io/objstore/pull/62/files so that Iter() would
always return an error on a timeout.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* pkg/reloader: use watchInterval timeout for initial apply (#6519)
* pkg/reloader: use watchInterval timeout for initial apply
Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com>
* changelog
Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com>
* use distinct context for initial sync
Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com>
---------
Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com>
* Standardize index cache metrics (#6523)
* Add histogram metrics for index cache item size (#6528)
* add histogram metrics for index cache item size
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Upgrade objstore (#6507)
* Upgrade objstore
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
* Update with the latest APIs
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
---------
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
* Make compact lifecycle more flexible to be overridden for sharded compaction (#5964)
* Updates busybox SHA (#6534)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* BucketedBytes to buffer byte slices when decoding postings from cache (#6531)
* BucketedBytes to buffer byte slices when deconding postings from cache
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Fixing Lint
Signed-off-by: Alan Protasio <alanprot@gmail.com>
---------
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Shipper: change upload compacted type from bool to a function (#6526)
* change shipper upload compacted type from bool to a function
Signed-off-by: Ben Ye <benye@amazon.com>
* add default to false
Signed-off-by: Ben Ye <benye@amazon.com>
* reset uploadedCompacted to 0
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Query: Forward tenant information via StoreAPI (#6530)
* Querier: Forward tenant information downstream
With this commit we attach tenant information to each query request and
forward it via the StoreAPI to any downstream Store Gateways and
Queriers.
We add the following command lines options which mimics the tenant
functionality in Receive. The options are currently hidden, as they
provide no real functionality yet. This will come in future steps.
--query.tenant-header
--query.default-tenant
--query.tenant-certificate
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Receive: Use CertificateField from Tenancy pkg
These consts are now defined in the Tenancy package, so we should use
those instead.
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
---------
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Deduplicate matchers in posting group (#6532)
* deduplicate matchers in posting group
Signed-off-by: Ben Ye <benye@amazon.com>
* don't use map for deduplication
Signed-off-by: Ben Ye <benye@amazon.com>
* lint
Signed-off-by: Ben Ye <benye@amazon.com>
* address comments
Signed-off-by: Ben Ye <benye@amazon.com>
* fi…
subbarao-meduri
added a commit
to stolostron/thanos
that referenced
this pull request
Sep 21, 2023
* fix 0.31 changelog (#6278)
Signed-off-by: junot <junotxiang@kubesphere.io>
* Query: Switch Multiple Engines (#6234)
* Query: Switch engines using `engine` param
Thanos query has two engine, prometheus (default) and thanos.
A single engine runs through thanos query command at a time, and
have to re run the command to switch between.
This commit adds a functionality to run multiple engines at once
and switch between them using `engine` query param inq query api.
To avoid duplicate matrics registration, the thanos engine is
provided with a different registerer having prefix `tpe_` (not
been finalized yet).
promql-engine command line flag has been removed that specifies
the query engine to run.
Currently this functionality not implemented on GRPCAPI.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add multiple engine support to GRPCAPI
Fix build fail for thanos, adds support for multiple engine in
GRPCAPI.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Create QueryEngineFactory to create engines
QueryEngineFactory makes a collection for all promql engines used
by thanos and returns it. Any engine can be created and returned
using `GetXEngine` method.
It is currently limited to 2 engines prometheus and thanos engines
that get created on the first call.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Use QueryEngineFactory in query API
thanos query commands pass `QueryEngineFactory` to query apis
that will use engine based on query params. It will provide more
flexibility to create multiple engines in thanos.
Adds `defaultEngine` CLI flag, A default engine to use if not
specified with query params.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update Query API tests
Fixes breaking tests
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Minor changes and Docs fixes
* Move defaultEngine argument to reduce diff.
* Generated Docs.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add Engine Selector/ Dropdown to Query UI
Engine Selector is a dropdown that sets an engine to be used to
run the query. Currently two engines `thanos` and `prometheus`.
This dropdown sends a query param `engine` to query api, which
runs the api using the engine provided. Provided to run query
using multiple query engines from Query UI.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Move Engine Selector to Panel
Removes Dropdown component, and renders Engine Selector directly.
Receives defaultEngine from `flags` API.
Updates parseOptions to parse engine query param and updates test
for Panel and utils.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Upgrade promql-engine dependency
Updates promql-engine that brings functionality to provide
fallback engine using enigne Opts.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add MinT to remote client
MinT method was missing from Client due to updated promql-engine.
This commits adds mint to the remote client.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Use prometheus fallback engine in thanos engine
Thanos engine creates a fallback prometheus engine that conflicts
with another prometheus engine created by thanos, while
registering metrics. To fix this, provided created thanos engine
as fallback engine to thanos engine in engine Opts.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Use enum for EngineType in GRPC
GRPC is used for communication between thanos components and
defaultEngine was a string before. Enum makes more sense, and
hence the request.Enigne type has been changed to
querypb.EngineType.
Default case is handled with another default value provided in
the enum.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update query UI bindata.go
Compile react app using `make assets`.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
---------
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* docs: mismatch in changelog
Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com>
* Updates busybox SHA (#6283)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287)
* Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346
Signed-off-by: Alex Le <leqiyue@amazon.com>
* Reverted test code
Signed-off-by: Alex Le <leqiyue@amazon.com>
* Updated comment
Signed-off-by: Alex Le <leqiyue@amazon.com>
* docs: mismatch in changelog
Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
* Updates busybox SHA (#6283)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
* trigger workflow
Signed-off-by: Alex Le <leqiyue@amazon.com>
* trigger workflow
Signed-off-by: Alex Le <leqiyue@amazon.com>
---------
Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com>
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Add CarTrade Tech as new adopter
Signed-off-by: naveadkazi <navead@carwale.com>
* sort ProxyResponseHeap comparing only diff labels
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* proxy heap: remove store labels before comparing response series
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* tests: Remove custom Between test matcher (#6310)
* Remove custom Between test matcher
The upstream PR to efficientgo/e2e has been merged, so we can use it from there.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Run go mod tidy
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Remove testingRespSet and update comparison func
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* Fix tests
Signed-off-by: Walther Lee <walthere.lee@gmail.com>
* query frontend, query UI: Native histogram support (#6071)
* Implemented native histogram support for qfe and query UI
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed marshalling for histograms in qfe
Started working on native histogram query ui
Copied histogram implementation for graph
Added query range support for native histograms in qfe
Use prom model (un-)marshal for native histograms in qfe
Use prom model (un-)marshal for native histograms in qfe
Fixed sample and sample stream marshal fn
Extended qfe native histogram e2e tests
Added copyright to qfe queryrange compat
Added query range test fo histograms and try to fix ui tests
Fixed DataTable test
Review feedback
Fixed native histogram e2e test
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Add histogram support for ApplyCounterResetsSeriesIterator
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Made assets
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Add chnagelog
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed changelog
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed qfe
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed PrometheusResponse minTime for histograms in qfe
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Updated prometheus common to v0.40.0 and queryrange.Sample fixes
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Updated Readme
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Addressed PR comments
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
trigger tests
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Made assets
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Made assets
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* fixed tsdbutil references
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* fixed imports
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Enabled pushdown for query native hist test and removed ToDo
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Refactored native histogram query UI
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
---------
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* store: add streamed snappy encoding for postings list (#6303)
* store: add streamed snappy encoding for postings list
We've noticed that decoding Snappy compressed postings list
takes a lot of RAM:
```
(pprof) top
Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total
Dropped 1069 nodes (cum <= 10.56GB)
Showing top 10 nodes out of 82
flat flat% sum% cum cum%
0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1
2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings
1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings
2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings
1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode
1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode
```
This is because we are creating a new []byte slice for the decoded data
each time. To avoid this RAM usage problem, let's stream the decoding
from a given buffer. Since Snappy block format doesn't support streamed
decoding, let's switch to Snappy stream format which is made for exactly
that.
Notice that our current `index.Postings` list does not
support going back through Seek() even if theoretically one could want
something like that. Fortunately, to search for posting intersection, we
need to only go forward.
Benchmark data:
```
name time/op
PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3%
PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4%
PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1%
PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4%
PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1%
PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13%
PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1%
PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4%
PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1%
PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4%
name alloc/op
PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0%
name allocs/op
PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0%
```
Compression ratios are still the same like previously:
```
$ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store
[snip]
=== RUN TestDiffVarintCodec/snappy/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44498 bytes
postings_codec_test.go:81: ratio: 0.012
=== RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44670 bytes
postings_codec_test.go:81: ratio: 0.012
```
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: clean up postings code
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: fix estimation
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: use buffer.Bytes()
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store/postings_codec: reuse extgrpc compressors/decompressors
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* CHANGELOG: add item
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* CHANGELOG: clean up whitespace
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
---------
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* compact: atomically replace no compact marked map (#6319)
With lots of blocks it could take some time to fill this no compact
marked map hence replace it atomically. I believe this leads to problems
in the compaction planner where it picks up no compact marked blocks
because meta syncer does synchronizations concurrently.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Reuse buffers for label comparison
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Use len instead of cap
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Use len instead of cap
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Fixed Jaeger exporter specification link (#6324)
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* receive: Make max backoff configurable (#6163)
Signed-off-by: Philip Gough <philip.p.gough@gmail.com>
* Replace summary in extprom metrics with histogram (#6327)
* Replaced summary in extprom metrics with histogram
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Added changelog
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Removed unused parameters from NewInstrumentationMiddleware
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Reverted NewInstrumentationMiddleware
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
---------
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Expose info for each TSDB
This commit exposes the label set alongside the min and max time
for each TSDB covered by a Store.
This information is used to scope the min time for a remote query
so that we do not produce partial aggregates in distriuted mode.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Add test case for proxy store
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Pass engine factory by reference
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Cleanup
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Use NewQueryEngineFactory in tests
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Avoid expensive log.Valuer evaluation for disallowed levels (#6322)
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
* Fix inconsistent error for series limits in Store API (#6330)
* store: fix inconsistent error for series limits
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* update changelog
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* Update pkg/store/bucket.go
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* Update pkg/store/bucket.go
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* rename labelValues serires liimiter test function
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
---------
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* *: remove unmaintained gzip library (#6332)
Switch from nytimes gzip library to the klaustpost's gzip code. The old
gzip HTTP handler shows up a lot in allocs so that's how I ended up
doing this change.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Traces sampler env var (#6306)
* Issue#5947 OTEL_TRACES_SAMPLER env var
Signed-off-by: shayyxi <shazi12384@gmail.com>
* Test correction
Signed-off-by: shayyxi <shazi12384@gmail.com>
* doc failure correction. parse float argument correction.
Signed-off-by: shayyxi <shazi12384@gmail.com>
* added the changelog.
Signed-off-by: shayyxi <shazi12384@gmail.com>
* ran make docs to fix the build failure.
Signed-off-by: shayyxi <shazi12384@gmail.com>
* corrected the incorrect change in tools.md
Signed-off-by: shayyxi <shazi12384@gmail.com>
* fixed review comments.
Signed-off-by: shayyxi <shazi12384@gmail.com>
---------
Signed-off-by: shayyxi <shazi12384@gmail.com>
Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com>
Co-authored-by: shayyxi <shazi12384@gmail.com>
* query: use storepb.SeriesServer (#6334)
Use storepb.SeriesServer instead of the concrete struct. This allows
implementing functionality on top of the proxy.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342)
* use github.com/onsi/gomega/gleak to detect goroutine leak with timeout
Signed-off-by: Rueian <rueiancsie@gmail.com>
* fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345)
Signed-off-by: aimuz <mr.imuz@gmail.com>
* Return grpc code resource exhausted for byte limit error (#6325)
* return grpc code resource exhausted for byte limit error
Signed-off-by: Ben Ye <benye@amazon.com>
* fix lint
Signed-off-by: Ben Ye <benye@amazon.com>
* update partial response strategy
Signed-off-by: Ben Ye <benye@amazon.com>
* fix limit
Signed-off-by: Ben Ye <benye@amazon.com>
* try to fix tests
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test error message
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349)
Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>
* Optimize generating store match reason (#6355)
Removes reason generation when debug logging is not enabled.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Updates busybox SHA (#6365)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Query: Add +Inf bucket to query duration metrics (#6358)
* Query: Add +Inf bucket to query duration metrics
For the query duration metrics
(`thanos_store_api_query_duration_seconds`), we record query respond
latency, based on the size of the query (samples/series), and save to a
histogram.
However, when a query is made which exceeds the biggest sample/serie
size, we would prior to this commit, put the request into the largest
bucket.
With this commit, we instead create an `+Inf` bucket, and put requests
which are larger than the biggest defined bucket into that. This gives
more accurate results, and also allow one to see if the bucket sizes are
incorrectly sized.
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Tests: Mutex around non-thread safe random source
When creating test blocks, we use a non-thread safe random source, in
multiple goroutines. Due to this, tests would sometime panic.
This commits puts a mutex around calls using the same source, in order
to avoid this.
This should hopefully improve reliability of e2e tests.
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
---------
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Check context when expanding postings (#6363)
* check context when expanding postings
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* ui: only keep name in store_matches param (#6371)
We are doing store matching on the `name` field hence only keep that
field in the URL because otherwise the URL could get quite lengthy with
external labelsets inside of it.
Besides unit tests, I have also tested locally:
- Enable store filtering;
- Select store(-s);
- Copy/paste URL into the new tab and see that the same stores are
loaded like expected;
- See that URL only has names in them.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* docs: replace --store with --endpoint
Replace deprecated `--store` with `--endpoint` in docs.
Signed-off-by: Paul Gier <paul.gier@datastax.com>
* Optimizing "grafana generated" regex matchers (#6376)
* Opmizing Group Regex
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* fixing native histogram tests
Signed-off-by: Alan Protasio <alanprot@gmail.com>
---------
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Cache: various index cache client improvements (#6374)
* e2e(query): Reproduce dedup issue from #6257
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add dedup e2e test for Receive
With internal and external labels support.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Simplify generated blocks for query test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve query dedup test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Write a query test for dedup with sidecar
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Refactor query dedup test with sidecar
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix Receive query test
Now it properly ensures the double dedup works (on internal and external labels).
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix receive drawing
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add one extra test caes for query dedup from store
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Complement test for Receive query with dedup
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Complement test for Sidecar query dedup
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Query Explanation (#6346)
* Return Query Explaination in QueryAPI
A param `explain` is added to QueryAPI, if true then explanation
returned by the `Explain()` method of the query having structure
`ExplainOutputNode` is returned in response.
Query Explanation is added under new field in response that is
`thanosInfo`.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add explain checkbox in thanos UI
A explain checkbox is added to Thanos Query UI, that requests for
query explanation from thanos query api.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add ExpandableNode Component
ExpandableNode component renders Query Explanation in the thanos
UI. Requires a new package `react-accessible-treeview`.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Disable Explain checkbox on prometheus engine
Prometheus engine sends out error if toggle explain button. To
provide better experience, the explain checkbox get disbaled on
switching to prometheus engine and enable back on switching to
thanos engine.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add alert box with horizontal scrolling for Explanation
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Remove ExpandableNode and Add ListTree
Updates the design for query explanation box, removes
`ExpandableNode` and the dependency. Builts a new `ListTree` that
does the same using reactstrap and custom css.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Minor refactor in Query API response
`thanosInfo` is removed from Query reponse and used `explanation`
directly. `disableCheckbox` is also renamed to
`disableExplainCheckbox` in thanos UI.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update UI tests to passing
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Minor UI changes and test fix
UI improvements and Panel test fix other way around, resetting
the results on panel construction.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update promql-engine to use Explain method
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Build UI assets
Build UI assets, that runs new thanos UI with explain button.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Revert proxy url change from package.json
`proxy` was accidently changed and committed with package.json
when removed dependency. Hence, reverting it back.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Minor changes in UI
Fix requested changes in UI.
- Rename `state` and `setState` to `mapping` and `setMapping`.
- Rename `NodeTree` to `QueryTree`.
- Use unicode characters instead of `-` and `+`.
- Fix blue box on explain button.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Update UI assets
Signed-off-by: Pradyumna Krishna <git@onpy.in>
---------
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379)
* Implementing Regex optimization on the MatchNotRegexp matcher type
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Opmizing MatchNotEqual
Signed-off-by: Alan Protasio <alanprot@gmail.com>
---------
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Expected failure of block label query dedup tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Put back the correct makefile
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* add de-cix as adopter (#6386)
Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de>
* Remove extra line that broke untouched test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add back line break at end of makefile
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix Receive single ingestor test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Reproduce dedup issue in Receive
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add even more test cases for dedup on store gw
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Reproduce dedup bug in Sidecar
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Reuse nginx image name
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Let all users read the metrics file from static metrics server
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* [chore] Updating Query Engine and Prometheus (#6392)
* Updating Query Engine
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* fix prometheus breaking change
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Update prometheus with https://github.com/prometheus/prometheus/pull/12387
Signed-off-by: Alan Protasio <alanprot@gmail.com>
---------
Signed-off-by: Alan Protasio <alanprot@gmail.com>
* Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777)
Signed-off-by: haanhvu <haanh6594@gmail.com>
* check context cancel when doing posting batches (#6396)
Signed-off-by: Ben Ye <benye@amazon.com>
* Expose store gateway query stats in series response hints (#6352)
* expose query stats hints
Signed-off-by: Ben Ye <benye@amazon.com>
* update
Signed-off-by: Ben Ye <benye@amazon.com>
* add query stats hints in result
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* add merge method
Signed-off-by: Ben Ye <benye@amazon.com>
* fix unit test
Signed-off-by: Ben Ye <benye@amazon.com>
modify hints proto
Signed-off-by: Ben Ye <benye@amazon.com>
fix unit test
Signed-off-by: Ben Ye <benye@amazon.com>
update format
Signed-off-by: Ben Ye <benye@amazon.com>
* update comments
Signed-off-by: Ben Ye <benye@amazon.com>
* try again
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* receive: make az aware ketama hashring (#6369)
* receive: make az aware ketama hashring
Signed-off-by: Alexander Rickardsson <alxric@aiven.io>
* receive: pass endpoints in hashring config as object
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* receive: add some tests for consistent hashing in presence of AZs
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* receive,docs: add migration note for az aware hashring
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
---------
Signed-off-by: Alexander Rickardsson <alxric@aiven.io>
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* Reformat asciiflow chart
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Proposal: query path tenancy (#6320)
* Add 1st version of query path tenancy proposal
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update proposal after initial feedback
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add cool picture
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Include example in cross tenant query complications
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve reasoning for why not using the QFE
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve writing in "How" section
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix owner profile link
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Address few more PR review comments
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Address feedback on flag name text
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update diagram
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve non-goals text
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update diagram
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update docs/proposals-accepted/202304-query-path-tenancy.md
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Clarify scenario for pitfalls of current solution
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Clarify that Store doesn't care about tenant label
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add an action plan
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Mention alternative idea of modifying Store API
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix typo
Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Address lots of feedback on the proposal
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Format query path tenancy proposal doc
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add a "Tenancy Model" subsection to "Goals"
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Mention header semanthics in comparison with gRPC message field
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve action plan structure and writing
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>
* Fix double-counting bug in http_request_duration metric (#6399)
* fix double-counting bug in http_request_duration metric
Signed-off-by: 4orty <kwk5178@gmail.com>
* Update Changelog
Signed-off-by: 4orty <kwk5178@gmail.com>
---------
Signed-off-by: 4orty <kwk5178@gmail.com>
* Replace group with resolution in compact metrics.
Compaction metrics have too high a cardinality, causing metric bloat on
large installations. The group information is better suited to logs.
* Replace with a `resolution` label to the compaction counters.
Fixes: https://github.com/thanos-io/thanos/issues/5841
Signed-off-by: SuperQ <superq@gmail.com>
* Reuse static metrics server from e2e framework
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Run go mod tidy
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Updates busybox SHA (#6403)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* Fix series stats merge (#6408)
* fix series stats merge
Signed-off-by: Ben Ye <benye@amazon.com>
* update license header
Signed-off-by: Ben Ye <benye@amazon.com>
* use reflect
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Receive: allow unlimited head_series_limit tenants (#6406)
With this commit we now allow to configure tenants with unlimited active
series limit by setting the limit to `0`. Prior to this commit setting a
per tenant limit to `0` would cause the tenant to be unable to write any
metrics at all.
This fixes: https://github.com/thanos-io/thanos/issues/6393
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* expose downloaded data size in query hints (#6409)
Signed-off-by: Ben Ye <benye@amazon.com>
* maintainers: add myself to triagers (#6414)
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
* Add `@douglascamata` to triagers (#6418)
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add Blog (#6411)
* Add LFX blog
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Add Headers to blog
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Lint blog
Signed-off-by: Pradyumna Krishna <git@onpy.in>
---------
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* blog: Fix images for LFX post (#6422)
* blog: Fix images for LFX post
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* fix lint
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
---------
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* Index Cache: Change cache key for postings (#6405)
* extend postings cache key with codec
Signed-off-by: Ben Ye <benye@amazon.com>
* add changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* update code back
Signed-off-by: Ben Ye <benye@amazon.com>
* add colon
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* fix another test
Signed-off-by: Ben Ye <benye@amazon.com>
* add compression scheme const to remote index cache
Signed-off-by: Ben Ye <benye@amazon.com>
* address required comments
Signed-off-by: Ben Ye <benye@amazon.com>
* fix compression scheme name
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Receive: upgrading logs for failed uploads to error (#6427)
* FIX: upgrading log for failed upload to error
Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>
* docs: added changelog entry
Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>
---------
Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>
* fix postings test
Signed-off-by: Ben Ye <benye@amazon.com>
* Add aiven as adopter... more soon!
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* First draft of the Aiven adoption story
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Add aiven as adopter... more soon! (#6430)
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Report gRPC connnection errors to the caller (#6428)
By default `grpc.DialContext()` is non-blocking so any connection issue
will not be surfaced to the user. This change makes it blocking and
configures the gRPC dialer to report the underlying error if any
happens.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432)
* chore: remove duplicated `gopkg.in/fsnotify.v1` dep
`github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same
dependency. We can keep `github.com/fsnotify/fsnotify` and remove
`gopkg.in/fsnotify.v1`.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* docs: add changelog
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
---------
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Expose estimated chunk and series size as configurable options (#6426)
* expose estimated chunk and series size as configurable options
Signed-off-by: Ben Ye <benye@amazon.com>
* fix lint
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Receive: make tsdb stats limit configurable (#6437)
* Receive: make tsdb stats limit configurable
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
* Receive: make tsdb stats limit configurable
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
---------
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
* *: wire new Engine/Explain fields in query-frontend (#6433)
- Pass Engine/Explain fields in query-frontend codecs
- Add Engine field to QFE cache key
- Add e2e tests for all cases
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* index cache: Cache expanded postings (#6420)
* cache expanded postings in index cache
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* fix
Signed-off-by: Ben Ye <benye@amazon.com>
* fix lint
Signed-off-by: Ben Ye <benye@amazon.com>
* rebase main and added compression name to key
Signed-off-by: Ben Ye <benye@amazon.com>
* update key
Signed-off-by: Ben Ye <benye@amazon.com>
* add e2e test for memcached
Signed-off-by: Ben Ye <benye@amazon.com>
* fix cache config
Signed-off-by: Ben Ye <benye@amazon.com>
* address review comments
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* add approximate series size to index stats (#6425)
Signed-off-by: Ben Ye <benye@amazon.com>
* index stats: fix chunk size calculation (#6424)
Signed-off-by: Ben Ye <benye@amazon.com>
* Remove some unused Cortex vendored code and metrics (#6440)
* Avoid direct cortex vendor dependency in promclient (#6443)
* use own explanation struct
Signed-off-by: Ben Ye <benye@amazon.com>
* omit empty
Signed-off-by: Ben Ye <benye@amazon.com>
* fix e2e test
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351)
Fixes: #6280
Signed-off-by: Jonathan Davies <jpds@protonmail.com>
* Updates busybox SHA (#6448)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* store: read postings directly into delta encoded format (#6442)
Instead of allocating bytes for raw postings, let's read them directly
into diff varint format to save memory.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* compact: Add Index Stats to block metadata (#6441)
* compactor set index stats on block
Signed-off-by: Ben Ye <benye@amazon.com>
* comment
Signed-off-by: Ben Ye <benye@amazon.com>
* change field
Signed-off-by: Ben Ye <benye@amazon.com>
* update downsample
Signed-off-by: Ben Ye <benye@amazon.com>
* add index stats into compaction
Signed-off-by: Ben Ye <benye@amazon.com>
* fix tests
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* estimate block chunk and series size from metadata (#6449)
Signed-off-by: Ben Ye <benye@amazon.com>
* Adding graphic for blog showing architecture
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing mermaid graph
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing mermaid graph again
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Remove image and fix mermaid for other flowchart
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing prod flowchart
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing prod flowchart title
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing md table
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing md table again
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Adding clarification on cost breakdowns
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Adding clarification on cost breakdowns
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Store: fix crash on empty regex matcher
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
* Some corrections on text and links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing links
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing diagams
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing currencies
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing diagams
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing images
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Website: Fix blog images width (#6450)
* Set blog post images width to 100%
Blog Post images width isn't set and hence the images doesn't get
fit to the page. Setting width to 100% make sure images fit to
the web page and screen size.
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Remove width constraint from blog images
Signed-off-by: Pradyumna Krishna <git@onpy.in>
---------
Signed-off-by: Pradyumna Krishna <git@onpy.in>
* Cache calculated mint and maxt for each remote engine (#6458)
The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times
on each remote engine. These calculations can get expensive when an engine
covers a large amount of TSDBs.
This commit introduces caching of the calculated values for a remote
engine's mint, maxt and label sets.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* cache empty expanded postings (#6464)
Signed-off-by: Ben Ye <benye@amazon.com>
* mixin/receive: add limits alerting (#6466)
* add some thanos receive limits alerts
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* update changelog
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* fix and generate examples
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* quick fix
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* quick fix
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* fix test
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* fix invalid changelog
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
---------
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* Update docs/blog/2023-06-08-thanos-at-aiven.md
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Add retention for M3
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Add image versus mermaid diagram. Couldn't comment it out unfortunately
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Adding text around link to fix linter
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* make docs
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* add alert for tenant reaching head series limit (#6467)
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
* go.mod: bump rest of otel libs (#6447)
* go.mod: bump rest of otel libs
Bump these two OTEL libs to be like the rest of the OTEL libs. We had a
strange issue where `service.name` was not set even though it was in
environment variables. Bumping these libs fixes the problem.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* otlp: only set service.name if provided
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* e2e: add test case
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
---------
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* store: disable pooling for postings benchmarks (#6473)
Pooling hides the real allocations in postings benchmarks because the
allocation only happens once really. In reality this is different when
multiple readers are created at once. Disable pooling in benchmarks to
understand what is happening.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Fixing date and author for Aiven blog (#6476)
* Fixing authors
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* make docs and fixing date
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Fixing authors
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
---------
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
* Check context when expanding postings 2nd attempt (#6471)
* check context when expanding postings
Signed-off-by: Ben Ye <benye@amazon.com>
* import
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* [FEAT] Add thanos ruler blog post (#6477)
Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com>
* blog: Small fixes to Aiven article (#6481)
* Correct some links and typos in 2023-06-08-thanos-at-aiven.md
The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links.
Also fixed a few other typos I spotted.
Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com>
* Update 2023-06-08-thanos-at-aiven.md
Clarify a couple of things in the final paragraph.
Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com>
---------
Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com>
* Rename series and postings fetch duration metrics (#6479)
* fix wrong metric names for series and postings fetch duration
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* note breaking change
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Update to prometheus/common v0.44.0 (#6483)
Signed-off-by: Dan Bason <colonel.dan@gmail.com>
* go.mod: update rueidis client (#6485)
The latest improvements have great perf improvements. Let's update it.
Thank you to [rueian](https://github.com/rueian) and all of the
contributors to that awesome Redis client!
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Introduce tenancy package (#6482)
This commit introduces a tenancy package in preparation for introduction
of tenancy in the query path. Tenancy related code which can be common
between components are moved out of the receive component into the
tenancy package.
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
* Ignore nginx.com links in mdox (#6489)
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
* blog: Update Aiven blog post (#6487)
Wording changes and better formatting for the tables.
Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io>
* store: optimized snappy streamed reading (#6475)
Optimize snappy streamed reading by traversing through the byte slice in
advance to determine how big of a buffer we will need. I have opted to
rewrite snappy streamed format decoding because it is a straightforward
format. Complex parts were deferred to klauspost/compress.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Remove Exists call in meta fetcher (#6474)
* Remove Exists call in meta fetcher
The meta fetcher lists the bucket by iterating through
top level paths from the entire keyspace, and then for each key
calls Exists on the meta.json file. This leads to an N+1 amplification
against object storage, manifesting as a very high increase in costs
and occasional throttling by cloud providers.
This commit changes the iteration logic to recursively list all files instead,
which can be done with far fewer API calls to object storage. Meta files
are then identified from the keys returned by the list operation instead
of making one API request from each file.
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* Revert e2e test change
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
---------
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* e2e/store: use now instead of time.Now() each time (#6493)
Consistently use the same timestamp instead of the current time each
time during a test.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* [FIX] Documentation issue (#6494)
Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com>
* Updates busybox SHA (#6497)
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
* chore: pkg imported more than once (#6499)
* chore: pkg imported more than once
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
* fix: remove pkg alias
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
---------
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
* *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496)
* Fix and improve PathContentReloader tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Run tests in parallel
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add changelog entry
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix table tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix typo
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
* add context check when decoding cached postings (#6506)
Signed-off-by: Ben Ye <benye@amazon.com>
* *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503)
* Add fallback config reload for symlinks
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve pooling config reload
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Make tests more reliable
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve tests again
Assert on runutil.Repeat output
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Pass the debounce/reload time to each test
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add comments and ensure interfaces are implemented
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Make the limit configuration reload time customizable
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* goimports file
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix lint warning
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Improve log for polling engine
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Make linter happy
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Extract symlink identification to a separate function
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update changelog
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix links to go-grpc-middleware after v2 merge into main
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Add period to make linter happy.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update changelog entry
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Replace the fsnotify based engine from `PathContentReloader` with the polling one
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Remove check for empty filePath
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Unexpose pollingEngine.Start
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix pollingEngine doc-comment
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Log file path when config is reloaded
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* receive: add float histogram support (#6323)
* Added receive float histogram support
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Fixed imports
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Added comments for fns copied from Prometheus
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Improved comment
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Removed unnecessary if
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Fixed native histogram proto conversion in remote engine
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Removed unused histogram conversion from remote engine
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Fix and renaming in native_histograms_test.go
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* Trigger Build
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
---------
Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
* go.mod: bump e2e framework version (#6516)
Bump e2e framework version and add a note about E2E_DOCKER_CPUS.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Add changelog for #6474 (#6518)
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
* cortex/redisclient: use rueidis client (#6520)
* cortex/redisclient: use rueidis client
Use the same rueidis client in query-frontend. Solves
https://github.com/thanos-io/thanos/issues/6094.
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* cortex/cache: gofumpt + fix errors.Errorf
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* cacheutil/docs: clean up more old stuff
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
---------
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* index header: Remove memWriter from fileWriter (#6509)
* index header: remove memWriter from fileWriter
Signed-off-by: Ben Ye <benye@amazon.com>
* update changelog
Signed-off-by: Ben Ye <benye@amazon.com>
* refactor
Signed-off-by: Ben Ye <benye@amazon.com>
* fix test
Signed-off-by: Ben Ye <benye@amazon.com>
* update comment
Signed-off-by: Ben Ye <benye@amazon.com>
---------
Signed-off-by: Ben Ye <benye@amazon.com>
* Wrap object store Prometheus registry (#6152)
* Wrap object store Prometheus registery
In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Update objstore to the latest version
This version removes the `thanos_` prefix from metrics, which is the
reason for wrapping the objstore's metrics registry in the first place.
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix modules
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Wrap metrics registerer for objstore bucket
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Remove prefix from Thanos Store metrics
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix goimports
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Put back upgraded objstore dep
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Rerun CI
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Move to more recent ref of thanos/objstore
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Ignore OBS in objstore tests
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix linting error on main test after objstore upgrade
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Skip OCS objstore test in circle ci
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix echo in makefile
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* Fix typo
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
---------
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
* e2e/store: try to fix Series() limit test again (#6522)
I finally managed to reproduce this failure locally with
http…
fpetkovski
added a commit
to fpetkovski/thanos
that referenced
this pull request
Jan 26, 2024
…hanos-io#199) * Replace summary in extprom metrics with histogram (thanos-io#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (thanos-io#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (thanos-io#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (thanos-io#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (thanos-io#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (thanos-io#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. redis/rueidis#209 (thanos-io#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (thanos-io#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (thanos-io#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Bump promql-engine to fix thanos-io/promql-engine#239 (thanos-io#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Updates busybox SHA (thanos-io#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (thanos-io#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * e2e(query): Reproduce dedup issue from thanos-io#6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Check context when expanding postings (thanos-io#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (thanos-io#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (thanos-io#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (thanos-io#6374) * Query Explanation (thanos-io#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (thanos-io#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (thanos-io#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * [chore] Updating Query Engine and Prometheus (thanos-io#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with prometheus/prometheus#12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (thanos-io#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (thanos-io#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (thanos-io#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (thanos-io#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Proposal: query path tenancy (thanos-io#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (thanos-io#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Updates busybox SHA (thanos-io#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (thanos-io#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (thanos-io#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: thanos-io#6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (thanos-io#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (thanos-io#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (thanos-io#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (thanos-io#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (thanos-io#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (thanos-io#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (thanos-io#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! (thanos-io#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (thanos-io#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (thanos-io#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (thanos-io#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (thanos-io#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (thanos-io#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (thanos-io#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (thanos-io#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (thanos-io#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (thanos-io#6440) * Fixed DefaultPromConfig * Fixed imports * Back to diffVarintSnappyEncode * Merge pull request thanos-io#180 from Shopify/optimize-timerange-calculation Cache calculated mint and maxt for each remote engine * Updated busybox * fixing lint * Fixing merge conflict Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixing missing import Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * fix lint again Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * resolving conflict merges Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> * Fixed import and fn order * Fixed unit tests * Updated promdoc.sum * Back to custom promql engine * Removed custom promql engine and moved to latest upstream * Ran go mod tidy * Fixed GetQueryAPIClients * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Signed-off-by: Rueian <rueiancsie@gmail.com> Signed-off-by: aimuz <mr.imuz@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Signed-off-by: Paul Gier <paul.gier@datastax.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Pradyumna Krishna <git@onpy.in> Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Signed-off-by: 4orty <kwk5178@gmail.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> Signed-off-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Co-authored-by: Sebastian Rabenhorst <4246554+rabenhorst@users.noreply.github.com> Co-authored-by: Xiaochao Dong <the.xcdong@gmail.com> Co-authored-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Co-authored-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: aimuz <mr.imuz@gmail.com> Co-authored-by: Ben Ye <benye@amazon.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Alban Hurtaud <alban.hurtaud@amadeus.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Co-authored-by: Jacob Baungård Hansen <jacobbaungard@redhat.com> Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Paul Gier <paul.gier@datastax.com> Co-authored-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Pradyumna Krishna <git@onpy.in> Co-authored-by: Raúl Garcia Sanchez <info@raulgarcia.de> Co-authored-by: Ha Anh Vu <75315486+haanhvu@users.noreply.github.com> Co-authored-by: Alexander Rickardsson <alxric@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Co-authored-by: Wonki Kim <kwk5178@gmail.com> Co-authored-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Victor Hugo Brito Fernandes <victorhbfernandes@gmail.com> Co-authored-by: Jonah Kowall <jkowall@kowall.net> Co-authored-by: Simon Pasquier <spasquie@redhat.com> Co-authored-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
fpetkovski
pushed a commit
to fpetkovski/thanos
that referenced
this pull request
Jan 26, 2024
Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change.