Skip to content

Commit

Permalink
Merge upstream 0651f33 excluding change to series sorting in proxy (t…
Browse files Browse the repository at this point in the history
…hanos-io#199)

* Replace summary in extprom metrics with histogram (thanos-io#6327)

* Replaced summary in extprom metrics with histogram

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Added changelog

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Removed unused parameters from NewInstrumentationMiddleware

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Reverted NewInstrumentationMiddleware

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

---------

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Avoid expensive log.Valuer evaluation for disallowed levels (thanos-io#6322)

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

* Fix inconsistent error for series limits in Store API (thanos-io#6330)

* store: fix inconsistent error for series limits

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

* update changelog

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

* Update pkg/store/bucket.go

Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

* Update pkg/store/bucket.go

Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

* rename labelValues serires liimiter test function

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

---------

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* *: remove unmaintained gzip library (thanos-io#6332)

Switch from nytimes gzip library to the klaustpost's gzip code. The old
gzip HTTP handler shows up a lot in allocs so that's how I ended up
doing this change.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Traces sampler env var (thanos-io#6306)

* Issue#5947 OTEL_TRACES_SAMPLER env var

Signed-off-by: shayyxi <shazi12384@gmail.com>

* Test correction

Signed-off-by: shayyxi <shazi12384@gmail.com>

* doc failure correction. parse float argument correction.

Signed-off-by: shayyxi <shazi12384@gmail.com>

* added the changelog.

Signed-off-by: shayyxi <shazi12384@gmail.com>

* ran make docs to fix the build failure.

Signed-off-by: shayyxi <shazi12384@gmail.com>

* corrected the incorrect change in tools.md

Signed-off-by: shayyxi <shazi12384@gmail.com>

* fixed review comments.

Signed-off-by: shayyxi <shazi12384@gmail.com>

---------

Signed-off-by: shayyxi <shazi12384@gmail.com>
Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com>
Co-authored-by: shayyxi <shazi12384@gmail.com>

* query: use storepb.SeriesServer (thanos-io#6334)

Use storepb.SeriesServer instead of the concrete struct. This allows
implementing functionality on top of the proxy.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. redis/rueidis#209 (thanos-io#6342)

* use github.com/onsi/gomega/gleak to detect goroutine leak with timeout

Signed-off-by: Rueian <rueiancsie@gmail.com>

* fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (thanos-io#6345)

Signed-off-by: aimuz <mr.imuz@gmail.com>

* Return grpc code resource exhausted for byte limit error (thanos-io#6325)

* return grpc code resource exhausted for byte limit error

Signed-off-by: Ben Ye <benye@amazon.com>

* fix lint

Signed-off-by: Ben Ye <benye@amazon.com>

* update partial response strategy

Signed-off-by: Ben Ye <benye@amazon.com>

* fix limit

Signed-off-by: Ben Ye <benye@amazon.com>

* try to fix tests

Signed-off-by: Ben Ye <benye@amazon.com>

* fix test error message

Signed-off-by: Ben Ye <benye@amazon.com>

* fix test

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Expose info for each TSDB

This commit exposes the label set alongside the min and max time
for each TSDB covered by a Store.

This information is used to scope the min time for a remote query
so that we do not produce partial aggregates in distriuted mode.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add test case for proxy store

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Bump promql-engine to fix thanos-io/promql-engine#239 (thanos-io#6349)

Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>

* Updates busybox SHA (thanos-io#6365)

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>

* Query: Add +Inf bucket to query duration metrics  (thanos-io#6358)

* Query: Add +Inf bucket to query duration metrics

For the query duration metrics
(`thanos_store_api_query_duration_seconds`), we record query respond
latency, based on the size of the query (samples/series), and save to a
histogram.

However, when a query is made which exceeds the biggest sample/serie
size, we would prior to this commit, put the request into the largest
bucket.

With this commit, we instead create an `+Inf` bucket, and put requests
which are larger than the biggest defined bucket into that. This gives
more accurate results, and also allow one to see if the bucket sizes are
incorrectly sized.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Tests: Mutex around non-thread safe random source

When creating test blocks, we use a non-thread safe random source, in
multiple goroutines. Due to this, tests would sometime panic.

This commits puts a mutex around calls using the same source, in order
to avoid this.

This should hopefully improve reliability of e2e tests.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

---------

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* e2e(query): Reproduce dedup issue from thanos-io#6257

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add dedup e2e test for Receive

With internal and external labels support.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify generated blocks for query test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve query dedup test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Write a query test for dedup with sidecar

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query dedup test with sidecar

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix Receive query test

Now it properly ensures the double dedup works (on internal and external labels).

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix receive drawing

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add one extra test caes for query dedup from store

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Complement test for Receive query with dedup

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Complement test for Sidecar query dedup

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Expected failure of block label query dedup tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rerun CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rerun CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Check context when expanding postings (thanos-io#6363)

* check context when expanding postings

Signed-off-by: Ben Ye <benye@amazon.com>

* update changelog

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* ui: only keep name in store_matches param (thanos-io#6371)

We are doing store matching on the `name` field hence only keep that
field in the URL because otherwise the URL could get quite lengthy with
external labelsets inside of it.

Besides unit tests, I have also tested locally:
- Enable store filtering;
- Select store(-s);
- Copy/paste URL into the new tab and see that the same stores are
  loaded like expected;
- See that URL only has names in them.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* docs: replace --store with --endpoint

Replace deprecated `--store` with `--endpoint` in docs.

Signed-off-by: Paul Gier <paul.gier@datastax.com>

* Optimizing "grafana generated" regex matchers (thanos-io#6376)

* Opmizing Group Regex

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* fixing native histogram tests

Signed-off-by: Alan Protasio <alanprot@gmail.com>

---------

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Cache: various index cache client improvements (thanos-io#6374)

* Query Explanation (thanos-io#6346)

* Return Query Explaination in QueryAPI

A param `explain` is added to QueryAPI, if true then explanation
returned by the `Explain()` method of the query having structure
`ExplainOutputNode` is returned in response.
Query Explanation is added under new field in response that is
`thanosInfo`.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Add explain checkbox in thanos UI

A explain checkbox is added to Thanos Query UI, that requests for
query explanation from thanos query api.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Add ExpandableNode Component

ExpandableNode component renders Query Explanation in the thanos
UI. Requires a new package `react-accessible-treeview`.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Disable Explain checkbox on prometheus engine

Prometheus engine sends out error if toggle explain button. To
provide better experience, the explain checkbox get disbaled on
switching to prometheus engine and enable back on switching to
thanos engine.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Add alert box with horizontal scrolling for Explanation

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Remove ExpandableNode and Add ListTree

Updates the design for query explanation box, removes
`ExpandableNode` and the dependency. Builts a new `ListTree` that
does the same using reactstrap and custom css.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Minor refactor in Query API response

`thanosInfo` is removed from Query reponse and used `explanation`
directly. `disableCheckbox` is also renamed to
`disableExplainCheckbox` in thanos UI.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Update UI tests to passing

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Minor UI changes and test fix

UI improvements and Panel test fix other way around, resetting
the results on panel construction.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Update promql-engine to use Explain method

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Build UI assets

Build UI assets, that runs new thanos UI with explain button.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Revert proxy url change from package.json

`proxy` was accidently changed and committed with package.json
when removed dependency. Hence, reverting it back.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Minor changes in UI

Fix requested changes in UI.
- Rename `state` and `setState` to `mapping` and `setMapping`.
- Rename `NodeTree` to `QueryTree`.
- Use unicode characters instead of `-` and `+`.
- Fix blue box on explain button.

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Update UI assets

Signed-off-by: Pradyumna Krishna <git@onpy.in>

---------

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (thanos-io#6379)

* Implementing Regex optimization on the MatchNotRegexp matcher type

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Opmizing MatchNotEqual

Signed-off-by: Alan Protasio <alanprot@gmail.com>

---------

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Put back the correct makefile

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra line that broke untouched test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add back line break at end of makefile

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix Receive single ingestor test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reproduce dedup issue in Receive

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add even more test cases for dedup on store gw

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reproduce dedup bug in Sidecar

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reuse nginx image name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Let all users read the metrics file from static metrics server

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rerun CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rerun CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat asciiflow chart

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reuse static metrics server from e2e framework

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* add de-cix as adopter (thanos-io#6386)

Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de>

* [chore] Updating Query Engine and Prometheus (thanos-io#6392)

* Updating Query Engine

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* fix prometheus breaking change

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Update prometheus with prometheus/prometheus#12387

Signed-off-by: Alan Protasio <alanprot@gmail.com>

---------

Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Receive: Allow specifying tenant-specific external labels in RouterIngestor (thanos-io#5777)

Signed-off-by: haanhvu <haanh6594@gmail.com>

* check context cancel when doing posting batches (thanos-io#6396)

Signed-off-by: Ben Ye <benye@amazon.com>

* Expose store gateway query stats in series response hints (thanos-io#6352)

* expose query stats hints

Signed-off-by: Ben Ye <benye@amazon.com>

* update

Signed-off-by: Ben Ye <benye@amazon.com>

* add query stats hints in result

Signed-off-by: Ben Ye <benye@amazon.com>

* update changelog

Signed-off-by: Ben Ye <benye@amazon.com>

* add merge method

Signed-off-by: Ben Ye <benye@amazon.com>

* fix unit test

Signed-off-by: Ben Ye <benye@amazon.com>

modify hints proto

Signed-off-by: Ben Ye <benye@amazon.com>

fix unit test

Signed-off-by: Ben Ye <benye@amazon.com>

update format

Signed-off-by: Ben Ye <benye@amazon.com>

* update comments

Signed-off-by: Ben Ye <benye@amazon.com>

* try again

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* receive: make az aware ketama hashring (thanos-io#6369)

* receive: make az aware ketama hashring

Signed-off-by: Alexander Rickardsson <alxric@aiven.io>

* receive: pass endpoints in hashring config as object

Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>

* receive: add some tests for consistent hashing in presence of AZs

Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>

* receive,docs: add migration note for az aware hashring

Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>

---------

Signed-off-by: Alexander Rickardsson <alxric@aiven.io>
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io>

* Proposal: query path tenancy  (thanos-io#6320)

* Add 1st version of query path tenancy proposal

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update proposal after initial feedback

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add cool picture

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Include example in cross tenant query complications

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve reasoning for why not using the QFE

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve writing in "How" section

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix owner profile link

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Address few more PR review comments

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Address feedback on flag name text

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update diagram

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve non-goals text

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update diagram

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update docs/proposals-accepted/202304-query-path-tenancy.md

Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clarify scenario for pitfalls of current solution

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clarify that Store doesn't care about tenant label

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add an action plan

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Mention alternative idea of modifying Store API

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix typo

Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Address lots of feedback on the proposal

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Format query path tenancy proposal doc

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a "Tenancy Model" subsection to "Goals"

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Mention header semanthics in comparison with gRPC message field

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve action plan structure and writing

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

---------

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* Fix double-counting bug in http_request_duration metric (thanos-io#6399)

* fix double-counting bug in http_request_duration metric

Signed-off-by: 4orty <kwk5178@gmail.com>

* Update Changelog

Signed-off-by: 4orty <kwk5178@gmail.com>

---------

Signed-off-by: 4orty <kwk5178@gmail.com>

* Updates busybox SHA (thanos-io#6403)

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>

* Fix series stats merge (thanos-io#6408)

* fix series stats merge

Signed-off-by: Ben Ye <benye@amazon.com>

* update license header

Signed-off-by: Ben Ye <benye@amazon.com>

* use reflect

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Receive: allow unlimited head_series_limit tenants (thanos-io#6406)

With this commit we now allow to configure tenants with unlimited active
series limit by setting the limit to `0`. Prior to this commit setting a
per tenant limit to `0` would cause the tenant to be unable to write any
metrics at all.

This fixes: thanos-io#6393

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* expose downloaded data size in query hints (thanos-io#6409)

Signed-off-by: Ben Ye <benye@amazon.com>

* maintainers: add myself to triagers (thanos-io#6414)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Add `@douglascamata` to triagers (thanos-io#6418)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add Blog (thanos-io#6411)

* Add LFX blog

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Add Headers to blog

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* Lint blog

Signed-off-by: Pradyumna Krishna <git@onpy.in>

---------

Signed-off-by: Pradyumna Krishna <git@onpy.in>

* blog: Fix images for LFX post (thanos-io#6422)

* blog: Fix images for LFX post

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* fix lint

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

---------

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Index Cache: Change cache key for postings (thanos-io#6405)

* extend postings cache key with codec

Signed-off-by: Ben Ye <benye@amazon.com>

* add changelog

Signed-off-by: Ben Ye <benye@amazon.com>

* update code back

Signed-off-by: Ben Ye <benye@amazon.com>

* add colon

Signed-off-by: Ben Ye <benye@amazon.com>

* update changelog

Signed-off-by: Ben Ye <benye@amazon.com>

* fix another test

Signed-off-by: Ben Ye <benye@amazon.com>

* add compression scheme const to remote index cache

Signed-off-by: Ben Ye <benye@amazon.com>

* address required comments

Signed-off-by: Ben Ye <benye@amazon.com>

* fix compression scheme name

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Receive: upgrading logs for failed uploads to error (thanos-io#6427)

* FIX: upgrading log for failed upload to error

Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>

* docs: added changelog entry

Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>

---------

Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>

* fix postings test

Signed-off-by: Ben Ye <benye@amazon.com>

* Add aiven as adopter... more soon! (thanos-io#6430)

Signed-off-by: Jonah Kowall <jkowall@kowall.net>

* Report gRPC connnection errors to the caller (thanos-io#6428)

By default `grpc.DialContext()` is non-blocking so any connection issue
will not be surfaced to the user. This change makes it blocking and
configures the gRPC dialer to report the underlying error if any
happens.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* chore: remove duplicated `gopkg.in/fsnotify.v1` dep (thanos-io#6432)

* chore: remove duplicated `gopkg.in/fsnotify.v1` dep

`github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same
dependency. We can keep `github.com/fsnotify/fsnotify` and remove
`gopkg.in/fsnotify.v1`.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* docs: add changelog

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

---------

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* Expose estimated chunk and series size as configurable options (thanos-io#6426)

* expose estimated chunk and series size as configurable options

Signed-off-by: Ben Ye <benye@amazon.com>

* fix lint

Signed-off-by: Ben Ye <benye@amazon.com>

* fix test

Signed-off-by: Ben Ye <benye@amazon.com>

* fix test

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Receive: make tsdb stats limit configurable (thanos-io#6437)

* Receive: make tsdb stats limit configurable

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Receive: make tsdb stats limit configurable

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

---------

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* *: wire new Engine/Explain fields in query-frontend (thanos-io#6433)

- Pass Engine/Explain fields in query-frontend codecs
- Add Engine field to QFE cache key
- Add e2e tests for all cases

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* index cache: Cache expanded postings (thanos-io#6420)

* cache expanded postings in index cache

Signed-off-by: Ben Ye <benye@amazon.com>

* update changelog

Signed-off-by: Ben Ye <benye@amazon.com>

* fix

Signed-off-by: Ben Ye <benye@amazon.com>

* fix lint

Signed-off-by: Ben Ye <benye@amazon.com>

* rebase main and added compression name to key

Signed-off-by: Ben Ye <benye@amazon.com>

* update key

Signed-off-by: Ben Ye <benye@amazon.com>

* add e2e test for memcached

Signed-off-by: Ben Ye <benye@amazon.com>

* fix cache config

Signed-off-by: Ben Ye <benye@amazon.com>

* address review comments

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* add approximate series size to index stats (thanos-io#6425)

Signed-off-by: Ben Ye <benye@amazon.com>

* index stats: fix chunk size calculation (thanos-io#6424)

Signed-off-by: Ben Ye <benye@amazon.com>

* Remove some unused Cortex vendored code and metrics (thanos-io#6440)

* Fixed DefaultPromConfig

* Fixed imports

* Back to diffVarintSnappyEncode

* Merge pull request thanos-io#180 from Shopify/optimize-timerange-calculation

Cache calculated mint and maxt for each remote engine

* Updated busybox

* fixing lint

* Fixing merge conflict

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Fixing missing import

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fix lint again

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* resolving conflict merges

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Fixed import and fn order

* Fixed unit tests

* Updated promdoc.sum

* Back to custom promql engine

* Removed custom promql engine and moved to latest upstream

* Ran go mod tidy

* Fixed GetQueryAPIClients

* Store: fix crash on empty regex matcher

Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>

---------

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: shayyxi <shazi12384@gmail.com>
Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com>
Signed-off-by: Rueian <rueiancsie@gmail.com>
Signed-off-by: aimuz <mr.imuz@gmail.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>
Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Paul Gier <paul.gier@datastax.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>
Signed-off-by: Pradyumna Krishna <git@onpy.in>
Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de>
Signed-off-by: haanhvu <haanh6594@gmail.com>
Signed-off-by: Alexander Rickardsson <alxric@aiven.io>
Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io>
Signed-off-by: 4orty <kwk5178@gmail.com>
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
Co-authored-by: Sebastian Rabenhorst <4246554+rabenhorst@users.noreply.github.com>
Co-authored-by: Xiaochao Dong <the.xcdong@gmail.com>
Co-authored-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Co-authored-by: Shazi <42436533+shayyxi@users.noreply.github.com>
Co-authored-by: shayyxi <shazi12384@gmail.com>
Co-authored-by: Rueian <rueiancsie@gmail.com>
Co-authored-by: aimuz <mr.imuz@gmail.com>
Co-authored-by: Ben Ye <benye@amazon.com>
Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: Alban Hurtaud <alban.hurtaud@amadeus.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>
Co-authored-by: Jacob Baungård Hansen <jacobbaungard@redhat.com>
Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Paul Gier <paul.gier@datastax.com>
Co-authored-by: Alan Protasio <alanprot@gmail.com>
Co-authored-by: Pradyumna Krishna <git@onpy.in>
Co-authored-by: Raúl Garcia Sanchez <info@raulgarcia.de>
Co-authored-by: Ha Anh Vu <75315486+haanhvu@users.noreply.github.com>
Co-authored-by: Alexander Rickardsson <alxric@aiven.io>
Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io>
Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com>
Co-authored-by: Wonki Kim <kwk5178@gmail.com>
Co-authored-by: Michael Hoffmann <mhoffm@posteo.de>
Co-authored-by: Victor Hugo Brito Fernandes <victorhbfernandes@gmail.com>
Co-authored-by: Jonah Kowall <jkowall@kowall.net>
Co-authored-by: Simon Pasquier <spasquie@redhat.com>
Co-authored-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
  • Loading branch information
1 parent a3c3d38 commit a93191e
Show file tree
Hide file tree
Showing 125 changed files with 6,441 additions and 1,269 deletions.
16 changes: 16 additions & 0 deletions .bingo/promdoc.sum
@@ -1,18 +1,34 @@
github.com/fsnotify/fsnotify v1.5.1 h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI=
github.com/fsnotify/fsnotify v1.5.1/go.mod h1:T3375wBYaZdLLcVNkcVbzGHY7f1l/uK5T5Ai1i3InKU=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/magiconair/properties v1.8.5 h1:b6kJs+EmPFMYGkow9GiUyCyOvIwYetYJ3fSaWak/Gls=
github.com/magiconair/properties v1.8.5/go.mod h1:y3VJvCyxH9uVvJTWEGAELF3aiYNyPKd5NZ3oSwXrF60=
github.com/mitchellh/mapstructure v1.4.2 h1:6h7AQ0yhTcIsmFmnAwQls75jp2Gzs4iB8W7pjMO+rqo=
github.com/mitchellh/mapstructure v1.4.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/pelletier/go-toml v1.9.4 h1:tjENF6MfZAg8e4ZmZTeWaWiT2vXtsoO6+iuOjFhECwM=
github.com/pelletier/go-toml v1.9.4/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
github.com/plexsystems/promdoc v0.8.0 h1:mNAp+WQkb2yZV5m7PeybHFTPYz+4pbaMCaH8iPLOMog=
github.com/plexsystems/promdoc v0.8.0/go.mod h1:CoTbHLEVPziXN+Y4GozwsiLvgdJqdOBYywqUy40sYuI=
github.com/spf13/afero v1.6.0 h1:xoax2sJ2DT8S8xA2paPFjDCScCNeWsg75VG0DLRreiY=
github.com/spf13/afero v1.6.0/go.mod h1:Ai8FlHk4v/PARR026UzYexafAt9roJ7LcLMAmO6Z93I=
github.com/spf13/cast v1.4.1 h1:s0hze+J0196ZfEMTs80N7UlFt0BDuQ7Q+JDnHiMWKdA=
github.com/spf13/cast v1.4.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cobra v1.2.1 h1:+KmjbUw1hriSNMF55oPrkZcb27aECyrj8V2ytv7kWDw=
github.com/spf13/cobra v1.2.1/go.mod h1:ExllRjgxM/piMAM+3tAZvg8fsklGAf3tPfi+i8t68Nk=
github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk=
github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.9.0 h1:yR6EXjTp0y0cLN8OZg1CRZmOBdI88UcGkhgyJhu6nZk=
github.com/spf13/viper v1.9.0/go.mod h1:+i6ajR7OX2XaiBkrcZJFK21htRk7eDeLg7+O6bhUPP4=
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
golang.org/x/sys v0.0.0-20210823070655-63515b42dcdf h1:2ucpDCmfkl8Bd/FsLtiD653Wf96cW37s+iGx93zsu4k=
golang.org/x/sys v0.0.0-20210823070655-63515b42dcdf/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
gopkg.in/ini.v1 v1.63.2 h1:tGK/CyBg7SMzb60vP1M03vNZ3VDu3wGQJwn7Sxi9r3c=
gopkg.in/ini.v1 v1.63.2/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
21 changes: 20 additions & 1 deletion CHANGELOG.md
Expand Up @@ -12,14 +12,20 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re

### Added

- [#6437](https://github.com/thanos-io/thanos/pull/6437) Receive: make tenant stats limit configurable
- [#6369](https://github.com/thanos-io/thanos/pull/6369) Receive: add az-aware replication support for Ketama algorithm
- [#6185](https://github.com/thanos-io/thanos/pull/6185) Tracing: tracing in OTLP support configuring service_name.
- [#6192](https://github.com/thanos-io/thanos/pull/6192) Store: add flag `bucket-web-label` to select the label to use as timeline title in web UI
- [#6167](https://github.com/thanos-io/thanos/pull/6195) Receive: add flag `tsdb.too-far-in-future.time-window` to prevent clock skewed samples to pollute TSDB head and block all valid incoming samples.
- [#6273](https://github.com/thanos-io/thanos/pull/6273) Mixin: Allow specifying an instance name filter in dashboards
- [#6163](https://github.com/thanos-io/thanos/pull/6163) Receiver: Add hidden flag `--receive-forward-max-backoff` to configure the max backoff for forwarding requests.
- [#5777](https://github.com/thanos-io/thanos/pull/5777) Receive: Allow specifying tenant-specific external labels in Router Ingestor.
- [#6352](https://github.com/thanos-io/thanos/pull/6352) Store: Expose store gateway query stats in series response hints.
- [#6420](https://github.com/thanos-io/thanos/pull/6420) Index Cache: Cache expanded postings.

### Fixed

- [#6456](https://github.com/thanos-io/thanos/pull/6456) Store: fix crash when computing set matches from regex pattern
- [#6427](https://github.com/thanos-io/thanos/pull/6427) Receive: increasing log level for failed uploads to error
- [#6172](https://github.com/thanos-io/thanos/pull/6172) query-frontend: return JSON formatted errors for invalid PromQL expression in the split by interval middleware.
- [#6171](https://github.com/thanos-io/thanos/pull/6171) Store: fix error handling on limits.
- [#6183](https://github.com/thanos-io/thanos/pull/6183) Receiver: fix off by one in multitsdb flush that will result in empty blocks if the head only contains one sample
Expand All @@ -29,6 +35,12 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#6222](https://github.com/thanos-io/thanos/pull/6222) mixin(Receive): Fix tenant series received charts.
- [#6218](https://github.com/thanos-io/thanos/pull/6218) mixin(Store): handle ResourceExhausted as a non-server error. As a consequence, this error won't contribute to Store's grpc errors alerts.
- [#6271](https://github.com/thanos-io/thanos/pull/6271) Receive: Fix segfault in `LabelValues` during head compaction.
- [#6306](https://github.com/thanos-io/thanos/pull/6306) Tracing: tracing in OTLP utilize the OTEL_TRACES_SAMPLER env variable
- [#6330](https://github.com/thanos-io/thanos/pull/6330) Store: Fix inconsistent error for series limits.
- [#6342](https://github.com/thanos-io/thanos/pull/6342) Cache/Redis: Upgrade `rueidis` to v1.0.2 to to improve error handling while shrinking a redis cluster.
- [#6325](https://github.com/thanos-io/thanos/pull/6325) Store: return gRPC resource exhausted error for byte limiter.
- [#6399](https://github.com/thanos-io/thanos/pull/6399) *: Fix double-counting bug in http_request_duration metric
- [#6428](https://github.com/thanos-io/thanos/pull/6428) Report gRPC connnection errors in the logs.

### Changed
- [#6168](https://github.com/thanos-io/thanos/pull/6168) Receiver: Make ketama hashring fail early when configured with number of nodes lower than the replication factor.
Expand All @@ -41,6 +53,12 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#6303](https://github.com/thanos-io/thanos/pull/6303) Store: added and start using streamed snappy encoding for postings list instead of block based one. This leads to constant memory usage during decompression. This approximately halves memory usage when decompressing a postings list in index cache.
- [#6071](https://github.com/thanos-io/thanos/pull/6071) Query Frontend: *breaking :warning:* Add experimental native histogram support for which we updated and aligned with the [Prometheus common](https://github.com/prometheus/common) model, which is used for caching so a cache reset required.
- [#6163](https://github.com/thanos-io/thanos/pull/6163) Receiver: changed max backoff from 30s to 5s for forwarding requests. Can be configured with `--receive-forward-max-backoff`.
- [#6327](https://github.com/thanos-io/thanos/pull/6327) *: *breaking :warning:* Use histograms instead of summaries for instrumented handlers.
- [#6322](https://github.com/thanos-io/thanos/pull/6322) Logging: Avoid expensive log.Valuer evaluation for disallowed levels.
- [#6358](https://github.com/thanos-io/thanos/pull/6358) Query: Add +Inf bucket to query duration metrics
- [#6363](https://github.com/thanos-io/thanos/pull/6363) Store: Check context error when expanding postings.
- [#6405](https://github.com/thanos-io/thanos/pull/6405) Index Cache: Change postings cache key to include the encoding format used so that older Thanos versions would not try to decode it during the deployment of a new version.
- [#6432](https://github.com/thanos-io/thanos/pull/6432) Receive: Remove duplicated `gopkg.in/fsnotify.v1` dependency

### Removed

Expand All @@ -59,6 +77,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#5548](https://github.com/thanos-io/thanos/pull/5548) Query: Add experimental support for load balancing across multiple Store endpoints.
- [#6148](https://github.com/thanos-io/thanos/pull/6148) Query-frontend: Add `traceID` to slow query detected log line.
- [#6153](https://github.com/thanos-io/thanos/pull/6153) Query-frontend: Add `remote_user` (from http basic auth) and `remote_addr` to slow query detected log line.
- [#6406](https://github.com/thanos-io/thanos/pull/6406) Receive: Allow tenants to be configured with unlimited active series by setting head_series_limit to 0.

### Fixed

Expand Down
18 changes: 10 additions & 8 deletions MAINTAINERS.md
Expand Up @@ -31,14 +31,16 @@ We also have some nice souls that help triaging issues and PRs. See [here](https

Full list of triage persons is displayed below:

| Name | Slack | GitHub | Company |
|---------------|--------------|--------------------------------------------|---------|
| Adrien Fillon | `@Adrien F` | [@adrien-f](https://github.com/adrien-f) | |
| Ian Billett | `@billett` | [@bill3tt](https://github.com/bill3tt) | Red Hat |
| Martin Chodur | `@FUSAKLA` | [@fusakla](https://github.com/fusakla) | |
| Michael Dai | `@jojohappy` | [@jojohappy](https://github.com/jojohappy) | |
| Xiang Dai | `@daixiang0` | [@daixiang0](https://github.com/daixiang0) | |
| Jimmie Han | `@hanjm` | [@hanjm](https://github.com/hanjm) | Tencent |
| Name | Slack | GitHub | Company |
|------------------|---------------------|----------------------------------------------------|---------|
| Adrien Fillon | `@Adrien F` | [@adrien-f](https://github.com/adrien-f) | |
| Ian Billett | `@billett` | [@bill3tt](https://github.com/bill3tt) | Red Hat |
| Martin Chodur | `@FUSAKLA` | [@fusakla](https://github.com/fusakla) | |
| Michael Dai | `@jojohappy` | [@jojohappy](https://github.com/jojohappy) | |
| Xiang Dai | `@daixiang0` | [@daixiang0](https://github.com/daixiang0) | |
| Jimmie Han | `@hanjm` | [@hanjm](https://github.com/hanjm) | Tencent |
| Michael Hoffmann | `@Michael Hoffmann` | [@MichaHoffmann](https://github.com/MichaHoffmann) | Aiven |
| Douglas Camata | `@douglascamata` | [@douglascamata](https://github.com/douglascamata) | Red Hat |

Please reach any of the maintainer on slack or email if you want to help as well.

Expand Down
1 change: 1 addition & 0 deletions Makefile
Expand Up @@ -395,6 +395,7 @@ github.com/prometheus/client_golang/prometheus.{DefaultGatherer,DefBuckets,NewUn
github.com/prometheus/client_golang/prometheus.{NewCounter,NewCounterVec,NewCounterVec,NewGauge,NewGaugeVec,NewGaugeFunc,\
NewHistorgram,NewHistogramVec,NewSummary,NewSummaryVec}=github.com/prometheus/client_golang/prometheus/promauto.{NewCounter,\
NewCounterVec,NewCounterVec,NewGauge,NewGaugeVec,NewGaugeFunc,NewHistorgram,NewHistogramVec,NewSummary,NewSummaryVec},\
github.com/NYTimes/gziphandler.{GzipHandler}=github.com/klauspost/compress/gzhttp.{GzipHandler},\
sync/atomic=go.uber.org/atomic,github.com/cortexproject/cortex=github.com/thanos-io/thanos/internal/cortex,\
io/ioutil.{Discard,NopCloser,ReadAll,ReadDir,ReadFile,TempDir,TempFile,Writefile}" $(shell go list ./... | grep -v -e "internal/cortex" -e "internal/mimir-prometheus")
@$(FAILLINT) -paths "fmt.{Print,Println,Sprint}" -ignore-tests ./...
Expand Down
9 changes: 5 additions & 4 deletions cmd/thanos/query.go
Expand Up @@ -217,8 +217,8 @@ func registerQuery(app *extkingpin.App) {
grpcProxyStrategy := cmd.Flag("grpc.proxy-strategy", "Strategy to use when proxying Series requests to leaf nodes. Hidden and only used for testing, will be removed after lazy becomes the default.").Default(string(store.EagerRetrieval)).Hidden().Enum(string(store.EagerRetrieval), string(store.LazyRetrieval))

queryTelemetryDurationQuantiles := cmd.Flag("query.telemetry.request-duration-seconds-quantiles", "The quantiles for exporting metrics about the request duration quantiles.").Default("0.1", "0.25", "0.75", "1.25", "1.75", "2.5", "3", "5", "10").Float64List()
queryTelemetrySamplesQuantiles := cmd.Flag("query.telemetry.request-samples-quantiles", "The quantiles for exporting metrics about the samples count quantiles.").Default("100", "1000", "10000", "100000", "1000000").Int64List()
queryTelemetrySeriesQuantiles := cmd.Flag("query.telemetry.request-series-seconds-quantiles", "The quantiles for exporting metrics about the series count quantiles.").Default("10", "100", "1000", "10000", "100000").Int64List()
queryTelemetrySamplesQuantiles := cmd.Flag("query.telemetry.request-samples-quantiles", "The quantiles for exporting metrics about the samples count quantiles.").Default("100", "1000", "10000", "100000", "1000000").Float64List()
queryTelemetrySeriesQuantiles := cmd.Flag("query.telemetry.request-series-seconds-quantiles", "The quantiles for exporting metrics about the series count quantiles.").Default("10", "100", "1000", "10000", "100000").Float64List()

storeSelectorRelabelConf := *extkingpin.RegisterSelectorRelabelFlags(cmd)

Expand Down Expand Up @@ -413,8 +413,8 @@ func runQuery(
grpcProxyStrategy string,
comp component.Component,
queryTelemetryDurationQuantiles []float64,
queryTelemetrySamplesQuantiles []int64,
queryTelemetrySeriesQuantiles []int64,
queryTelemetrySamplesQuantiles []float64,
queryTelemetrySeriesQuantiles []float64,
defaultEngine string,
storeRateLimits store.SeriesSelectLimits,
storeSelectorRelabelConf extflag.PathOrContent,
Expand Down Expand Up @@ -804,6 +804,7 @@ func runQuery(
GuaranteedMinTime: proxy.GuaranteedMinTime(),
SupportsSharding: true,
SupportsWithoutReplicaLabels: true,
TsdbInfos: proxy.TSDBInfos(),
}
}
return nil
Expand Down
6 changes: 3 additions & 3 deletions cmd/thanos/query_frontend.go
Expand Up @@ -8,10 +8,10 @@ import (
"net/http"
"time"

"github.com/NYTimes/gziphandler"
extflag "github.com/efficientgo/tools/extkingpin"
"github.com/go-kit/log"
"github.com/go-kit/log/level"
"github.com/klauspost/compress/gzhttp"
"github.com/oklog/run"
"github.com/opentracing/opentracing-go"
"github.com/pkg/errors"
Expand Down Expand Up @@ -277,7 +277,7 @@ func runQueryFrontend(
// Create the query frontend transport.
handler := transport.NewHandler(*cfg.CortexHandlerConfig, roundTripper, logger, nil)
if cfg.CompressResponses {
handler = gziphandler.GzipHandler(handler)
handler = gzhttp.GzipHandler(handler)
}

httpProbe := prober.NewHTTP()
Expand Down Expand Up @@ -311,7 +311,7 @@ func runQueryFrontend(
logger,
ins.NewHandler(
name,
gziphandler.GzipHandler(
gzhttp.GzipHandler(
middleware.RequestID(
logMiddleware.HTTPMiddleware(name, f),
),
Expand Down
47 changes: 30 additions & 17 deletions cmd/thanos/receive.go
Expand Up @@ -281,7 +281,7 @@ func runReceive(

level.Debug(logger).Log("msg", "setting up hashring")
{
if err := setupHashring(g, logger, reg, conf, hashringChangedChan, webHandler, statusProber, enableIngestion); err != nil {
if err := setupHashring(g, logger, reg, conf, hashringChangedChan, webHandler, statusProber, enableIngestion, dbs); err != nil {
return err
}
}
Expand Down Expand Up @@ -346,6 +346,7 @@ func runReceive(
GuaranteedMinTime: proxy.GuaranteedMinTime(),
SupportsSharding: true,
SupportsWithoutReplicaLabels: true,
TsdbInfos: proxy.TSDBInfos(),
}
}
return nil
Expand Down Expand Up @@ -453,12 +454,13 @@ func setupHashring(g *run.Group,
webHandler *receive.Handler,
statusProber prober.Probe,
enableIngestion bool,
dbs *receive.MultiTSDB,
) error {
// Note: the hashring configuration watcher
// is the sender and thus closes the chan.
// In the single-node case, which has no configuration
// watcher, we close the chan ourselves.
updates := make(chan receive.Hashring, 1)
updates := make(chan []receive.HashringConfig, 1)
algorithm := receive.HashringAlgorithm(conf.hashringsAlgorithm)

// The Hashrings config file path is given initializing config watcher.
Expand All @@ -477,33 +479,28 @@ func setupHashring(g *run.Group,

ctx, cancel := context.WithCancel(context.Background())
g.Add(func() error {
level.Info(logger).Log("msg", "the hashring initialized with config watcher.")
return receive.HashringFromConfigWatcher(ctx, algorithm, conf.replicationFactor, updates, cw)
return receive.ConfigFromWatcher(ctx, updates, cw)
}, func(error) {
cancel()
})
} else {
var (
ring receive.Hashring
err error
cf []receive.HashringConfig
err error
)
// The Hashrings config file content given initialize configuration from content.
if len(conf.hashringsFileContent) > 0 {
ring, err = receive.HashringFromConfig(algorithm, conf.replicationFactor, conf.hashringsFileContent)
cf, err = receive.ParseConfig([]byte(conf.hashringsFileContent))
if err != nil {
close(updates)
return errors.Wrap(err, "failed to validate hashring configuration file")
return errors.Wrap(err, "failed to validate hashring configuration content")
}
level.Info(logger).Log("msg", "the hashring initialized directly with the given content through the flag.")
} else {
level.Info(logger).Log("msg", "the hashring file is not specified use single node hashring.")
ring = receive.SingleNodeHashring(conf.endpoint)
}

cancel := make(chan struct{})
g.Add(func() error {
defer close(updates)
updates <- ring
updates <- cf
<-cancel
return nil
}, func(error) {
Expand All @@ -520,11 +517,27 @@ func setupHashring(g *run.Group,

for {
select {
case h, ok := <-updates:
case c, ok := <-updates:
if !ok {
return nil
}
webHandler.Hashring(h)

if c == nil {
webHandler.Hashring(receive.SingleNodeHashring(conf.endpoint))
level.Info(logger).Log("msg", "Empty hashring config. Set up single node hashring.")
} else {
h, err := receive.NewMultiHashring(algorithm, conf.replicationFactor, c)
if err != nil {
return errors.Wrap(err, "unable to create new hashring from config")
}
webHandler.Hashring(h)
level.Info(logger).Log("msg", "Set up hashring for the given hashring config.")
}

if err := dbs.SetHashringConfig(c); err != nil {
return errors.Wrap(err, "failed to set hashring config in MultiTSDB")
}

// If ingestion is enabled, send a signal to TSDB to flush.
if enableIngestion {
hashringChangedChan <- struct{}{}
Expand Down Expand Up @@ -697,12 +710,12 @@ func startTSDBAndUpload(g *run.Group,
case <-uploadC:
// Upload on demand.
if err := upload(ctx); err != nil {
level.Warn(logger).Log("msg", "on demand upload failed", "err", err)
level.Error(logger).Log("msg", "on demand upload failed", "err", err)
}
uploadDone <- struct{}{}
case <-tick.C:
if err := upload(ctx); err != nil {
level.Warn(logger).Log("msg", "recurring upload failed", "err", err)
level.Error(logger).Log("msg", "recurring upload failed", "err", err)
}
}
}
Expand Down
1 change: 1 addition & 0 deletions cmd/thanos/rule.go
Expand Up @@ -676,6 +676,7 @@ func runRule(
GuaranteedMinTime: guaranteedMinTime,
SupportsSharding: true,
SupportsWithoutReplicaLabels: true,
TsdbInfos: tsdbStore.TSDBInfos(),
}
}
return nil
Expand Down
1 change: 1 addition & 0 deletions cmd/thanos/sidecar.go
Expand Up @@ -279,6 +279,7 @@ func runSidecar(
GuaranteedMinTime: guaranteedMinTime,
SupportsSharding: true,
SupportsWithoutReplicaLabels: true,
TsdbInfos: promStore.TSDBInfos(),
}
}
return nil
Expand Down

0 comments on commit a93191e

Please sign in to comment.