From 139ae3f224e0ca389f16d35485df464cee337c18 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Tue, 2 Mar 2021 10:16:31 +0100 Subject: [PATCH 01/15] Cut v0.19.0-rc.0 (#3860) Signed-off-by: Bartlomiej Plotka --- CHANGELOG.md | 28 +++++++++++-------- VERSION | 2 +- .../thanos/1-globalview/courseBase.sh | 2 +- .../katacoda/thanos/1-globalview/step2.md | 8 +++--- .../katacoda/thanos/1-globalview/step3.md | 2 +- tutorials/katacoda/thanos/2-lts/courseBase.sh | 2 +- tutorials/katacoda/thanos/2-lts/step1.md | 4 +-- tutorials/katacoda/thanos/2-lts/step2.md | 2 +- tutorials/katacoda/thanos/2-lts/step3.md | 6 ++-- tutorials/katacoda/thanos/2-lts/step4.md | 2 +- .../thanos/6-query-caching/courseBase.sh | 2 +- .../katacoda/thanos/6-query-caching/step1.md | 4 +-- .../katacoda/thanos/6-query-caching/step2.md | 2 +- .../thanos/7-multi-tenancy/courseBase.sh | 2 +- .../katacoda/thanos/7-multi-tenancy/step1.md | 10 +++---- .../katacoda/thanos/7-multi-tenancy/step2.md | 2 +- .../thanos/x-playground/courseBase.sh | 2 +- .../katacoda/thanos/x-playground/step1.md | 10 +++---- .../katacoda/thanos/x-playground/step2.md | 12 ++++---- 19 files changed, 55 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 35620b7218..85669ed24a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,27 +10,35 @@ NOTE: As semantic versioning states all 0.y.z releases can contain breaking chan We use _breaking :warning:_ to mark changes that are not backward compatible (relates only to v0.y.z releases.) -## Unreleased +## [v0.19.0-rc.0](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.0) - 2021.03.02 ### Added -- [#3740](https://github.com/thanos-io/thanos/pull/3740) Query: Added `--query.default-step` flag to set default step. -- [#3700](https://github.com/thanos-io/thanos/pull/3700) ui: make old bucket viewer UI work with vanilla Prometheus blocks +- [#3700](https://github.com/thanos-io/thanos/pull/3700) Compact/Web: Make old bucket viewer UI work with vanilla Prometheus blocks. +- [#3657](https://github.com/thanos-io/thanos/pull/3657) *: It's now possible to configure HTTP transport options for S3 client. +- [#3752](https://github.com/thanos-io/thanos/pull/3752) Compact/Store: Added `--block-meta-fetch-concurrency` allowing to configure number of go routines for block metadata synchronization. - [#2641](https://github.com/thanos-io/thanos/issues/2641) Query Frontend: Added `--query-range.request-downsampled` flag enabling additional queries for downsampled data in case of empty or incomplete response to range request. -- [#3792](https://github.com/thanos-io/thanos/pull/3792) Receiver: Added `--tsdb.allow-overlapping-blocks` flag to allow overlapping tsdb blocks and enable vertical compaction -- [#3031](https://github.com/thanos-io/thanos/pull/3031) Compact/Sidecar/other writers: added `--hash-func`. If some function has been specified, writers calculate hashes using that function of each file in a block before uploading them. If those hashes exist in the `meta.json` file then Compact does not download the files if they already exist on disk and with the same hash. This also means that the data directory passed to Thanos Compact is only *cleared once at boot* or *if everything succeeds*. So, if you, for example, use persistent volumes on k8s and your Thanos Compact crashes or fails to make an iteration properly then the last downloaded files are not wiped from the disk. The directories that were created the last time are only wiped again after a successful iteration or if the previously picked up blocks have disappeared. -- [#3686](https://github.com/thanos-io/thanos/pull/3686) Query: Added federated metric metadata support. +???? -[#3579](https://github.com/thanos-io/thanos/pull/3579) Cache: Added inmemory cache for caching bucket. +- [#3792](https://github.com/thanos-io/thanos/pull/3792) Receiver: Added `--tsdb.allow-overlapping-blocks` flag to allow overlapping tsdb blocks and enable vertical compaction. +- [#3740](https://github.com/thanos-io/thanos/pull/3740) Query: Added `--query.default-step` flag to set default step. Useful when your tenant scrape interval is stable and far from default UI's 1s. +- [#3686](https://github.com/thanos-io/thanos/pull/3686) Query,Sidecar: Added metric metadata API support. You can now configure you Querier to fetch Prometheus metrics metadata from leaf Prometheus-es! +- [#3031](https://github.com/thanos-io/thanos/pull/3031) Compact/Sidecar/Receive/Rule: Added `--hash-func`. If some function has been specified, writers calculate hashes using that function of each file in a block before uploading them. If those hashes exist in the `meta.json` file then Compact does not download the files if they already exist on disk and with the same hash. This also means that the data directory passed to Thanos Compact is only *cleared once at boot* or *if everything succeeds*. So, if you, for example, use persistent volumes on k8s and your Thanos Compact crashes or fails to make an iteration properly then the last downloaded files are not wiped from the disk. The directories that were created the last time are only wiped again after a successful iteration or if the previously picked up blocks have disappeared. + ### Fixed -- [#3773](https://github.com/thanos-io/thanos/pull/3773) Compact: Pad compaction planner size check -- [#3795](https://github.com/thanos-io/thanos/pull/3795) s3: A truncated "get object" response is reported as error. +- [#3705](https://github.com/thanos-io/thanos/pull/3705) Store: Fix race condition leading to failing queries or possibly incorrect query results. +- [#3661](https://github.com/thanos-io/thanos/pull/3661) Compact: Deletion-mark.json is deleted as the last one, which could in theory lead to potential store gateway load or query error for such in-deletion block. +- [#3760](https://github.com/thanos-io/thanos/pull/3760) Store: Fix panic caused by a race condition happening on concurrent index-header reader usage and unload, when `--store.enable-index-header-lazy-reader` is enabled. +- [#3759](https://github.com/thanos-io/thanos/pull/3759) Store: Fix panic caused by a race condition happening on concurrent index-header lazy load and unload, when `--store.enable-index-header-lazy-reader` is enabled. +- [#3773](https://github.com/thanos-io/thanos/pull/3773) Compact: Fixed compaction planner size check, making sure we don't create too large blocks. - [#3814](https://github.com/thanos-io/thanos/pull/3814) Store: Decreased memory utilisation while fetching block's chunks. - [#3815](https://github.com/thanos-io/thanos/pull/3815) Receive: Improve handling of empty time series from clients +- [#3795](https://github.com/thanos-io/thanos/pull/3795) s3: A truncated "get object" response is reported as error. ### Changed -- [#3705](https://github.com/thanos-io/thanos/pull/3705) Store: Fix race condition leading to failing queries or possibly incorrect query results. +- [#3804](https://github.com/thanos-io/thanos/pull/3804) Ruler, Receive, Querier: Updated Prometheus dependency. TSDB characteristics might have changed. ## [v0.18.0](https://github.com/thanos-io/thanos/releases/tag/v0.18.0) - 2021.01.27 @@ -52,8 +60,6 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re - [#3567](https://github.com/thanos-io/thanos/pull/3567) Mixin: Reintroduce `thanos_objstore_bucket_operation_failures_total` alert. - [#3527](https://github.com/thanos-io/thanos/pull/3527) Query Frontend: Fix query_range behavior when start/end times are the same -- [#3760](https://github.com/thanos-io/thanos/pull/3760) Store: Fix panic caused by a race condition happening on concurrent index-header reader usage and unload, when `--store.enable-index-header-lazy-reader` is enabled. -- [#3759](https://github.com/thanos-io/thanos/pull/3759) Store: Fix panic caused by a race condition happening on concurrent index-header lazy load and unload, when `--store.enable-index-header-lazy-reader` is enabled. - [#3560](https://github.com/thanos-io/thanos/pull/3560) Query Frontend: Allow separate label cache - [#3672](https://github.com/thanos-io/thanos/pull/3672) Rule: Prevent crashing due to `no such host error` when using `dnssrv+` or `dnssrvnoa+`. - [#3461](https://github.com/thanos-io/thanos/pull/3461) Compact, Shipper, Store: Fixed panic when no external labels are set in block metadata. diff --git a/VERSION b/VERSION index 66333910a4..d92951ecef 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.18.0 +0.19.0-rc.0 \ No newline at end of file diff --git a/tutorials/katacoda/thanos/1-globalview/courseBase.sh b/tutorials/katacoda/thanos/1-globalview/courseBase.sh index 4d53f3e701..6e35780f49 100644 --- a/tutorials/katacoda/thanos/1-globalview/courseBase.sh +++ b/tutorials/katacoda/thanos/1-globalview/courseBase.sh @@ -1,4 +1,4 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.16.0 -docker pull quay.io/thanos/thanos:v0.18.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.0 diff --git a/tutorials/katacoda/thanos/1-globalview/step2.md b/tutorials/katacoda/thanos/1-globalview/step2.md index b7c0ae07fc..703c9600cb 100644 --- a/tutorials/katacoda/thanos/1-globalview/step2.md +++ b/tutorials/katacoda/thanos/1-globalview/step2.md @@ -10,7 +10,7 @@ component and can be invoked in a single command. Let's take a look at all the Thanos commands: ``` -docker run --rm quay.io/thanos/thanos:v0.18.0 --help +docker run --rm quay.io/thanos/thanos:v0.19.0-rc.0 --help ```{{execute}} You should see multiple commands that solves different purposes. @@ -53,7 +53,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-eu1 \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -68,7 +68,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -81,7 +81,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus1_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-1-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ diff --git a/tutorials/katacoda/thanos/1-globalview/step3.md b/tutorials/katacoda/thanos/1-globalview/step3.md index 40be4f6264..35ae6cfc72 100644 --- a/tutorials/katacoda/thanos/1-globalview/step3.md +++ b/tutorials/katacoda/thanos/1-globalview/step3.md @@ -28,7 +28,7 @@ Click below snippet to start the Querier. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:29090 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/courseBase.sh b/tutorials/katacoda/thanos/2-lts/courseBase.sh index e3dd261757..e9865553bc 100644 --- a/tutorials/katacoda/thanos/2-lts/courseBase.sh +++ b/tutorials/katacoda/thanos/2-lts/courseBase.sh @@ -2,7 +2,7 @@ docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.18.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.0 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 mkdir /root/editor diff --git a/tutorials/katacoda/thanos/2-lts/step1.md b/tutorials/katacoda/thanos/2-lts/step1.md index 46a7045679..2aee230e8a 100644 --- a/tutorials/katacoda/thanos/2-lts/step1.md +++ b/tutorials/katacoda/thanos/2-lts/step1.md @@ -117,7 +117,7 @@ Similar to previous course, let's setup global view querying with sidecar: docker run -d --net=host --rm \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -130,7 +130,7 @@ so we will make sure we point the Querier to the gRPC endpoints of the sidecar: ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step2.md b/tutorials/katacoda/thanos/2-lts/step2.md index 3cc24c6df1..3c269bf97d 100644 --- a/tutorials/katacoda/thanos/2-lts/step2.md +++ b/tutorials/katacoda/thanos/2-lts/step2.md @@ -79,7 +79,7 @@ docker run -d --net=host --rm \ -v /root/prom-eu1:/prometheus \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ diff --git a/tutorials/katacoda/thanos/2-lts/step3.md b/tutorials/katacoda/thanos/2-lts/step3.md index d96fdf4e94..e008665237 100644 --- a/tutorials/katacoda/thanos/2-lts/step3.md +++ b/tutorials/katacoda/thanos/2-lts/step3.md @@ -6,7 +6,7 @@ In this step, we will learn about Thanos Store Gateway and how to deploy it. Let's take a look at all the Thanos commands: -```docker run --rm quay.io/thanos/thanos:v0.18.0 --help```{{execute}} +```docker run --rm quay.io/thanos/thanos:v0.19.0-rc.0 --help```{{execute}} You should see multiple commands that solve different purposes, block storage based long-term storage for Prometheus. @@ -32,7 +32,7 @@ You can read more about [Store](https://thanos.io/tip/components/store.md/) here docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19091 \ @@ -49,7 +49,7 @@ Currently querier does not know about store yet. Let's change it by adding Store docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step4.md b/tutorials/katacoda/thanos/2-lts/step4.md index 001773acb7..56a2dd2bda 100644 --- a/tutorials/katacoda/thanos/2-lts/step4.md +++ b/tutorials/katacoda/thanos/2-lts/step4.md @@ -25,7 +25,7 @@ Click below snippet to start the Compactor. docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name thanos-compact \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ diff --git a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh index 804e416ebe..707a6abd4d 100644 --- a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh +++ b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.22.2 -docker pull quay.io/thanos/thanos:v0.18.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.0 docker pull yannrobert/docker-nginx diff --git a/tutorials/katacoda/thanos/6-query-caching/step1.md b/tutorials/katacoda/thanos/6-query-caching/step1.md index 824e3d1f97..9ebc4431a7 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step1.md +++ b/tutorials/katacoda/thanos/6-query-caching/step1.md @@ -103,7 +103,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \ --name prometheus-sidecar"${i}" \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address=0.0.0.0:1909"${i}" \ --grpc-address=0.0.0.0:1919"${i}" \ @@ -129,7 +129,7 @@ And now, let's deploy Thanos Querier to have a global overview on our services. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:10912 \ --grpc-address 0.0.0.0:10901 \ diff --git a/tutorials/katacoda/thanos/6-query-caching/step2.md b/tutorials/katacoda/thanos/6-query-caching/step2.md index d897435efc..db9309e1c7 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step2.md +++ b/tutorials/katacoda/thanos/6-query-caching/step2.md @@ -62,7 +62,7 @@ And deploy Query Frontend: docker run -d --net=host --rm \ -v $(pwd)/frontend.yml:/etc/thanos/frontend.yml \ --name query-frontend \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query-frontend \ --http-address 0.0.0.0:20902 \ --query-frontend.compress-responses \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh index 9d96284940..35b9ecd2af 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh +++ b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.18.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.0 docker pull quay.io/thanos/prom-label-proxy:v0.3.0-rc.0-ext1 docker pull caddy:2.2.1 diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md index b34c112756..fc34680033 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md @@ -88,7 +88,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_fruit.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-fruit \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -120,7 +120,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -152,7 +152,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus1_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-01-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -170,7 +170,7 @@ Fruit: ``` docker run -d --net=host --rm \ --name querier-fruit \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:29091 \ --grpc-address 0.0.0.0:29191 \ @@ -183,7 +183,7 @@ Veggie: ``` docker run -d --net=host --rm \ --name querier-veggie \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:29092 \ --grpc-address 0.0.0.0:29192 \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md index 5a63ed2dae..ef07b7ea4d 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md @@ -11,7 +11,7 @@ docker stop querier-fruit && docker stop querier-veggie ``` docker run -d --net=host --rm \ --name querier-multi \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:29090 \ --grpc-address 0.0.0.0:29190 \ diff --git a/tutorials/katacoda/thanos/x-playground/courseBase.sh b/tutorials/katacoda/thanos/x-playground/courseBase.sh index 9a5e40ba35..48f1878903 100644 --- a/tutorials/katacoda/thanos/x-playground/courseBase.sh +++ b/tutorials/katacoda/thanos/x-playground/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.18.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.0 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z diff --git a/tutorials/katacoda/thanos/x-playground/step1.md b/tutorials/katacoda/thanos/x-playground/step1.md index 402ebb756a..c8f128d556 100644 --- a/tutorials/katacoda/thanos/x-playground/step1.md +++ b/tutorials/katacoda/thanos/x-playground/step1.md @@ -169,7 +169,7 @@ docker run -d --net=host --rm \ ### Step1: Sidecar ``` -docker run -it --rm quay.io/thanos/thanos:v0.18.0 --help +docker run -it --rm quay.io/thanos/thanos:v0.19.0-rc.0 --help ```{{execute}} @@ -180,7 +180,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -195,7 +195,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -210,7 +210,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --http-address 0.0.0.0:19093 \ --grpc-address 0.0.0.0:19193 \ @@ -223,7 +223,7 @@ docker run -d --net=host --rm \ ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ diff --git a/tutorials/katacoda/thanos/x-playground/step2.md b/tutorials/katacoda/thanos/x-playground/step2.md index efaa56c3fc..453762d058 100644 --- a/tutorials/katacoda/thanos/x-playground/step2.md +++ b/tutorials/katacoda/thanos/x-playground/step2.md @@ -65,7 +65,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0:/prometheus \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -85,7 +85,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1:/prometheus \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -105,7 +105,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0:/prometheus \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -130,7 +130,7 @@ Let's run Store Gateway server: docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19094 \ @@ -143,7 +143,7 @@ docker run -d --net=host --rm \ docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ @@ -162,7 +162,7 @@ Visit https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.co docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name compactor \ - quay.io/thanos/thanos:v0.18.0 \ + quay.io/thanos/thanos:v0.19.0-rc.0 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ From 93f26050e3c248a169623b142f7b6075aa99fb4f Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Tue, 2 Mar 2021 12:17:05 +0100 Subject: [PATCH 02/15] Fixed missed changelog comments. (#3865) Missed from: https://github.com/thanos-io/thanos/pull/3860 Signed-off-by: Bartlomiej Plotka --- CHANGELOG.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85669ed24a..6cdfaab601 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,11 +17,11 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re - [#3700](https://github.com/thanos-io/thanos/pull/3700) Compact/Web: Make old bucket viewer UI work with vanilla Prometheus blocks. - [#3657](https://github.com/thanos-io/thanos/pull/3657) *: It's now possible to configure HTTP transport options for S3 client. - [#3752](https://github.com/thanos-io/thanos/pull/3752) Compact/Store: Added `--block-meta-fetch-concurrency` allowing to configure number of go routines for block metadata synchronization. -- [#2641](https://github.com/thanos-io/thanos/issues/2641) Query Frontend: Added `--query-range.request-downsampled` flag enabling additional queries for downsampled data in case of empty or incomplete response to range request. -???? -[#3579](https://github.com/thanos-io/thanos/pull/3579) Cache: Added inmemory cache for caching bucket. +- [#3723](https://github.com/thanos-io/thanos/pull/3723) Query Frontend: Added `--query-range.request-downsampled` flag enabling additional queries for downsampled data in case of empty or incomplete response to range request. +- [#3579](https://github.com/thanos-io/thanos/pull/3579) Cache: Added inmemory cache for caching bucket. - [#3792](https://github.com/thanos-io/thanos/pull/3792) Receiver: Added `--tsdb.allow-overlapping-blocks` flag to allow overlapping tsdb blocks and enable vertical compaction. - [#3740](https://github.com/thanos-io/thanos/pull/3740) Query: Added `--query.default-step` flag to set default step. Useful when your tenant scrape interval is stable and far from default UI's 1s. -- [#3686](https://github.com/thanos-io/thanos/pull/3686) Query,Sidecar: Added metric metadata API support. You can now configure you Querier to fetch Prometheus metrics metadata from leaf Prometheus-es! +- [#3686](https://github.com/thanos-io/thanos/pull/3686) Query/Sidecar: Added metric metadata API support. You can now configure you Querier to fetch Prometheus metrics metadata from leaf Prometheus-es! - [#3031](https://github.com/thanos-io/thanos/pull/3031) Compact/Sidecar/Receive/Rule: Added `--hash-func`. If some function has been specified, writers calculate hashes using that function of each file in a block before uploading them. If those hashes exist in the `meta.json` file then Compact does not download the files if they already exist on disk and with the same hash. This also means that the data directory passed to Thanos Compact is only *cleared once at boot* or *if everything succeeds*. So, if you, for example, use persistent volumes on k8s and your Thanos Compact crashes or fails to make an iteration properly then the last downloaded files are not wiped from the disk. The directories that were created the last time are only wiped again after a successful iteration or if the previously picked up blocks have disappeared. From af4ee3a09f6b00d280f5b6913b18864a595dc96f Mon Sep 17 00:00:00 2001 From: Ben Ye Date: Fri, 5 Mar 2021 14:11:03 -0500 Subject: [PATCH 03/15] rename Metadata API to MetricMetadata API (#3877) Signed-off-by: yeya24 --- pkg/api/query/v1.go | 4 +- pkg/metadata/metadata.go | 14 +- pkg/metadata/metadatapb/custom.go | 12 +- pkg/metadata/metadatapb/rpc.pb.go | 241 +++++++++++++++--------------- pkg/metadata/metadatapb/rpc.proto | 6 +- pkg/metadata/prometheus.go | 10 +- pkg/metadata/prometheus_test.go | 2 +- pkg/metadata/proxy.go | 22 +-- pkg/promclient/promclient.go | 5 +- test/e2e/metadata_api_test.go | 12 +- 10 files changed, 165 insertions(+), 163 deletions(-) diff --git a/pkg/api/query/v1.go b/pkg/api/query/v1.go index 9a86612687..e186057208 100644 --- a/pkg/api/query/v1.go +++ b/pkg/api/query/v1.go @@ -808,7 +808,7 @@ func NewMetricMetadataHandler(client metadata.UnaryClient, enablePartialResponse } return func(r *http.Request) (interface{}, []error, *api.ApiError) { - req := &metadatapb.MetadataRequest{ + req := &metadatapb.MetricMetadataRequest{ // By default we use -1, which means no limit. Limit: -1, Metric: r.URL.Query().Get("metric"), @@ -824,7 +824,7 @@ func NewMetricMetadataHandler(client metadata.UnaryClient, enablePartialResponse req.Limit = int32(limit) } - t, warnings, err := client.Metadata(r.Context(), req) + t, warnings, err := client.MetricMetadata(r.Context(), req) if err != nil { return nil, nil, &api.ApiError{Typ: api.ErrorInternal, Err: errors.Wrap(err, "retrieving metadata")} } diff --git a/pkg/metadata/metadata.go b/pkg/metadata/metadata.go index 9396b88192..c92bfda160 100644 --- a/pkg/metadata/metadata.go +++ b/pkg/metadata/metadata.go @@ -13,10 +13,10 @@ import ( var _ UnaryClient = &GRPCClient{} -// UnaryClient is gRPC metadatapb.Metadata client which expands streaming metadata API. Useful for consumers that does not +// UnaryClient is a gRPC metadatapb.Metadata client which expands streaming metadata API. Useful for consumers that does not // support streaming. type UnaryClient interface { - Metadata(ctx context.Context, req *metadatapb.MetadataRequest) (map[string][]metadatapb.Meta, storage.Warnings, error) + MetricMetadata(ctx context.Context, req *metadatapb.MetricMetadataRequest) (map[string][]metadatapb.Meta, storage.Warnings, error) } // GRPCClient allows to retrieve metadata from local gRPC streaming server implementation. @@ -31,7 +31,7 @@ func NewGRPCClient(ts metadatapb.MetadataServer) *GRPCClient { } } -func (rr *GRPCClient) Metadata(ctx context.Context, req *metadatapb.MetadataRequest) (map[string][]metadatapb.Meta, storage.Warnings, error) { +func (rr *GRPCClient) MetricMetadata(ctx context.Context, req *metadatapb.MetricMetadataRequest) (map[string][]metadatapb.Meta, storage.Warnings, error) { srv := &metadataServer{ctx: ctx, metric: req.Metric, limit: int(req.Limit)} if req.Limit >= 0 { @@ -46,8 +46,8 @@ func (rr *GRPCClient) Metadata(ctx context.Context, req *metadatapb.MetadataRequ srv.metadataMap = make(map[string][]metadatapb.Meta) } - if err := rr.proxy.Metadata(req, srv); err != nil { - return nil, nil, errors.Wrap(err, "proxy Metadata") + if err := rr.proxy.MetricMetadata(req, srv); err != nil { + return nil, nil, errors.Wrap(err, "proxy MetricMetadata") } return srv.metadataMap, srv.warnings, nil @@ -55,7 +55,7 @@ func (rr *GRPCClient) Metadata(ctx context.Context, req *metadatapb.MetadataRequ type metadataServer struct { // This field just exist to pseudo-implement the unused methods of the interface. - metadatapb.Metadata_MetadataServer + metadatapb.Metadata_MetricMetadataServer ctx context.Context metric string @@ -65,7 +65,7 @@ type metadataServer struct { metadataMap map[string][]metadatapb.Meta } -func (srv *metadataServer) Send(res *metadatapb.MetadataResponse) error { +func (srv *metadataServer) Send(res *metadatapb.MetricMetadataResponse) error { if res.GetWarning() != "" { srv.warnings = append(srv.warnings, errors.New(res.GetWarning())) return nil diff --git a/pkg/metadata/metadatapb/custom.go b/pkg/metadata/metadatapb/custom.go index 36f53a0889..697277aaa3 100644 --- a/pkg/metadata/metadatapb/custom.go +++ b/pkg/metadata/metadatapb/custom.go @@ -7,17 +7,17 @@ import ( "unsafe" ) -func NewMetadataResponse(metadata *MetricMetadata) *MetadataResponse { - return &MetadataResponse{ - Result: &MetadataResponse_Metadata{ +func NewMetricMetadataResponse(metadata *MetricMetadata) *MetricMetadataResponse { + return &MetricMetadataResponse{ + Result: &MetricMetadataResponse_Metadata{ Metadata: metadata, }, } } -func NewWarningMetadataResponse(warning error) *MetadataResponse { - return &MetadataResponse{ - Result: &MetadataResponse_Warning{ +func NewWarningMetadataResponse(warning error) *MetricMetadataResponse { + return &MetricMetadataResponse{ + Result: &MetricMetadataResponse_Warning{ Warning: warning.Error(), }, } diff --git a/pkg/metadata/metadatapb/rpc.pb.go b/pkg/metadata/metadatapb/rpc.pb.go index 33f3b09a96..7631ed49cb 100644 --- a/pkg/metadata/metadatapb/rpc.pb.go +++ b/pkg/metadata/metadatapb/rpc.pb.go @@ -29,24 +29,24 @@ var _ = math.Inf // proto package needs to be updated. const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package -type MetadataRequest struct { +type MetricMetadataRequest struct { Metric string `protobuf:"bytes,1,opt,name=metric,proto3" json:"metric,omitempty"` Limit int32 `protobuf:"varint,2,opt,name=limit,proto3" json:"limit,omitempty"` PartialResponseStrategy storepb.PartialResponseStrategy `protobuf:"varint,3,opt,name=partial_response_strategy,json=partialResponseStrategy,proto3,enum=thanos.PartialResponseStrategy" json:"partial_response_strategy,omitempty"` } -func (m *MetadataRequest) Reset() { *m = MetadataRequest{} } -func (m *MetadataRequest) String() string { return proto.CompactTextString(m) } -func (*MetadataRequest) ProtoMessage() {} -func (*MetadataRequest) Descriptor() ([]byte, []int) { +func (m *MetricMetadataRequest) Reset() { *m = MetricMetadataRequest{} } +func (m *MetricMetadataRequest) String() string { return proto.CompactTextString(m) } +func (*MetricMetadataRequest) ProtoMessage() {} +func (*MetricMetadataRequest) Descriptor() ([]byte, []int) { return fileDescriptor_1d9ae5661e0dc3fc, []int{0} } -func (m *MetadataRequest) XXX_Unmarshal(b []byte) error { +func (m *MetricMetadataRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) } -func (m *MetadataRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { +func (m *MetricMetadataRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { if deterministic { - return xxx_messageInfo_MetadataRequest.Marshal(b, m, deterministic) + return xxx_messageInfo_MetricMetadataRequest.Marshal(b, m, deterministic) } else { b = b[:cap(b)] n, err := m.MarshalToSizedBuffer(b) @@ -56,37 +56,37 @@ func (m *MetadataRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, err return b[:n], nil } } -func (m *MetadataRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_MetadataRequest.Merge(m, src) +func (m *MetricMetadataRequest) XXX_Merge(src proto.Message) { + xxx_messageInfo_MetricMetadataRequest.Merge(m, src) } -func (m *MetadataRequest) XXX_Size() int { +func (m *MetricMetadataRequest) XXX_Size() int { return m.Size() } -func (m *MetadataRequest) XXX_DiscardUnknown() { - xxx_messageInfo_MetadataRequest.DiscardUnknown(m) +func (m *MetricMetadataRequest) XXX_DiscardUnknown() { + xxx_messageInfo_MetricMetadataRequest.DiscardUnknown(m) } -var xxx_messageInfo_MetadataRequest proto.InternalMessageInfo +var xxx_messageInfo_MetricMetadataRequest proto.InternalMessageInfo -type MetadataResponse struct { +type MetricMetadataResponse struct { // Types that are valid to be assigned to Result: - // *MetadataResponse_Metadata - // *MetadataResponse_Warning - Result isMetadataResponse_Result `protobuf_oneof:"result"` + // *MetricMetadataResponse_Metadata + // *MetricMetadataResponse_Warning + Result isMetricMetadataResponse_Result `protobuf_oneof:"result"` } -func (m *MetadataResponse) Reset() { *m = MetadataResponse{} } -func (m *MetadataResponse) String() string { return proto.CompactTextString(m) } -func (*MetadataResponse) ProtoMessage() {} -func (*MetadataResponse) Descriptor() ([]byte, []int) { +func (m *MetricMetadataResponse) Reset() { *m = MetricMetadataResponse{} } +func (m *MetricMetadataResponse) String() string { return proto.CompactTextString(m) } +func (*MetricMetadataResponse) ProtoMessage() {} +func (*MetricMetadataResponse) Descriptor() ([]byte, []int) { return fileDescriptor_1d9ae5661e0dc3fc, []int{1} } -func (m *MetadataResponse) XXX_Unmarshal(b []byte) error { +func (m *MetricMetadataResponse) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) } -func (m *MetadataResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { +func (m *MetricMetadataResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { if deterministic { - return xxx_messageInfo_MetadataResponse.Marshal(b, m, deterministic) + return xxx_messageInfo_MetricMetadataResponse.Marshal(b, m, deterministic) } else { b = b[:cap(b)] n, err := m.MarshalToSizedBuffer(b) @@ -96,60 +96,60 @@ func (m *MetadataResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, er return b[:n], nil } } -func (m *MetadataResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_MetadataResponse.Merge(m, src) +func (m *MetricMetadataResponse) XXX_Merge(src proto.Message) { + xxx_messageInfo_MetricMetadataResponse.Merge(m, src) } -func (m *MetadataResponse) XXX_Size() int { +func (m *MetricMetadataResponse) XXX_Size() int { return m.Size() } -func (m *MetadataResponse) XXX_DiscardUnknown() { - xxx_messageInfo_MetadataResponse.DiscardUnknown(m) +func (m *MetricMetadataResponse) XXX_DiscardUnknown() { + xxx_messageInfo_MetricMetadataResponse.DiscardUnknown(m) } -var xxx_messageInfo_MetadataResponse proto.InternalMessageInfo +var xxx_messageInfo_MetricMetadataResponse proto.InternalMessageInfo -type isMetadataResponse_Result interface { - isMetadataResponse_Result() +type isMetricMetadataResponse_Result interface { + isMetricMetadataResponse_Result() MarshalTo([]byte) (int, error) Size() int } -type MetadataResponse_Metadata struct { +type MetricMetadataResponse_Metadata struct { Metadata *MetricMetadata `protobuf:"bytes,1,opt,name=metadata,proto3,oneof" json:"metadata,omitempty"` } -type MetadataResponse_Warning struct { +type MetricMetadataResponse_Warning struct { Warning string `protobuf:"bytes,2,opt,name=warning,proto3,oneof" json:"warning,omitempty"` } -func (*MetadataResponse_Metadata) isMetadataResponse_Result() {} -func (*MetadataResponse_Warning) isMetadataResponse_Result() {} +func (*MetricMetadataResponse_Metadata) isMetricMetadataResponse_Result() {} +func (*MetricMetadataResponse_Warning) isMetricMetadataResponse_Result() {} -func (m *MetadataResponse) GetResult() isMetadataResponse_Result { +func (m *MetricMetadataResponse) GetResult() isMetricMetadataResponse_Result { if m != nil { return m.Result } return nil } -func (m *MetadataResponse) GetMetadata() *MetricMetadata { - if x, ok := m.GetResult().(*MetadataResponse_Metadata); ok { +func (m *MetricMetadataResponse) GetMetadata() *MetricMetadata { + if x, ok := m.GetResult().(*MetricMetadataResponse_Metadata); ok { return x.Metadata } return nil } -func (m *MetadataResponse) GetWarning() string { - if x, ok := m.GetResult().(*MetadataResponse_Warning); ok { +func (m *MetricMetadataResponse) GetWarning() string { + if x, ok := m.GetResult().(*MetricMetadataResponse_Warning); ok { return x.Warning } return "" } // XXX_OneofWrappers is for the internal use of the proto package. -func (*MetadataResponse) XXX_OneofWrappers() []interface{} { +func (*MetricMetadataResponse) XXX_OneofWrappers() []interface{} { return []interface{}{ - (*MetadataResponse_Metadata)(nil), - (*MetadataResponse_Warning)(nil), + (*MetricMetadataResponse_Metadata)(nil), + (*MetricMetadataResponse_Warning)(nil), } } @@ -267,8 +267,8 @@ func (m *Meta) XXX_DiscardUnknown() { var xxx_messageInfo_Meta proto.InternalMessageInfo func init() { - proto.RegisterType((*MetadataRequest)(nil), "thanos.MetadataRequest") - proto.RegisterType((*MetadataResponse)(nil), "thanos.MetadataResponse") + proto.RegisterType((*MetricMetadataRequest)(nil), "thanos.MetricMetadataRequest") + proto.RegisterType((*MetricMetadataResponse)(nil), "thanos.MetricMetadataResponse") proto.RegisterType((*MetricMetadata)(nil), "thanos.MetricMetadata") proto.RegisterMapType((map[string]MetricMetadataEntry)(nil), "thanos.MetricMetadata.MetadataEntry") proto.RegisterType((*MetricMetadataEntry)(nil), "thanos.MetricMetadataEntry") @@ -278,36 +278,37 @@ func init() { func init() { proto.RegisterFile("metadata/metadatapb/rpc.proto", fileDescriptor_1d9ae5661e0dc3fc) } var fileDescriptor_1d9ae5661e0dc3fc = []byte{ - // 463 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x6c, 0x53, 0xcd, 0x6e, 0xd3, 0x40, - 0x10, 0xf6, 0xe6, 0x8f, 0x74, 0x02, 0xa5, 0x5a, 0xaa, 0xd6, 0x35, 0xe0, 0x44, 0x16, 0x07, 0x9f, - 0x62, 0x30, 0x1c, 0x10, 0x97, 0x4a, 0x91, 0x40, 0x95, 0x50, 0x25, 0xb4, 0x5c, 0x10, 0x1c, 0xca, - 0xa6, 0xac, 0x52, 0x0b, 0xc7, 0x5e, 0x76, 0xd7, 0x20, 0xbf, 0x05, 0x0f, 0xc0, 0x53, 0xf0, 0x14, - 0x39, 0xf6, 0xc8, 0xa9, 0x82, 0xe4, 0xc6, 0x53, 0xa0, 0xdd, 0xb5, 0x5b, 0x47, 0xf8, 0x32, 0xfa, - 0x66, 0xbe, 0xcf, 0x33, 0x9f, 0x66, 0xc7, 0xf0, 0x70, 0xc9, 0x14, 0xfd, 0x44, 0x15, 0x8d, 0x6a, - 0xc0, 0xe7, 0x91, 0xe0, 0xe7, 0x53, 0x2e, 0x72, 0x95, 0xe3, 0x81, 0xba, 0xa0, 0x59, 0x2e, 0xbd, - 0x23, 0xa9, 0x72, 0xc1, 0x22, 0x13, 0xf9, 0x3c, 0x52, 0x25, 0x67, 0xd2, 0x4a, 0xbc, 0xfd, 0x45, - 0xbe, 0xc8, 0x0d, 0x8c, 0x34, 0xb2, 0xd5, 0xe0, 0x07, 0x82, 0xbb, 0xa7, 0x55, 0x47, 0xc2, 0xbe, - 0x14, 0x4c, 0x2a, 0x7c, 0x00, 0x83, 0x25, 0x53, 0x22, 0x39, 0x77, 0xd1, 0x04, 0x85, 0x3b, 0xa4, - 0xca, 0xf0, 0x3e, 0xf4, 0xd3, 0x64, 0x99, 0x28, 0xb7, 0x33, 0x41, 0x61, 0x9f, 0xd8, 0x04, 0x7f, - 0x80, 0x23, 0x4e, 0x85, 0x4a, 0x68, 0x7a, 0x26, 0x98, 0xe4, 0x79, 0x26, 0xd9, 0x99, 0x54, 0x82, - 0x2a, 0xb6, 0x28, 0xdd, 0xee, 0x04, 0x85, 0xbb, 0xf1, 0x78, 0x6a, 0xed, 0x4d, 0xdf, 0x58, 0x21, - 0xa9, 0x74, 0x6f, 0x2b, 0x19, 0x39, 0xe4, 0xed, 0x44, 0x90, 0xc1, 0xde, 0x8d, 0x3b, 0xcb, 0xe1, - 0x67, 0x30, 0xac, 0x77, 0x60, 0x0c, 0x8e, 0xe2, 0x83, 0xba, 0xff, 0xa9, 0x31, 0x5a, 0x7f, 0x71, - 0xe2, 0x90, 0x6b, 0x25, 0xf6, 0xe0, 0xd6, 0x37, 0x2a, 0xb2, 0x24, 0x5b, 0x18, 0xfb, 0x3b, 0x27, - 0x0e, 0xa9, 0x0b, 0xb3, 0x21, 0x0c, 0x04, 0x93, 0x45, 0xaa, 0x82, 0x9f, 0x08, 0x76, 0xb7, 0x9b, - 0xe0, 0x57, 0x5b, 0xe3, 0xba, 0xe1, 0x28, 0x7e, 0xd4, 0x3e, 0x6e, 0x5a, 0x83, 0x97, 0x99, 0x12, - 0xe5, 0xac, 0xb7, 0xba, 0x1a, 0x37, 0x0c, 0x78, 0xef, 0xe0, 0xce, 0x96, 0x00, 0xef, 0x41, 0xf7, - 0x33, 0x2b, 0xab, 0x1d, 0x6b, 0x88, 0x9f, 0x40, 0xff, 0x2b, 0x4d, 0x0b, 0x66, 0x1c, 0x8e, 0xe2, - 0xfb, 0xed, 0x73, 0xcc, 0xd7, 0xc4, 0x2a, 0x5f, 0x74, 0x9e, 0xa3, 0xe0, 0x18, 0xee, 0xb5, 0x28, - 0x70, 0x08, 0x7d, 0x3d, 0x5c, 0xba, 0x1d, 0xe3, 0xfa, 0x76, 0xa3, 0x1b, 0xad, 0xdc, 0x59, 0x41, - 0xf0, 0x11, 0x7a, 0xba, 0x88, 0x1f, 0x40, 0x4f, 0x5f, 0x8c, 0xb5, 0x34, 0x1b, 0xfe, 0xbd, 0x1a, - 0x9b, 0x9c, 0x98, 0xa8, 0xd9, 0x0b, 0x96, 0x72, 0xbb, 0x3e, 0xcb, 0xea, 0x9c, 0x98, 0xa8, 0xd9, - 0x22, 0x4b, 0x94, 0x79, 0xf1, 0x8a, 0xd5, 0x39, 0x31, 0x31, 0x7e, 0x0d, 0xc3, 0xeb, 0x85, 0x1e, - 0x37, 0xf0, 0x61, 0xd3, 0x54, 0xe3, 0x06, 0x3d, 0xf7, 0x7f, 0xc2, 0x3e, 0xff, 0x63, 0x34, 0x0b, - 0x57, 0x7f, 0x7c, 0x67, 0xb5, 0xf6, 0xd1, 0xe5, 0xda, 0x47, 0xbf, 0xd7, 0x3e, 0xfa, 0xbe, 0xf1, - 0x9d, 0xcb, 0x8d, 0xef, 0xfc, 0xda, 0xf8, 0xce, 0x7b, 0xb8, 0xf9, 0x41, 0xe6, 0x03, 0x73, 0xe4, - 0x4f, 0xff, 0x05, 0x00, 0x00, 0xff, 0xff, 0x08, 0x77, 0xe4, 0x56, 0x3e, 0x03, 0x00, 0x00, + // 465 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x74, 0x53, 0xcd, 0x6e, 0x13, 0x31, + 0x10, 0x5e, 0xe7, 0x8f, 0x74, 0x02, 0x15, 0x32, 0x25, 0xa4, 0x0b, 0xdd, 0x44, 0x2b, 0x0e, 0x7b, + 0xca, 0xc2, 0xc2, 0x01, 0x71, 0x41, 0x8a, 0x04, 0xea, 0xa5, 0x12, 0x98, 0x0b, 0x02, 0xa1, 0xe2, + 0x14, 0x2b, 0x5d, 0xb1, 0xd9, 0x35, 0xf6, 0x04, 0x94, 0xb7, 0xe0, 0x19, 0x78, 0x04, 0x9e, 0x22, + 0xc7, 0x1e, 0x39, 0x55, 0x90, 0xdc, 0x78, 0x0a, 0x64, 0x7b, 0xb7, 0x34, 0xb0, 0x5c, 0x46, 0x33, + 0xf3, 0x7d, 0x9e, 0xf9, 0x3c, 0x1e, 0xc3, 0xc1, 0x5c, 0x20, 0x7f, 0xcf, 0x91, 0xc7, 0x95, 0x23, + 0xa7, 0xb1, 0x92, 0x27, 0x63, 0xa9, 0x0a, 0x2c, 0x68, 0x07, 0x4f, 0x79, 0x5e, 0x68, 0x7f, 0x5f, + 0x63, 0xa1, 0x44, 0x6c, 0xad, 0x9c, 0xc6, 0xb8, 0x94, 0x42, 0x3b, 0x8a, 0xbf, 0x37, 0x2b, 0x66, + 0x85, 0x75, 0x63, 0xe3, 0xb9, 0x6c, 0xf8, 0x95, 0xc0, 0xcd, 0x23, 0x81, 0x2a, 0x3d, 0x39, 0x2a, + 0xeb, 0x32, 0xf1, 0x71, 0x21, 0x34, 0xd2, 0x3e, 0x74, 0xe6, 0x16, 0x18, 0x90, 0x11, 0x89, 0x76, + 0x58, 0x19, 0xd1, 0x3d, 0x68, 0x67, 0xe9, 0x3c, 0xc5, 0x41, 0x63, 0x44, 0xa2, 0x36, 0x73, 0x01, + 0x7d, 0x03, 0xfb, 0x92, 0x2b, 0x4c, 0x79, 0x76, 0xac, 0x84, 0x96, 0x45, 0xae, 0xc5, 0xb1, 0x46, + 0xc5, 0x51, 0xcc, 0x96, 0x83, 0xe6, 0x88, 0x44, 0xbb, 0xc9, 0x70, 0xec, 0x44, 0x8e, 0x9f, 0x3b, + 0x22, 0x2b, 0x79, 0x2f, 0x4b, 0x1a, 0xbb, 0x25, 0xeb, 0x81, 0x10, 0xa1, 0xff, 0xb7, 0x46, 0xc7, + 0xa0, 0x0f, 0xa1, 0x5b, 0xcd, 0xc3, 0xca, 0xec, 0x25, 0xfd, 0xaa, 0xcb, 0xf6, 0x89, 0x43, 0x8f, + 0x5d, 0x30, 0xa9, 0x0f, 0x57, 0x3e, 0x73, 0x95, 0xa7, 0xf9, 0xcc, 0x5e, 0x62, 0xe7, 0xd0, 0x63, + 0x55, 0x62, 0xd2, 0x85, 0x8e, 0x12, 0x7a, 0x91, 0x61, 0xf8, 0x8d, 0xc0, 0xee, 0x76, 0x11, 0xfa, + 0x6c, 0xab, 0x5d, 0x33, 0xea, 0x25, 0x77, 0xeb, 0xdb, 0x8d, 0x2b, 0xe7, 0x69, 0x8e, 0x6a, 0x39, + 0x69, 0xad, 0xce, 0x87, 0x97, 0x04, 0xf8, 0xaf, 0xe0, 0xda, 0x16, 0x81, 0x5e, 0x87, 0xe6, 0x07, + 0xb1, 0x2c, 0x27, 0x6d, 0x5c, 0x7a, 0x1f, 0xda, 0x9f, 0x78, 0xb6, 0x10, 0x56, 0x61, 0x2f, 0xb9, + 0x5d, 0xdf, 0xc7, 0x9e, 0x66, 0x8e, 0xf9, 0xb8, 0xf1, 0x88, 0x84, 0x4f, 0xe0, 0x46, 0x0d, 0x83, + 0x46, 0xd0, 0x36, 0xcd, 0xf5, 0xa0, 0x61, 0x55, 0x5f, 0xbd, 0x54, 0x8d, 0x97, 0xea, 0x1c, 0x21, + 0x7c, 0x07, 0x2d, 0x93, 0xa4, 0x77, 0xa0, 0x65, 0xb6, 0xc7, 0x49, 0x9a, 0x74, 0x7f, 0x9d, 0x0f, + 0x6d, 0xcc, 0xac, 0x35, 0xe8, 0xa9, 0xc8, 0xa4, 0x1b, 0x9f, 0x43, 0x4d, 0xcc, 0xac, 0x35, 0xe8, + 0x22, 0x4f, 0xd1, 0xbe, 0x7b, 0x89, 0x9a, 0x98, 0x59, 0x9b, 0xbc, 0x85, 0xee, 0xc5, 0x40, 0x5f, + 0xfc, 0x33, 0xe2, 0x83, 0xfa, 0x8b, 0x96, 0x5b, 0xe9, 0x07, 0xff, 0x83, 0xdd, 0x42, 0xdc, 0x23, + 0x93, 0x68, 0xf5, 0x33, 0xf0, 0x56, 0xeb, 0x80, 0x9c, 0xad, 0x03, 0xf2, 0x63, 0x1d, 0x90, 0x2f, + 0x9b, 0xc0, 0x3b, 0xdb, 0x04, 0xde, 0xf7, 0x4d, 0xe0, 0xbd, 0x86, 0x3f, 0xdf, 0x67, 0xda, 0xb1, + 0x5f, 0xe0, 0xc1, 0xef, 0x00, 0x00, 0x00, 0xff, 0xff, 0xfa, 0x76, 0xe8, 0xa7, 0x5c, 0x03, 0x00, + 0x00, } // Reference imports to suppress errors if they are not otherwise used. @@ -322,7 +323,7 @@ const _ = grpc.SupportPackageIsVersion4 // // For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream. type MetadataClient interface { - Metadata(ctx context.Context, in *MetadataRequest, opts ...grpc.CallOption) (Metadata_MetadataClient, error) + MetricMetadata(ctx context.Context, in *MetricMetadataRequest, opts ...grpc.CallOption) (Metadata_MetricMetadataClient, error) } type metadataClient struct { @@ -333,12 +334,12 @@ func NewMetadataClient(cc *grpc.ClientConn) MetadataClient { return &metadataClient{cc} } -func (c *metadataClient) Metadata(ctx context.Context, in *MetadataRequest, opts ...grpc.CallOption) (Metadata_MetadataClient, error) { - stream, err := c.cc.NewStream(ctx, &_Metadata_serviceDesc.Streams[0], "/thanos.Metadata/Metadata", opts...) +func (c *metadataClient) MetricMetadata(ctx context.Context, in *MetricMetadataRequest, opts ...grpc.CallOption) (Metadata_MetricMetadataClient, error) { + stream, err := c.cc.NewStream(ctx, &_Metadata_serviceDesc.Streams[0], "/thanos.Metadata/MetricMetadata", opts...) if err != nil { return nil, err } - x := &metadataMetadataClient{stream} + x := &metadataMetricMetadataClient{stream} if err := x.ClientStream.SendMsg(in); err != nil { return nil, err } @@ -348,17 +349,17 @@ func (c *metadataClient) Metadata(ctx context.Context, in *MetadataRequest, opts return x, nil } -type Metadata_MetadataClient interface { - Recv() (*MetadataResponse, error) +type Metadata_MetricMetadataClient interface { + Recv() (*MetricMetadataResponse, error) grpc.ClientStream } -type metadataMetadataClient struct { +type metadataMetricMetadataClient struct { grpc.ClientStream } -func (x *metadataMetadataClient) Recv() (*MetadataResponse, error) { - m := new(MetadataResponse) +func (x *metadataMetricMetadataClient) Recv() (*MetricMetadataResponse, error) { + m := new(MetricMetadataResponse) if err := x.ClientStream.RecvMsg(m); err != nil { return nil, err } @@ -367,39 +368,39 @@ func (x *metadataMetadataClient) Recv() (*MetadataResponse, error) { // MetadataServer is the server API for Metadata service. type MetadataServer interface { - Metadata(*MetadataRequest, Metadata_MetadataServer) error + MetricMetadata(*MetricMetadataRequest, Metadata_MetricMetadataServer) error } // UnimplementedMetadataServer can be embedded to have forward compatible implementations. type UnimplementedMetadataServer struct { } -func (*UnimplementedMetadataServer) Metadata(req *MetadataRequest, srv Metadata_MetadataServer) error { - return status.Errorf(codes.Unimplemented, "method Metadata not implemented") +func (*UnimplementedMetadataServer) MetricMetadata(req *MetricMetadataRequest, srv Metadata_MetricMetadataServer) error { + return status.Errorf(codes.Unimplemented, "method MetricMetadata not implemented") } func RegisterMetadataServer(s *grpc.Server, srv MetadataServer) { s.RegisterService(&_Metadata_serviceDesc, srv) } -func _Metadata_Metadata_Handler(srv interface{}, stream grpc.ServerStream) error { - m := new(MetadataRequest) +func _Metadata_MetricMetadata_Handler(srv interface{}, stream grpc.ServerStream) error { + m := new(MetricMetadataRequest) if err := stream.RecvMsg(m); err != nil { return err } - return srv.(MetadataServer).Metadata(m, &metadataMetadataServer{stream}) + return srv.(MetadataServer).MetricMetadata(m, &metadataMetricMetadataServer{stream}) } -type Metadata_MetadataServer interface { - Send(*MetadataResponse) error +type Metadata_MetricMetadataServer interface { + Send(*MetricMetadataResponse) error grpc.ServerStream } -type metadataMetadataServer struct { +type metadataMetricMetadataServer struct { grpc.ServerStream } -func (x *metadataMetadataServer) Send(m *MetadataResponse) error { +func (x *metadataMetricMetadataServer) Send(m *MetricMetadataResponse) error { return x.ServerStream.SendMsg(m) } @@ -409,15 +410,15 @@ var _Metadata_serviceDesc = grpc.ServiceDesc{ Methods: []grpc.MethodDesc{}, Streams: []grpc.StreamDesc{ { - StreamName: "Metadata", - Handler: _Metadata_Metadata_Handler, + StreamName: "MetricMetadata", + Handler: _Metadata_MetricMetadata_Handler, ServerStreams: true, }, }, Metadata: "metadata/metadatapb/rpc.proto", } -func (m *MetadataRequest) Marshal() (dAtA []byte, err error) { +func (m *MetricMetadataRequest) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) n, err := m.MarshalToSizedBuffer(dAtA[:size]) @@ -427,12 +428,12 @@ func (m *MetadataRequest) Marshal() (dAtA []byte, err error) { return dAtA[:n], nil } -func (m *MetadataRequest) MarshalTo(dAtA []byte) (int, error) { +func (m *MetricMetadataRequest) MarshalTo(dAtA []byte) (int, error) { size := m.Size() return m.MarshalToSizedBuffer(dAtA[:size]) } -func (m *MetadataRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) { +func (m *MetricMetadataRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) { i := len(dAtA) _ = i var l int @@ -457,7 +458,7 @@ func (m *MetadataRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) { return len(dAtA) - i, nil } -func (m *MetadataResponse) Marshal() (dAtA []byte, err error) { +func (m *MetricMetadataResponse) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) n, err := m.MarshalToSizedBuffer(dAtA[:size]) @@ -467,12 +468,12 @@ func (m *MetadataResponse) Marshal() (dAtA []byte, err error) { return dAtA[:n], nil } -func (m *MetadataResponse) MarshalTo(dAtA []byte) (int, error) { +func (m *MetricMetadataResponse) MarshalTo(dAtA []byte) (int, error) { size := m.Size() return m.MarshalToSizedBuffer(dAtA[:size]) } -func (m *MetadataResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { +func (m *MetricMetadataResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { i := len(dAtA) _ = i var l int @@ -489,12 +490,12 @@ func (m *MetadataResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { return len(dAtA) - i, nil } -func (m *MetadataResponse_Metadata) MarshalTo(dAtA []byte) (int, error) { +func (m *MetricMetadataResponse_Metadata) MarshalTo(dAtA []byte) (int, error) { size := m.Size() return m.MarshalToSizedBuffer(dAtA[:size]) } -func (m *MetadataResponse_Metadata) MarshalToSizedBuffer(dAtA []byte) (int, error) { +func (m *MetricMetadataResponse_Metadata) MarshalToSizedBuffer(dAtA []byte) (int, error) { i := len(dAtA) if m.Metadata != nil { { @@ -510,12 +511,12 @@ func (m *MetadataResponse_Metadata) MarshalToSizedBuffer(dAtA []byte) (int, erro } return len(dAtA) - i, nil } -func (m *MetadataResponse_Warning) MarshalTo(dAtA []byte) (int, error) { +func (m *MetricMetadataResponse_Warning) MarshalTo(dAtA []byte) (int, error) { size := m.Size() return m.MarshalToSizedBuffer(dAtA[:size]) } -func (m *MetadataResponse_Warning) MarshalToSizedBuffer(dAtA []byte) (int, error) { +func (m *MetricMetadataResponse_Warning) MarshalToSizedBuffer(dAtA []byte) (int, error) { i := len(dAtA) i -= len(m.Warning) copy(dAtA[i:], m.Warning) @@ -663,7 +664,7 @@ func encodeVarintRpc(dAtA []byte, offset int, v uint64) int { dAtA[offset] = uint8(v) return base } -func (m *MetadataRequest) Size() (n int) { +func (m *MetricMetadataRequest) Size() (n int) { if m == nil { return 0 } @@ -682,7 +683,7 @@ func (m *MetadataRequest) Size() (n int) { return n } -func (m *MetadataResponse) Size() (n int) { +func (m *MetricMetadataResponse) Size() (n int) { if m == nil { return 0 } @@ -694,7 +695,7 @@ func (m *MetadataResponse) Size() (n int) { return n } -func (m *MetadataResponse_Metadata) Size() (n int) { +func (m *MetricMetadataResponse_Metadata) Size() (n int) { if m == nil { return 0 } @@ -706,7 +707,7 @@ func (m *MetadataResponse_Metadata) Size() (n int) { } return n } -func (m *MetadataResponse_Warning) Size() (n int) { +func (m *MetricMetadataResponse_Warning) Size() (n int) { if m == nil { return 0 } @@ -776,7 +777,7 @@ func sovRpc(x uint64) (n int) { func sozRpc(x uint64) (n int) { return sovRpc(uint64((x << 1) ^ uint64((int64(x) >> 63)))) } -func (m *MetadataRequest) Unmarshal(dAtA []byte) error { +func (m *MetricMetadataRequest) Unmarshal(dAtA []byte) error { l := len(dAtA) iNdEx := 0 for iNdEx < l { @@ -799,10 +800,10 @@ func (m *MetadataRequest) Unmarshal(dAtA []byte) error { fieldNum := int32(wire >> 3) wireType := int(wire & 0x7) if wireType == 4 { - return fmt.Errorf("proto: MetadataRequest: wiretype end group for non-group") + return fmt.Errorf("proto: MetricMetadataRequest: wiretype end group for non-group") } if fieldNum <= 0 { - return fmt.Errorf("proto: MetadataRequest: illegal tag %d (wire type %d)", fieldNum, wire) + return fmt.Errorf("proto: MetricMetadataRequest: illegal tag %d (wire type %d)", fieldNum, wire) } switch fieldNum { case 1: @@ -899,7 +900,7 @@ func (m *MetadataRequest) Unmarshal(dAtA []byte) error { } return nil } -func (m *MetadataResponse) Unmarshal(dAtA []byte) error { +func (m *MetricMetadataResponse) Unmarshal(dAtA []byte) error { l := len(dAtA) iNdEx := 0 for iNdEx < l { @@ -922,10 +923,10 @@ func (m *MetadataResponse) Unmarshal(dAtA []byte) error { fieldNum := int32(wire >> 3) wireType := int(wire & 0x7) if wireType == 4 { - return fmt.Errorf("proto: MetadataResponse: wiretype end group for non-group") + return fmt.Errorf("proto: MetricMetadataResponse: wiretype end group for non-group") } if fieldNum <= 0 { - return fmt.Errorf("proto: MetadataResponse: illegal tag %d (wire type %d)", fieldNum, wire) + return fmt.Errorf("proto: MetricMetadataResponse: illegal tag %d (wire type %d)", fieldNum, wire) } switch fieldNum { case 1: @@ -961,7 +962,7 @@ func (m *MetadataResponse) Unmarshal(dAtA []byte) error { if err := v.Unmarshal(dAtA[iNdEx:postIndex]); err != nil { return err } - m.Result = &MetadataResponse_Metadata{v} + m.Result = &MetricMetadataResponse_Metadata{v} iNdEx = postIndex case 2: if wireType != 2 { @@ -993,7 +994,7 @@ func (m *MetadataResponse) Unmarshal(dAtA []byte) error { if postIndex > l { return io.ErrUnexpectedEOF } - m.Result = &MetadataResponse_Warning{string(dAtA[iNdEx:postIndex])} + m.Result = &MetricMetadataResponse_Warning{string(dAtA[iNdEx:postIndex])} iNdEx = postIndex default: iNdEx = preIndex diff --git a/pkg/metadata/metadatapb/rpc.proto b/pkg/metadata/metadatapb/rpc.proto index 16f0706692..96ff350634 100644 --- a/pkg/metadata/metadatapb/rpc.proto +++ b/pkg/metadata/metadatapb/rpc.proto @@ -21,16 +21,16 @@ option (gogoproto.goproto_unrecognized_all) = false; option (gogoproto.goproto_sizecache_all) = false; service Metadata { - rpc Metadata(MetadataRequest) returns (stream MetadataResponse); + rpc MetricMetadata(MetricMetadataRequest) returns (stream MetricMetadataResponse); } -message MetadataRequest { +message MetricMetadataRequest { string metric = 1; int32 limit = 2; PartialResponseStrategy partial_response_strategy = 3; } -message MetadataResponse { +message MetricMetadataResponse { oneof result { /// A collection of metric metadata entries. MetricMetadata metadata = 1; diff --git a/pkg/metadata/prometheus.go b/pkg/metadata/prometheus.go index 40be80ffcb..49e6087fa0 100644 --- a/pkg/metadata/prometheus.go +++ b/pkg/metadata/prometheus.go @@ -10,7 +10,7 @@ import ( "github.com/thanos-io/thanos/pkg/promclient" ) -// Prometheus implements metadatapb.Metadata gRPC that allows to fetch metric metadata from Prometheus HTTP /api/v1/metadata endpoint. +// Prometheus implements metadatapb.Metadata gRPC service that allows to fetch metric metadata from Prometheus HTTP /api/v1/metadata endpoint. type Prometheus struct { base *url.URL client *promclient.Client @@ -24,13 +24,13 @@ func NewPrometheus(base *url.URL, client *promclient.Client) *Prometheus { } } -// Metadata returns all specified metric metadata from Prometheus. -func (p *Prometheus) Metadata(r *metadatapb.MetadataRequest, s metadatapb.Metadata_MetadataServer) error { - md, err := p.client.MetadataInGRPC(s.Context(), p.base, r.Metric, int(r.Limit)) +// MetricMetadata returns all specified metric metadata from Prometheus. +func (p *Prometheus) MetricMetadata(r *metadatapb.MetricMetadataRequest, s metadatapb.Metadata_MetricMetadataServer) error { + md, err := p.client.MetricMetadataInGRPC(s.Context(), p.base, r.Metric, int(r.Limit)) if err != nil { return err } - return s.Send(&metadatapb.MetadataResponse{Result: &metadatapb.MetadataResponse_Metadata{ + return s.Send(&metadatapb.MetricMetadataResponse{Result: &metadatapb.MetricMetadataResponse_Metadata{ Metadata: metadatapb.FromMetadataMap(md)}}) } diff --git a/pkg/metadata/prometheus_test.go b/pkg/metadata/prometheus_test.go index d87457959c..060c67da84 100644 --- a/pkg/metadata/prometheus_test.go +++ b/pkg/metadata/prometheus_test.go @@ -84,7 +84,7 @@ scrape_configs: }, } { t.Run(tcase.name, func(t *testing.T) { - meta, w, err := NewGRPCClient(prom).Metadata(context.Background(), &metadatapb.MetadataRequest{ + meta, w, err := NewGRPCClient(prom).MetricMetadata(context.Background(), &metadatapb.MetricMetadataRequest{ Metric: tcase.metric, Limit: tcase.limit, }) diff --git a/pkg/metadata/proxy.go b/pkg/metadata/proxy.go index 8046053582..b133f3e89f 100644 --- a/pkg/metadata/proxy.go +++ b/pkg/metadata/proxy.go @@ -39,7 +39,7 @@ func NewProxy(logger log.Logger, metadata func() []metadatapb.MetadataClient) *P } } -func (s *Proxy) Metadata(req *metadatapb.MetadataRequest, srv metadatapb.Metadata_MetadataServer) error { +func (s *Proxy) MetricMetadata(req *metadatapb.MetricMetadataRequest, srv metadatapb.Metadata_MetricMetadataServer) error { var ( g, gctx = errgroup.WithContext(srv.Context()) respChan = make(chan *metadatapb.MetricMetadata, 10) @@ -47,7 +47,7 @@ func (s *Proxy) Metadata(req *metadatapb.MetadataRequest, srv metadatapb.Metadat ) for _, metadataClient := range s.metadata() { - rs := &metadataStream{ + rs := &metricMetadataStream{ client: metadataClient, request: req, channel: respChan, @@ -71,25 +71,25 @@ func (s *Proxy) Metadata(req *metadatapb.MetadataRequest, srv metadatapb.Metadat } for _, t := range metas { - if err := srv.Send(metadatapb.NewMetadataResponse(t)); err != nil { - return status.Error(codes.Unknown, errors.Wrap(err, "send metadata response").Error()) + if err := srv.Send(metadatapb.NewMetricMetadataResponse(t)); err != nil { + return status.Error(codes.Unknown, errors.Wrap(err, "send metric metadata response").Error()) } } return nil } -type metadataStream struct { +type metricMetadataStream struct { client metadatapb.MetadataClient - request *metadatapb.MetadataRequest + request *metadatapb.MetricMetadataRequest channel chan<- *metadatapb.MetricMetadata - server metadatapb.Metadata_MetadataServer + server metadatapb.Metadata_MetricMetadataServer } -func (stream *metadataStream) receive(ctx context.Context) error { - metadataCli, err := stream.client.Metadata(ctx, stream.request) +func (stream *metricMetadataStream) receive(ctx context.Context) error { + metadataCli, err := stream.client.MetricMetadata(ctx, stream.request) if err != nil { - err = errors.Wrapf(err, "fetching metadata from metadata client %v", stream.client) + err = errors.Wrapf(err, "fetching metric metadata from metadata client %v", stream.client) if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT { return err @@ -109,7 +109,7 @@ func (stream *metadataStream) receive(ctx context.Context) error { } if err != nil { - err = errors.Wrapf(err, "receiving metadata from metadata client %v", stream.client) + err = errors.Wrapf(err, "receiving metric metadata from metadata client %v", stream.client) if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT { return err diff --git a/pkg/promclient/promclient.go b/pkg/promclient/promclient.go index 58fb296a03..9557f3ff52 100644 --- a/pkg/promclient/promclient.go +++ b/pkg/promclient/promclient.go @@ -738,7 +738,8 @@ func (c *Client) RulesInGRPC(ctx context.Context, base *url.URL, typeRules strin return m.Data.Groups, nil } -func (c *Client) MetadataInGRPC(ctx context.Context, base *url.URL, metric string, limit int) (map[string][]metadatapb.Meta, error) { +// MetricMetadataInGRPC returns the metadata from Prometheus metric metadata API. It uses gRPC errors. +func (c *Client) MetricMetadataInGRPC(ctx context.Context, base *url.URL, metric string, limit int) (map[string][]metadatapb.Meta, error) { u := *base u.Path = path.Join(u.Path, "/api/v1/metadata") q := u.Query() @@ -756,5 +757,5 @@ func (c *Client) MetadataInGRPC(ctx context.Context, base *url.URL, metric strin var v struct { Data map[string][]metadatapb.Meta `json:"data"` } - return v.Data, c.get2xxResultWithGRPCErrors(ctx, "/metadata HTTP[client]", &u, &v) + return v.Data, c.get2xxResultWithGRPCErrors(ctx, "/prom_metric_metadata HTTP[client]", &u, &v) } diff --git a/test/e2e/metadata_api_test.go b/test/e2e/metadata_api_test.go index 6211525f0c..30a39d571a 100644 --- a/test/e2e/metadata_api_test.go +++ b/test/e2e/metadata_api_test.go @@ -64,32 +64,32 @@ func TestMetadataAPI_Fanout(t *testing.T) { testutil.Ok(t, q.WaitSumMetricsWithOptions(e2e.Equals(2), []string{"thanos_store_nodes_grpc_connections"}, e2e.WaitMissingMetrics)) testutil.Ok(t, q.WaitSumMetricsWithOptions(e2e.Equals(2), []string{"thanos_query_metadata_apis_dns_provider_results"}, e2e.WaitMissingMetrics)) - promMeta, err := promclient.NewDefaultClient().MetadataInGRPC(ctx, mustURLParse(t, "http://"+prom1.HTTPEndpoint()), "", -1) + promMeta, err := promclient.NewDefaultClient().MetricMetadataInGRPC(ctx, mustURLParse(t, "http://"+prom1.HTTPEndpoint()), "", -1) testutil.Ok(t, err) - thanosMeta, err := promclient.NewDefaultClient().MetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", -1) + thanosMeta, err := promclient.NewDefaultClient().MetricMetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", -1) testutil.Ok(t, err) // Metadata response from Prometheus and Thanos Querier should be the same after deduplication. testutil.Equals(t, thanosMeta, promMeta) // We only expect to see one metadata returned. - thanosMeta, err = promclient.NewDefaultClient().MetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", 1) + thanosMeta, err = promclient.NewDefaultClient().MetricMetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", 1) testutil.Ok(t, err) testutil.Assert(t, true, len(thanosMeta) == 1) // We only expect to see ten metadata returned. - thanosMeta, err = promclient.NewDefaultClient().MetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", 10) + thanosMeta, err = promclient.NewDefaultClient().MetricMetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", 10) testutil.Ok(t, err) testutil.Assert(t, true, len(thanosMeta) == 10) // No metadata returned. - thanosMeta, err = promclient.NewDefaultClient().MetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", 0) + thanosMeta, err = promclient.NewDefaultClient().MetricMetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "", 0) testutil.Ok(t, err) testutil.Assert(t, true, len(thanosMeta) == 0) // Only prometheus_build_info metric will be returned. - thanosMeta, err = promclient.NewDefaultClient().MetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "prometheus_build_info", -1) + thanosMeta, err = promclient.NewDefaultClient().MetricMetadataInGRPC(ctx, mustURLParse(t, "http://"+q.HTTPEndpoint()), "prometheus_build_info", -1) testutil.Ok(t, err) testutil.Assert(t, true, len(thanosMeta) == 1 && len(thanosMeta["prometheus_build_info"]) > 0) } From 26077e0b4a90adb10c3eefcd95eafa3342a4a4ee Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Fri, 5 Mar 2021 22:21:37 +0100 Subject: [PATCH 04/15] Fix parseStep bug introduced in PR #3740 (#3887) (#3889) * Fix bug introduced in PR #3740 Signed-off-by: Hitanshu Mehta * Minor change in test Signed-off-by: Hitanshu Mehta Co-authored-by: Hitanshu Mehta <44025541+hitanshu-mehta@users.noreply.github.com> --- pkg/api/query/v1.go | 1 + pkg/api/query/v1_test.go | 14 ++++++++------ pkg/ui/static/js/graph.js | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/pkg/api/query/v1.go b/pkg/api/query/v1.go index e186057208..b27031e6e3 100644 --- a/pkg/api/query/v1.go +++ b/pkg/api/query/v1.go @@ -247,6 +247,7 @@ func (qapi *QueryAPI) parseStep(r *http.Request, defaultRangeQueryStep time.Dura if err != nil { return 0, &api.ApiError{Typ: api.ErrorBadData, Err: errors.Wrapf(err, "'%s' parameter", Step)} } + return defaultRangeQueryStep, nil } // Default step is used this way to make it consistent with UI. diff --git a/pkg/api/query/v1_test.go b/pkg/api/query/v1_test.go index f87deff16b..f93f6b2b17 100644 --- a/pkg/api/query/v1_test.go +++ b/pkg/api/query/v1_test.go @@ -465,18 +465,20 @@ func TestQueryEndpoints(t *testing.T) { query: url.Values{ "query": []string{"time()"}, "start": []string{"0"}, - "end": []string{"2"}, + "end": []string{"500"}, "step": []string{"1"}, }, response: &queryData{ ResultType: parser.ValueTypeMatrix, Result: promql.Matrix{ promql.Series{ - Points: []promql.Point{ - {V: 0, T: timestamp.FromTime(start)}, - {V: 1, T: timestamp.FromTime(start.Add(1 * time.Second))}, - {V: 2, T: timestamp.FromTime(start.Add(2 * time.Second))}, - }, + Points: func(end, step float64) []promql.Point { + var res []promql.Point + for v := float64(0); v <= end; v += step { + res = append(res, promql.Point{V: v, T: timestamp.FromTime(start.Add(time.Duration(v) * time.Second))}) + } + return res + }(500, 1), Metric: nil, }, }, diff --git a/pkg/ui/static/js/graph.js b/pkg/ui/static/js/graph.js index 6ba577a6c1..30d1fcaf46 100644 --- a/pkg/ui/static/js/graph.js +++ b/pkg/ui/static/js/graph.js @@ -309,7 +309,7 @@ Prometheus.Graph.prototype.setDefaultStep = function(el) { self.showError("Error querying flags."); return; } - el.defaultStep = (json.data && "query.default-stp" in json.data) ? json.data["query.default-stp"] : "1s" + el.defaultStep = (json.data && "query.default-step" in json.data) ? json.data["query.default-step"] : "1s" }, error: function() { From 92b4452ec6b404195c11ac4079c14a84d453ee79 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Fri, 5 Mar 2021 22:21:58 +0100 Subject: [PATCH 05/15] Tools: rewrite delete can delete series only if it matches all matchers (#3886) (#3890) * rewrite delete should delete a series only if it matches all matchers in a deletion request Signed-off-by: yeya24 * add test case Signed-off-by: yeya24 Co-authored-by: Ben Ye --- pkg/compactv2/compactor_test.go | 32 +++++++++++++++++++++++ pkg/compactv2/modifiers.go | 46 +++++++++++++++++---------------- 2 files changed, 56 insertions(+), 22 deletions(-) diff --git a/pkg/compactv2/compactor_test.go b/pkg/compactv2/compactor_test.go index cc10a1897a..c1af292d1b 100644 --- a/pkg/compactv2/compactor_test.go +++ b/pkg/compactv2/compactor_test.go @@ -274,6 +274,38 @@ func TestCompactor_WriteSeries_e2e(t *testing.T) { NumChunks: 2, }, }, + { + name: "1 blocks + delete modifier, deletion request contains multiple matchers, delete second series", + input: [][]seriesSamples{ + { + {lset: labels.Labels{{Name: "a", Value: "1"}, {Name: "b", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 10}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "1"}, {Name: "b", Value: "2"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "3"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + }, + }, + modifiers: []Modifier{WithDeletionModifier( + metadata.DeletionRequest{ + Matchers: []*labels.Matcher{ + labels.MustNewMatcher(labels.MatchEqual, "a", "1"), + labels.MustNewMatcher(labels.MatchEqual, "b", "2"), + }, + })}, + expected: []seriesSamples{ + {lset: labels.Labels{{Name: "a", Value: "1"}, {Name: "b", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 10}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "3"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + }, + expectedChanges: "Deleted {a=\"1\", b=\"2\"} [{0 20}]\n", + expectedStats: tsdb.BlockStats{ + NumSamples: 12, + NumSeries: 2, + NumChunks: 2, + }, + }, } { t.Run(tcase.name, func(t *testing.T) { tmpDir, err := ioutil.TempDir("", "test-series-writer") diff --git a/pkg/compactv2/modifiers.go b/pkg/compactv2/modifiers.go index 2139d98d85..9d521c0fd2 100644 --- a/pkg/compactv2/modifiers.go +++ b/pkg/compactv2/modifiers.go @@ -58,6 +58,7 @@ SeriesLoop: lbls := s.Labels() var intervals tombstones.Intervals + DeletionsLoop: for _, deletions := range d.d.deletions { for _, m := range deletions.Matchers { v := lbls.Get(m.Name) @@ -65,34 +66,35 @@ SeriesLoop: continue } + // Only if all matchers in the deletion request are matched can we proceed to deletion. if !m.Matches(v) { - continue + continue DeletionsLoop } - if len(deletions.Intervals) > 0 { - for _, in := range deletions.Intervals { - intervals = intervals.Add(in) - } - break + } + if len(deletions.Intervals) > 0 { + for _, in := range deletions.Intervals { + intervals = intervals.Add(in) } + continue + } - // Special case: Delete whole series. - chksIter := s.Iterator() - var chks []chunks.Meta - for chksIter.Next() { - chks = append(chks, chksIter.At()) - } - if d.err = chksIter.Err(); d.err != nil { - return false - } + // Special case: Delete whole series. + chksIter := s.Iterator() + var chks []chunks.Meta + for chksIter.Next() { + chks = append(chks, chksIter.At()) + } + if d.err = chksIter.Err(); d.err != nil { + return false + } - var deleted tombstones.Intervals - if len(chks) > 0 { - deleted = deleted.Add(tombstones.Interval{Mint: chks[0].MinTime, Maxt: chks[len(chks)-1].MaxTime}) - } - d.log.DeleteSeries(lbls, deleted) - d.p.SeriesProcessed() - continue SeriesLoop + var deleted tombstones.Intervals + if len(chks) > 0 { + deleted = deleted.Add(tombstones.Interval{Mint: chks[0].MinTime, Maxt: chks[len(chks)-1].MaxTime}) } + d.log.DeleteSeries(lbls, deleted) + d.p.SeriesProcessed() + continue SeriesLoop } d.curr = &storage.ChunkSeriesEntry{ From b7da5a1129168809c5b190ef86eb9a7c78f3c55a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lucas=20Serv=C3=A9n=20Mar=C3=ADn?= Date: Tue, 9 Mar 2021 11:43:30 +0100 Subject: [PATCH 06/15] cmd/thanos/receive.go: Receive client infers TLS (#3899) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Currently, the thanos Receiver infers whether it should use TLS for the gRPC clients that forwards time series to other receivers based on whether the remote-write HTTP server uses TLS. This is not correct, as the HTTP server may use TLS without the gRPC server using TLS. This commit fixes the inference. Longer-term, this will be taken care of by the receive/router split. Signed-off-by: Lucas Servén Marín --- CHANGELOG.md | 1 + cmd/thanos/receive.go | 4 ++-- docs/components/receive.md | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6cdfaab601..e6be934d6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,7 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re - [#3814](https://github.com/thanos-io/thanos/pull/3814) Store: Decreased memory utilisation while fetching block's chunks. - [#3815](https://github.com/thanos-io/thanos/pull/3815) Receive: Improve handling of empty time series from clients - [#3795](https://github.com/thanos-io/thanos/pull/3795) s3: A truncated "get object" response is reported as error. +- [#3899](https://github.com/thanos-io/thanos/pull/3899) Receive: Correct the inference of client gRPC configuration. ### Changed diff --git a/cmd/thanos/receive.go b/cmd/thanos/receive.go index 6d401a7d08..b06551f263 100644 --- a/cmd/thanos/receive.go +++ b/cmd/thanos/receive.go @@ -55,7 +55,7 @@ func registerReceive(app *extkingpin.App) { rwClientCert := cmd.Flag("remote-write.client-tls-cert", "TLS Certificates to use to identify this client to the server.").Default("").String() rwClientKey := cmd.Flag("remote-write.client-tls-key", "TLS Key for the client's certificate.").Default("").String() rwClientServerCA := cmd.Flag("remote-write.client-tls-ca", "TLS CA Certificates to use to verify servers.").Default("").String() - rwClientServerName := cmd.Flag("remote-write.client-server-name", "Server name to verify the hostname on the returned gRPC certificates. See https://tools.ietf.org/html/rfc4366#section-3.1").Default("").String() + rwClientServerName := cmd.Flag("remote-write.client-server-name", "Server name to verify the hostname on the returned TLS certificates. See https://tools.ietf.org/html/rfc4366#section-3.1").Default("").String() dataDir := cmd.Flag("tsdb.path", "Data directory of TSDB."). Default("./data").String() @@ -220,7 +220,7 @@ func runReceive( if err != nil { return err } - dialOpts, err := extgrpc.StoreClientGRPCOpts(logger, reg, tracer, rwServerCert != "", rwClientCert, rwClientKey, rwClientServerCA, rwClientServerName) + dialOpts, err := extgrpc.StoreClientGRPCOpts(logger, reg, tracer, grpcCert != "", rwClientCert, rwClientKey, rwClientServerCA, rwClientServerName) if err != nil { return err } diff --git a/docs/components/receive.md b/docs/components/receive.md index 47a71febdc..42ada656b4 100644 --- a/docs/components/receive.md +++ b/docs/components/receive.md @@ -129,7 +129,7 @@ Flags: TLS CA Certificates to use to verify servers. --remote-write.client-server-name="" Server name to verify the hostname on the - returned gRPC certificates. See + returned TLS certificates. See https://tools.ietf.org/html/rfc4366#section-3.1 --tsdb.path="./data" Data directory of TSDB. --label=key="value" ... External labels to announce. This flag will be From 05280c95aead6f445ecfbf9a4fea7a1a46975194 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Tue, 9 Mar 2021 11:32:46 +0000 Subject: [PATCH 07/15] Cut v0.19.0-rc.1 (#3900) Signed-off-by: Bartlomiej Plotka --- CHANGELOG.md | 2 +- VERSION | 2 +- tutorials/katacoda/thanos/1-globalview/courseBase.sh | 2 +- tutorials/katacoda/thanos/1-globalview/step2.md | 8 ++++---- tutorials/katacoda/thanos/1-globalview/step3.md | 2 +- tutorials/katacoda/thanos/2-lts/courseBase.sh | 2 +- tutorials/katacoda/thanos/2-lts/step1.md | 4 ++-- tutorials/katacoda/thanos/2-lts/step2.md | 2 +- tutorials/katacoda/thanos/2-lts/step3.md | 6 +++--- tutorials/katacoda/thanos/2-lts/step4.md | 2 +- .../katacoda/thanos/6-query-caching/courseBase.sh | 2 +- tutorials/katacoda/thanos/6-query-caching/step1.md | 4 ++-- tutorials/katacoda/thanos/6-query-caching/step2.md | 2 +- .../katacoda/thanos/7-multi-tenancy/courseBase.sh | 2 +- tutorials/katacoda/thanos/7-multi-tenancy/step1.md | 10 +++++----- tutorials/katacoda/thanos/7-multi-tenancy/step2.md | 2 +- tutorials/katacoda/thanos/x-playground/courseBase.sh | 2 +- tutorials/katacoda/thanos/x-playground/step1.md | 10 +++++----- tutorials/katacoda/thanos/x-playground/step2.md | 12 ++++++------ 19 files changed, 39 insertions(+), 39 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e6be934d6d..c0aa5e5579 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ NOTE: As semantic versioning states all 0.y.z releases can contain breaking chan We use _breaking :warning:_ to mark changes that are not backward compatible (relates only to v0.y.z releases.) -## [v0.19.0-rc.0](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.0) - 2021.03.02 +## [v0.19.0-rc.1](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.1) - 2021.03.09 ### Added diff --git a/VERSION b/VERSION index d92951ecef..196f1821a1 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.19.0-rc.0 \ No newline at end of file +0.19.0-rc.1 \ No newline at end of file diff --git a/tutorials/katacoda/thanos/1-globalview/courseBase.sh b/tutorials/katacoda/thanos/1-globalview/courseBase.sh index 6e35780f49..6c77da4e83 100644 --- a/tutorials/katacoda/thanos/1-globalview/courseBase.sh +++ b/tutorials/katacoda/thanos/1-globalview/courseBase.sh @@ -1,4 +1,4 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.16.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.1 diff --git a/tutorials/katacoda/thanos/1-globalview/step2.md b/tutorials/katacoda/thanos/1-globalview/step2.md index 703c9600cb..0d6feef8d0 100644 --- a/tutorials/katacoda/thanos/1-globalview/step2.md +++ b/tutorials/katacoda/thanos/1-globalview/step2.md @@ -10,7 +10,7 @@ component and can be invoked in a single command. Let's take a look at all the Thanos commands: ``` -docker run --rm quay.io/thanos/thanos:v0.19.0-rc.0 --help +docker run --rm quay.io/thanos/thanos:v0.19.0-rc.1 --help ```{{execute}} You should see multiple commands that solves different purposes. @@ -53,7 +53,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-eu1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -68,7 +68,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -81,7 +81,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus1_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-1-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ diff --git a/tutorials/katacoda/thanos/1-globalview/step3.md b/tutorials/katacoda/thanos/1-globalview/step3.md index 35ae6cfc72..116f24c081 100644 --- a/tutorials/katacoda/thanos/1-globalview/step3.md +++ b/tutorials/katacoda/thanos/1-globalview/step3.md @@ -28,7 +28,7 @@ Click below snippet to start the Querier. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:29090 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/courseBase.sh b/tutorials/katacoda/thanos/2-lts/courseBase.sh index e9865553bc..f740592076 100644 --- a/tutorials/katacoda/thanos/2-lts/courseBase.sh +++ b/tutorials/katacoda/thanos/2-lts/courseBase.sh @@ -2,7 +2,7 @@ docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.1 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 mkdir /root/editor diff --git a/tutorials/katacoda/thanos/2-lts/step1.md b/tutorials/katacoda/thanos/2-lts/step1.md index 2aee230e8a..fba61a8d70 100644 --- a/tutorials/katacoda/thanos/2-lts/step1.md +++ b/tutorials/katacoda/thanos/2-lts/step1.md @@ -117,7 +117,7 @@ Similar to previous course, let's setup global view querying with sidecar: docker run -d --net=host --rm \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -130,7 +130,7 @@ so we will make sure we point the Querier to the gRPC endpoints of the sidecar: ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step2.md b/tutorials/katacoda/thanos/2-lts/step2.md index 3c269bf97d..561a72f2b9 100644 --- a/tutorials/katacoda/thanos/2-lts/step2.md +++ b/tutorials/katacoda/thanos/2-lts/step2.md @@ -79,7 +79,7 @@ docker run -d --net=host --rm \ -v /root/prom-eu1:/prometheus \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ diff --git a/tutorials/katacoda/thanos/2-lts/step3.md b/tutorials/katacoda/thanos/2-lts/step3.md index e008665237..a0c69d9ccf 100644 --- a/tutorials/katacoda/thanos/2-lts/step3.md +++ b/tutorials/katacoda/thanos/2-lts/step3.md @@ -6,7 +6,7 @@ In this step, we will learn about Thanos Store Gateway and how to deploy it. Let's take a look at all the Thanos commands: -```docker run --rm quay.io/thanos/thanos:v0.19.0-rc.0 --help```{{execute}} +```docker run --rm quay.io/thanos/thanos:v0.19.0-rc.1 --help```{{execute}} You should see multiple commands that solve different purposes, block storage based long-term storage for Prometheus. @@ -32,7 +32,7 @@ You can read more about [Store](https://thanos.io/tip/components/store.md/) here docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19091 \ @@ -49,7 +49,7 @@ Currently querier does not know about store yet. Let's change it by adding Store docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step4.md b/tutorials/katacoda/thanos/2-lts/step4.md index 56a2dd2bda..8b78709b7a 100644 --- a/tutorials/katacoda/thanos/2-lts/step4.md +++ b/tutorials/katacoda/thanos/2-lts/step4.md @@ -25,7 +25,7 @@ Click below snippet to start the Compactor. docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name thanos-compact \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ diff --git a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh index 707a6abd4d..b611da79ed 100644 --- a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh +++ b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.22.2 -docker pull quay.io/thanos/thanos:v0.19.0-rc.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.1 docker pull yannrobert/docker-nginx diff --git a/tutorials/katacoda/thanos/6-query-caching/step1.md b/tutorials/katacoda/thanos/6-query-caching/step1.md index 9ebc4431a7..cc2e6e3b6f 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step1.md +++ b/tutorials/katacoda/thanos/6-query-caching/step1.md @@ -103,7 +103,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \ --name prometheus-sidecar"${i}" \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address=0.0.0.0:1909"${i}" \ --grpc-address=0.0.0.0:1919"${i}" \ @@ -129,7 +129,7 @@ And now, let's deploy Thanos Querier to have a global overview on our services. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:10912 \ --grpc-address 0.0.0.0:10901 \ diff --git a/tutorials/katacoda/thanos/6-query-caching/step2.md b/tutorials/katacoda/thanos/6-query-caching/step2.md index db9309e1c7..1b521ca2eb 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step2.md +++ b/tutorials/katacoda/thanos/6-query-caching/step2.md @@ -62,7 +62,7 @@ And deploy Query Frontend: docker run -d --net=host --rm \ -v $(pwd)/frontend.yml:/etc/thanos/frontend.yml \ --name query-frontend \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query-frontend \ --http-address 0.0.0.0:20902 \ --query-frontend.compress-responses \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh index 35b9ecd2af..2c1a02a1cd 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh +++ b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.1 docker pull quay.io/thanos/prom-label-proxy:v0.3.0-rc.0-ext1 docker pull caddy:2.2.1 diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md index fc34680033..4c1979851f 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md @@ -88,7 +88,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_fruit.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-fruit \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -120,7 +120,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -152,7 +152,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus1_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-01-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -170,7 +170,7 @@ Fruit: ``` docker run -d --net=host --rm \ --name querier-fruit \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:29091 \ --grpc-address 0.0.0.0:29191 \ @@ -183,7 +183,7 @@ Veggie: ``` docker run -d --net=host --rm \ --name querier-veggie \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:29092 \ --grpc-address 0.0.0.0:29192 \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md index ef07b7ea4d..3e3ed83628 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md @@ -11,7 +11,7 @@ docker stop querier-fruit && docker stop querier-veggie ``` docker run -d --net=host --rm \ --name querier-multi \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:29090 \ --grpc-address 0.0.0.0:29190 \ diff --git a/tutorials/katacoda/thanos/x-playground/courseBase.sh b/tutorials/katacoda/thanos/x-playground/courseBase.sh index 48f1878903..f5db1b0183 100644 --- a/tutorials/katacoda/thanos/x-playground/courseBase.sh +++ b/tutorials/katacoda/thanos/x-playground/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.0 +docker pull quay.io/thanos/thanos:v0.19.0-rc.1 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z diff --git a/tutorials/katacoda/thanos/x-playground/step1.md b/tutorials/katacoda/thanos/x-playground/step1.md index c8f128d556..44883dd871 100644 --- a/tutorials/katacoda/thanos/x-playground/step1.md +++ b/tutorials/katacoda/thanos/x-playground/step1.md @@ -169,7 +169,7 @@ docker run -d --net=host --rm \ ### Step1: Sidecar ``` -docker run -it --rm quay.io/thanos/thanos:v0.19.0-rc.0 --help +docker run -it --rm quay.io/thanos/thanos:v0.19.0-rc.1 --help ```{{execute}} @@ -180,7 +180,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -195,7 +195,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -210,7 +210,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --http-address 0.0.0.0:19093 \ --grpc-address 0.0.0.0:19193 \ @@ -223,7 +223,7 @@ docker run -d --net=host --rm \ ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ diff --git a/tutorials/katacoda/thanos/x-playground/step2.md b/tutorials/katacoda/thanos/x-playground/step2.md index 453762d058..e8956fa1e1 100644 --- a/tutorials/katacoda/thanos/x-playground/step2.md +++ b/tutorials/katacoda/thanos/x-playground/step2.md @@ -65,7 +65,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0:/prometheus \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -85,7 +85,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1:/prometheus \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -105,7 +105,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0:/prometheus \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -130,7 +130,7 @@ Let's run Store Gateway server: docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19094 \ @@ -143,7 +143,7 @@ docker run -d --net=host --rm \ docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ @@ -162,7 +162,7 @@ Visit https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.co docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name compactor \ - quay.io/thanos/thanos:v0.19.0-rc.0 \ + quay.io/thanos/thanos:v0.19.0-rc.1 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ From 38d78b5a4f272ea4cf0c21b3fad38b87d86dc4ec Mon Sep 17 00:00:00 2001 From: Ben Ye Date: Tue, 9 Mar 2021 09:11:19 -0500 Subject: [PATCH 08/15] tools: Fix partial and empty matchers in rewrite (#3891) * Fix partial and empty matchers match Signed-off-by: yeya24 * add testcase for non-equal matchers Signed-off-by: yeya24 --- pkg/compactv2/compactor_test.go | 91 +++++++++++++++++++++++++++++++++ pkg/compactv2/modifiers.go | 5 +- 2 files changed, 92 insertions(+), 4 deletions(-) diff --git a/pkg/compactv2/compactor_test.go b/pkg/compactv2/compactor_test.go index c1af292d1b..e5c979a053 100644 --- a/pkg/compactv2/compactor_test.go +++ b/pkg/compactv2/compactor_test.go @@ -306,6 +306,97 @@ func TestCompactor_WriteSeries_e2e(t *testing.T) { NumChunks: 2, }, }, + { + name: "1 blocks + delete modifier. For deletion request, full match is required. Delete the first two series", + input: [][]seriesSamples{ + { + {lset: labels.Labels{{Name: "a", Value: "1"}, {Name: "b", Value: "2"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "1"}, {Name: "b", Value: "2"}, {Name: "foo", Value: "bar"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "b", Value: "2"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "c", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + }, + }, + modifiers: []Modifier{WithDeletionModifier( + metadata.DeletionRequest{ + Matchers: []*labels.Matcher{ + labels.MustNewMatcher(labels.MatchEqual, "a", "1"), + labels.MustNewMatcher(labels.MatchEqual, "b", "2"), + }, + })}, + expected: []seriesSamples{ + {lset: labels.Labels{{Name: "a", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "b", Value: "2"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "c", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}, {10, 12}, {11, 11}, {20, 20}}}}, + }, + expectedChanges: "Deleted {a=\"1\", b=\"2\"} [{0 20}]\nDeleted {a=\"1\", b=\"2\", foo=\"bar\"} [{0 20}]\n", + expectedStats: tsdb.BlockStats{ + NumSamples: 18, + NumSeries: 3, + NumChunks: 3, + }, + }, + { + name: "1 blocks + delete modifier. Deletion request contains non-equal matchers.", + input: [][]seriesSamples{ + { + {lset: labels.Labels{{Name: "a", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "2"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "2"}, {Name: "foo", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "2"}, {Name: "foo", Value: "bar"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "3"}, {Name: "foo", Value: "baz"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "foo", Value: "bat"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + + // Label a is present but with an empty value. + {lset: labels.Labels{{Name: "a", Value: ""}, {Name: "foo", Value: "bat"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + // Series with unrelated labels. + {lset: labels.Labels{{Name: "c", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + }, + }, + modifiers: []Modifier{WithDeletionModifier( + metadata.DeletionRequest{ + Matchers: []*labels.Matcher{ + labels.MustNewMatcher(labels.MatchNotEqual, "a", "1"), + labels.MustNewMatcher(labels.MatchRegexp, "foo", "^ba.$"), + }, + })}, + expected: []seriesSamples{ + {lset: labels.Labels{{Name: "a", Value: ""}, {Name: "foo", Value: "bat"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "2"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "a", Value: "2"}, {Name: "foo", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "c", Value: "1"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + {lset: labels.Labels{{Name: "foo", Value: "bat"}}, + chunks: [][]sample{{{0, 0}, {1, 1}, {2, 2}}, {{10, 11}, {11, 11}, {20, 20}}}}, + }, + expectedChanges: "Deleted {a=\"2\", foo=\"bar\"} [{0 20}]\nDeleted {a=\"3\", foo=\"baz\"} [{0 20}]\n", + expectedStats: tsdb.BlockStats{ + NumSamples: 36, + NumSeries: 6, + NumChunks: 12, + }, + }, } { t.Run(tcase.name, func(t *testing.T) { tmpDir, err := ioutil.TempDir("", "test-series-writer") diff --git a/pkg/compactv2/modifiers.go b/pkg/compactv2/modifiers.go index 9d521c0fd2..aef4ec065e 100644 --- a/pkg/compactv2/modifiers.go +++ b/pkg/compactv2/modifiers.go @@ -62,12 +62,9 @@ SeriesLoop: for _, deletions := range d.d.deletions { for _, m := range deletions.Matchers { v := lbls.Get(m.Name) - if v == "" { - continue - } // Only if all matchers in the deletion request are matched can we proceed to deletion. - if !m.Matches(v) { + if v == "" || !m.Matches(v) { continue DeletionsLoop } } From b446faec5ed5c415ab7574d77739ce55fb6a1c4f Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Sat, 20 Mar 2021 16:23:35 +0100 Subject: [PATCH 09/15] v0.19.0 patch: Added receive benchmark; Fixed Receiver excessive mem usage introduced in 0.17 (#3943) * Added receive benchmark, baseline. ``` goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK-12 22260 1550152 ns/op 1380340 B/op 6093 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors-12 6619 6430408 ns/op 4522487 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK-12 2695 17208794 ns/op 15072963 B/op 60441 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors-12 474 72533286 ns/op 46396932 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 270 137050518 ns/op 226595379 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 21 1616025443 ns/op 698724321 B/op 408 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka * Copy labels. ``` GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchmem -test.benchtime=30s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 25887 1537262 ns/op 1380023 B/op 6092 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4237 7547968 ns/op 4522583 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 2205 16513380 ns/op 15071092 B/op 60420 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 525 67278233 ns/op 46396645 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 285 148049189 ns/op 226596168 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 20 1731361499 ns/op 698722550 B/op 401 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka * Addded bench., Signed-off-by: Bartlomiej Plotka * Fix. Signed-off-by: Bartlomiej Plotka * Improved API. Signed-off-by: Bartlomiej Plotka * Changelog. Signed-off-by: Bartlomiej Plotka * Addressed Lucas comments. Signed-off-by: Bartlomiej Plotka --- CHANGELOG.md | 3 +- pkg/receive/handler.go | 6 +- pkg/receive/handler_test.go | 256 ++++++++++++++++++++++++++++++-- pkg/receive/writer.go | 14 +- pkg/store/labelpb/label.go | 13 +- pkg/store/labelpb/label_test.go | 35 +++++ 6 files changed, 308 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c0aa5e5579..980959d992 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ NOTE: As semantic versioning states all 0.y.z releases can contain breaking chan We use _breaking :warning:_ to mark changes that are not backward compatible (relates only to v0.y.z releases.) -## [v0.19.0-rc.1](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.1) - 2021.03.09 +## [v0.19.0-rc.1](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.2) - 2021.03.22 ### Added @@ -36,6 +36,7 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re - [#3815](https://github.com/thanos-io/thanos/pull/3815) Receive: Improve handling of empty time series from clients - [#3795](https://github.com/thanos-io/thanos/pull/3795) s3: A truncated "get object" response is reported as error. - [#3899](https://github.com/thanos-io/thanos/pull/3899) Receive: Correct the inference of client gRPC configuration. +- [#3943](https://github.com/thanos-io/thanos/pull/3943): Receive: Fixed memory regression introduced in v0.17.0. ### Changed diff --git a/pkg/receive/handler.go b/pkg/receive/handler.go index 0337fe495f..693bd74336 100644 --- a/pkg/receive/handler.go +++ b/pkg/receive/handler.go @@ -277,6 +277,7 @@ func (h *Handler) receiveHTTP(w http.ResponseWriter, r *http.Request) { span, ctx := tracing.StartSpan(r.Context(), "receive_http") defer span.Finish() + // TODO(bwplotka): Optimize readAll https://github.com/thanos-io/thanos/pull/3334/files. compressed, err := ioutil.ReadAll(r.Body) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) @@ -290,6 +291,9 @@ func (h *Handler) receiveHTTP(w http.ResponseWriter, r *http.Request) { return } + // NOTE: Due to zero copy ZLabels, Labels used from WriteRequests keeps memory + // from the whole request. Ensure that we always copy those when we want to + // store them for longer time. var wreq prompb.WriteRequest if err := proto.Unmarshal(reqBuf, &wreq); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) @@ -310,7 +314,7 @@ func (h *Handler) receiveHTTP(w http.ResponseWriter, r *http.Request) { tenant = h.options.DefaultTenantID } - // exit early if the request contained no data + // Exit early if the request contained no data. if len(wreq.Timeseries) == 0 { level.Info(h.logger).Log("msg", "empty timeseries from client", "tenant", tenant) return diff --git a/pkg/receive/handler_test.go b/pkg/receive/handler_test.go index 70637d49fd..8e4287ffd5 100644 --- a/pkg/receive/handler_test.go +++ b/pkg/receive/handler_test.go @@ -7,9 +7,16 @@ import ( "bytes" "context" "fmt" + "io/ioutil" + "math" "math/rand" "net/http" "net/http/httptest" + "os" + "path/filepath" + "runtime" + "runtime/pprof" + "strings" "sync" "testing" "time" @@ -18,9 +25,12 @@ import ( "github.com/gogo/protobuf/proto" "github.com/golang/snappy" "github.com/pkg/errors" + "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/prometheus/pkg/labels" "github.com/prometheus/prometheus/storage" "github.com/prometheus/prometheus/tsdb" + "github.com/thanos-io/thanos/pkg/block/metadata" + "github.com/thanos-io/thanos/pkg/runutil" "google.golang.org/grpc" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" @@ -175,13 +185,11 @@ func TestDetermineWriteErrorCause(t *testing.T) { } } -func newHandlerHashring(appendables []*fakeAppendable, replicationFactor uint64) ([]*Handler, Hashring) { - cfg := []HashringConfig{ - { - Hashring: "test", - }, - } - var handlers []*Handler +func newTestHandlerHashring(appendables []*fakeAppendable, replicationFactor uint64) ([]*Handler, Hashring) { + var ( + cfg = []HashringConfig{{Hashring: "test"}} + handlers []*Handler + ) // create a fake peer group where we manually fill the cache with fake addresses pointed to our handlers // This removes the network from the tests and creates a more consistent testing harness. peers := &peerGroup{ @@ -511,7 +519,7 @@ func TestReceiveQuorum(t *testing.T) { }, } { t.Run(tc.name, func(t *testing.T) { - handlers, hashring := newHandlerHashring(tc.appendables, tc.replicationFactor) + handlers, hashring := newTestHandlerHashring(tc.appendables, tc.replicationFactor) tenant := "test" // Test from the point of view of every node // so that we know status code does not depend @@ -850,7 +858,7 @@ func TestReceiveWithConsistencyDelay(t *testing.T) { // to see all requests completing all the time, since we're using local // network we are not expecting anything to go wrong with these. t.Run(tc.name, func(t *testing.T) { - handlers, hashring := newHandlerHashring(tc.appendables, tc.replicationFactor) + handlers, hashring := newTestHandlerHashring(tc.appendables, tc.replicationFactor) tenant := "test" // Test from the point of view of every node // so that we know status code does not depend @@ -957,3 +965,233 @@ type fakeRemoteWriteGRPCServer struct { func (f *fakeRemoteWriteGRPCServer) RemoteWrite(ctx context.Context, in *storepb.WriteRequest, opts ...grpc.CallOption) (*storepb.WriteResponse, error) { return f.h.RemoteWrite(ctx, in) } + +func BenchmarkHandlerReceiveHTTP(b *testing.B) { + benchmarkHandlerMultiTSDBReceiveRemoteWrite(testutil.NewTB(b)) +} + +func TestHandlerReceiveHTTP(t *testing.T) { + benchmarkHandlerMultiTSDBReceiveRemoteWrite(testutil.NewTB(t)) +} + +// tsOverrideTenantStorage is storage that overrides timestamp to make it have consistent interval. +type tsOverrideTenantStorage struct { + TenantStorage + + interval int64 +} + +func (s *tsOverrideTenantStorage) TenantAppendable(tenant string) (Appendable, error) { + a, err := s.TenantStorage.TenantAppendable(tenant) + return &tsOverrideAppendable{Appendable: a, interval: s.interval}, err +} + +type tsOverrideAppendable struct { + Appendable + + interval int64 +} + +func (a *tsOverrideAppendable) Appender(ctx context.Context) (storage.Appender, error) { + ret, err := a.Appendable.Appender(ctx) + return &tsOverrideAppender{Appender: ret, interval: a.interval}, err +} + +type tsOverrideAppender struct { + storage.Appender + + interval int64 +} + +var cnt int64 + +func (a *tsOverrideAppender) Add(l labels.Labels, _ int64, v float64) (uint64, error) { + cnt += a.interval + return a.Appender.Add(l, cnt, v) +} + +func (a *tsOverrideAppender) AddFast(ref uint64, _ int64, v float64) error { + cnt += a.interval + return a.Appender.AddFast(ref, cnt, v) +} + +// serializeSeriesWithOneSample returns marshaled and compressed remote write requests like it would +// be send to Thanos receive. +// It has one sample and allow passing multiple series, in same manner as typical Prometheus would batch it. +func serializeSeriesWithOneSample(t testing.TB, series [][]labelpb.ZLabel) []byte { + r := &prompb.WriteRequest{Timeseries: make([]prompb.TimeSeries, 0, len(series))} + + for _, s := range series { + r.Timeseries = append(r.Timeseries, prompb.TimeSeries{ + Labels: s, + // Timestamp does not matter, it will be overridden. + Samples: []prompb.Sample{{Value: math.MaxFloat64, Timestamp: math.MinInt64}}, + }) + } + body, err := proto.Marshal(r) + testutil.Ok(t, err) + return snappy.Encode(nil, body) +} + +func benchmarkHandlerMultiTSDBReceiveRemoteWrite(b testutil.TB) { + dir, err := ioutil.TempDir("", "test_receive") + testutil.Ok(b, err) + defer func() { testutil.Ok(b, os.RemoveAll(dir)) }() + + handlers, _ := newTestHandlerHashring([]*fakeAppendable{nil}, 1) + handler := handlers[0] + + reg := prometheus.NewRegistry() + + logger := log.NewNopLogger() + m := NewMultiTSDB( + dir, logger, reg, &tsdb.Options{ + MinBlockDuration: int64(2 * time.Hour / time.Millisecond), + MaxBlockDuration: int64(2 * time.Hour / time.Millisecond), + RetentionDuration: int64(6 * time.Hour / time.Millisecond), + NoLockfile: true, + StripeSize: 1, // Disable stripe pre allocation so we can have clear profiles. + }, + labels.FromStrings("replica", "01"), + "tenant_id", + nil, + false, + metadata.NoneFunc, + ) + defer func() { testutil.Ok(b, m.Close()) }() + handler.writer = NewWriter(logger, m) + + testutil.Ok(b, m.Flush()) + testutil.Ok(b, m.Open()) + + for _, tcase := range []struct { + name string + writeRequest []byte + }{ + { + name: "typical labels under 1KB, 500 of them", + writeRequest: serializeSeriesWithOneSample(b, func() [][]labelpb.ZLabel { + series := make([][]labelpb.ZLabel, 500) + for s := 0; s < len(series); s++ { + lbls := make([]labelpb.ZLabel, 10) + for i := 0; i < len(lbls); i++ { + // Label ~20B name, 50B value. + lbls[i] = labelpb.ZLabel{Name: fmt.Sprintf("abcdefghijabcdefghijabcdefghij%d", i), Value: fmt.Sprintf("abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij%d", i)} + } + series[s] = lbls + } + return series + }()), + }, + { + name: "typical labels under 1KB, 5000 of them", + writeRequest: serializeSeriesWithOneSample(b, func() [][]labelpb.ZLabel { + series := make([][]labelpb.ZLabel, 5000) + for s := 0; s < len(series); s++ { + lbls := make([]labelpb.ZLabel, 10) + for i := 0; i < len(lbls); i++ { + // Label ~20B name, 50B value. + lbls[i] = labelpb.ZLabel{Name: fmt.Sprintf("abcdefghijabcdefghijabcdefghij%d", i), Value: fmt.Sprintf("abcdefghijabcdefghijabcdefghijabcdefghijabcdefghij%d", i)} + } + series[s] = lbls + } + return series + }()), + }, + { + name: "extremely large label value 10MB, 10 of them", + writeRequest: serializeSeriesWithOneSample(b, func() [][]labelpb.ZLabel { + series := make([][]labelpb.ZLabel, 10) + for s := 0; s < len(series); s++ { + lbl := &strings.Builder{} + lbl.Grow(1024 * 1024 * 10) // 10MB. + word := "abcdefghij" + for i := 0; i < lbl.Cap()/len(word); i++ { + _, _ = lbl.WriteString(word) + } + series[s] = []labelpb.ZLabel{{Name: "__name__", Value: lbl.String()}} + } + return series + }()), + }, + } { + b.Run(tcase.name, func(b testutil.TB) { + handler.options.DefaultTenantID = fmt.Sprintf("%v-ok", tcase.name) + handler.writer.multiTSDB = &tsOverrideTenantStorage{TenantStorage: m, interval: 1} + + // It takes time to create new tenant, wait for it. + { + app, err := m.TenantAppendable(handler.options.DefaultTenantID) + testutil.Ok(b, err) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + testutil.Ok(b, runutil.Retry(1*time.Second, ctx.Done(), func() error { + _, err = app.Appender(ctx) + return err + })) + } + + b.Run("OK", func(b testutil.TB) { + n := b.N() + b.ResetTimer() + for i := 0; i < n; i++ { + r := httptest.NewRecorder() + handler.receiveHTTP(r, &http.Request{ContentLength: int64(len(tcase.writeRequest)), Body: ioutil.NopCloser(bytes.NewReader(tcase.writeRequest))}) + testutil.Equals(b, http.StatusOK, r.Code, "got non 200 error: %v", r.Body.String()) + } + }) + + handler.options.DefaultTenantID = fmt.Sprintf("%v-conflicting", tcase.name) + handler.writer.multiTSDB = &tsOverrideTenantStorage{TenantStorage: m, interval: -1} // Timestamp can't go down, which will cause conflict error. + + // It takes time to create new tenant, wait for it. + { + app, err := m.TenantAppendable(handler.options.DefaultTenantID) + testutil.Ok(b, err) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + + testutil.Ok(b, runutil.Retry(1*time.Second, ctx.Done(), func() error { + _, err = app.Appender(ctx) + return err + })) + } + + // First request should be fine, since we don't change timestamp, rest is wrong. + r := httptest.NewRecorder() + handler.receiveHTTP(r, &http.Request{ContentLength: int64(len(tcase.writeRequest)), Body: ioutil.NopCloser(bytes.NewReader(tcase.writeRequest))}) + testutil.Equals(b, http.StatusOK, r.Code, "got non 200 error: %v", r.Body.String()) + + b.Run("conflict errors", func(b testutil.TB) { + n := b.N() + b.ResetTimer() + for i := 0; i < n; i++ { + r := httptest.NewRecorder() + handler.receiveHTTP(r, &http.Request{ContentLength: int64(len(tcase.writeRequest)), Body: ioutil.NopCloser(bytes.NewReader(tcase.writeRequest))}) + testutil.Equals(b, http.StatusConflict, r.Code, "%v", i) + } + }) + }) + } + + runtime.GC() + // Take snapshot at the end to reveal how much memory we keep in TSDB. + testutil.Ok(b, Heap("../../")) + +} + +func Heap(dir string) (err error) { + if err := os.MkdirAll(dir, os.ModePerm); err != nil { + return err + } + + f, err := os.Create(filepath.Join(dir, "mem.pprof")) + if err != nil { + return err + } + defer runutil.CloseWithErrCapture(&err, f, "close") + return pprof.WriteHeapProfile(f) +} diff --git a/pkg/receive/writer.go b/pkg/receive/writer.go index 8fe1cfe68c..0256040262 100644 --- a/pkg/receive/writer.go +++ b/pkg/receive/writer.go @@ -13,6 +13,7 @@ import ( "github.com/prometheus/prometheus/pkg/labels" "github.com/prometheus/prometheus/storage" "github.com/prometheus/prometheus/tsdb" + "github.com/thanos-io/thanos/pkg/store/labelpb" "github.com/thanos-io/thanos/pkg/errutil" "github.com/thanos-io/thanos/pkg/store/storepb/prompb" @@ -61,13 +62,12 @@ func (r *Writer) Write(ctx context.Context, tenantID string, wreq *prompb.WriteR var errs errutil.MultiError for _, t := range wreq.Timeseries { - lset := make(labels.Labels, len(t.Labels)) - for j := range t.Labels { - lset[j] = labels.Label{ - Name: t.Labels[j].Name, - Value: t.Labels[j].Value, - } - } + // Copy labels so we allocate memory only for labels, nothing else. + labelpb.ReAllocZLabelsStrings(&t.Labels) + + // TODO(bwplotka): Use improvement https://github.com/prometheus/prometheus/pull/8600, so we do that only when + // we need it (when we store labels for longer). + lset := labelpb.ZLabelsToPromLabels(t.Labels) // Append as many valid samples as possible, but keep track of the errors. for _, s := range t.Samples { diff --git a/pkg/store/labelpb/label.go b/pkg/store/labelpb/label.go index 5712cd912f..99083f82ab 100644 --- a/pkg/store/labelpb/label.go +++ b/pkg/store/labelpb/label.go @@ -34,10 +34,21 @@ func ZLabelsFromPromLabels(lset labels.Labels) []ZLabel { // ZLabelsToPromLabels convert slice of labelpb.ZLabel to Prometheus labels in type unsafe manner. // It reuses the same memory. Caller should abort using passed []ZLabel. +// NOTE: Use with care. ZLabels holds memory from the whole protobuf unmarshal, so the returned +// Prometheus Labels will hold this memory as well. func ZLabelsToPromLabels(lset []ZLabel) labels.Labels { return *(*labels.Labels)(unsafe.Pointer(&lset)) } +// ReAllocZLabelsStrings re-allocates all underlying bytes for string, detaching it from bigger memory pool. +func ReAllocZLabelsStrings(lset *[]ZLabel) { + for j, l := range *lset { + // NOTE: This trick converts from string to byte without copy, but copy when creating string. + (*lset)[j].Name = string(noAllocBytes(l.Name)) + (*lset)[j].Value = string(noAllocBytes(l.Value)) + } +} + // LabelsFromPromLabels converts Prometheus labels to slice of labelpb.ZLabel in type unsafe manner. // It reuses the same memory. Caller should abort using passed labels.Labels. func LabelsFromPromLabels(lset labels.Labels) []Label { @@ -61,7 +72,7 @@ func ZLabelSetsToPromLabelSets(lss ...ZLabelSet) []labels.Labels { // ZLabel is a Label (also easily transformable to Prometheus labels.Labels) that can be unmarshalled from protobuf // reusing the same memory address for string bytes. -// NOTE: While unmarshal use exactly same bytes that were allocated for protobuf, this will mean that *whole* protobuf +// NOTE: While unmarshalling it uses exactly same bytes that were allocated for protobuf. This mean that *whole* protobuf // bytes will be not GC-ed as long as ZLabels are referenced somewhere. Use it carefully, only for short living // protobuf message processing. type ZLabel Label diff --git a/pkg/store/labelpb/label_test.go b/pkg/store/labelpb/label_test.go index a6227dfd27..ee80b5ad47 100644 --- a/pkg/store/labelpb/label_test.go +++ b/pkg/store/labelpb/label_test.go @@ -174,6 +174,41 @@ func BenchmarkZLabelsMarshalUnmarshal(b *testing.B) { }) } +var ret labels.Labels + +func BenchmarkTransformWithAndWithoutCopy(b *testing.B) { + const ( + fmtLbl = "%07daaaaaaaaaabbbbbbbbbbccccccccccdddddddddd" + num = 1000000 + ) + + b.Run("ZLabelsToPromLabels", func(b *testing.B) { + b.ReportAllocs() + lbls := make([]ZLabel, num) + for i := 0; i < num; i++ { + lbls[i] = ZLabel{Name: fmt.Sprintf(fmtLbl, i), Value: fmt.Sprintf(fmtLbl, i)} + } + b.ResetTimer() + + for i := 0; i < b.N; i++ { + ret = ZLabelsToPromLabels(lbls) + } + }) + b.Run("ZLabelsToPromLabelsWithRealloc", func(b *testing.B) { + b.ReportAllocs() + lbls := make([]ZLabel, num) + for i := 0; i < num; i++ { + lbls[i] = ZLabel{Name: fmt.Sprintf(fmtLbl, i), Value: fmt.Sprintf(fmtLbl, i)} + } + b.ResetTimer() + + for i := 0; i < b.N; i++ { + ReAllocZLabelsStrings(&lbls) + ret = ZLabelsToPromLabels(lbls) + } + }) +} + func TestSortZLabelSets(t *testing.T) { expectedResult := ZLabelSets{ { From 1dce9969ea348b6faeec5d479b8981b87530da20 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Giedrius=20Statkevi=C4=8Dius?= Date: Wed, 24 Mar 2021 17:17:54 +0200 Subject: [PATCH 10/15] compact: clean up directories thoroughly (#3869) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * compact: clean up directories properly I couldn't stop thinking about this code for some reason and I have figured that I had missed one case in #3031. We need to also clean up the directories in compaction groups. A compaction could fail leaving some new directory with a random ULID on the disk. Before attempting to do another compaction loop, we need to remove it as well because the compaction process always produces a new, unique directory. Signed-off-by: Giedrius Statkevičius * Remove all non expected dirs. Signed-off-by: Bartlomiej Plotka * Addressed comment. Signed-off-by: Bartlomiej Plotka Co-authored-by: Bartlomiej Plotka --- pkg/compact/compact.go | 4 +++- pkg/runutil/runutil.go | 31 ++++++++++++++++++++++++------- pkg/runutil/runutil_test.go | 34 +++++++++++++++++++++------------- test/e2e/compact_test.go | 24 +++++++++++++++++++++++- 4 files changed, 71 insertions(+), 22 deletions(-) diff --git a/pkg/compact/compact.go b/pkg/compact/compact.go index 7b0c6c9084..23ee8c39b0 100644 --- a/pkg/compact/compact.go +++ b/pkg/compact/compact.go @@ -957,7 +957,9 @@ func (c *BucketCompactor) Compact(ctx context.Context) (rerr error) { ignoreDirs := []string{} for _, gr := range groups { - ignoreDirs = append(ignoreDirs, gr.Key()) + for _, grID := range gr.IDs() { + ignoreDirs = append(ignoreDirs, filepath.Join(gr.Key(), grID.String())) + } } if err := runutil.DeleteAll(c.compactDir, ignoreDirs...); err != nil { diff --git a/pkg/runutil/runutil.go b/pkg/runutil/runutil.go index 97171bce82..f726eb25c3 100644 --- a/pkg/runutil/runutil.go +++ b/pkg/runutil/runutil.go @@ -55,6 +55,7 @@ import ( "io/ioutil" "os" "path/filepath" + "strings" "time" "github.com/go-kit/kit/log" @@ -162,6 +163,7 @@ func ExhaustCloseWithErrCapture(err *error, r io.ReadCloser, format string, a .. // DeleteAll deletes all files and directories inside the given // dir except for the ignoreDirs directories. +// NOTE: DeleteAll is not idempotent. func DeleteAll(dir string, ignoreDirs ...string) error { entries, err := ioutil.ReadDir(dir) if err != nil { @@ -169,6 +171,7 @@ func DeleteAll(dir string, ignoreDirs ...string) error { } var groupErrs errutil.MultiError + var matchingIgnores []string for _, d := range entries { if !d.IsDir() { if err := os.RemoveAll(filepath.Join(dir, d.Name())); err != nil { @@ -177,22 +180,36 @@ func DeleteAll(dir string, ignoreDirs ...string) error { continue } - var found bool - for _, id := range ignoreDirs { - if id == d.Name() { - found = true - break + // ignoreDirs might be multi-directory paths. + matchingIgnores = matchingIgnores[:0] + ignore := false + for _, ignoreDir := range ignoreDirs { + id := strings.Split(ignoreDir, "/") + if id[0] == d.Name() { + if len(id) == 1 { + ignore = true + break + } + matchingIgnores = append(matchingIgnores, filepath.Join(id[1:]...)) } } - if !found { + if ignore { + continue + } + + if len(matchingIgnores) == 0 { if err := os.RemoveAll(filepath.Join(dir, d.Name())); err != nil { groupErrs.Add(err) } + continue + } + if err := DeleteAll(filepath.Join(dir, d.Name()), matchingIgnores...); err != nil { + groupErrs.Add(err) } } - if groupErrs != nil { + if groupErrs.Err() != nil { return errors.Wrap(groupErrs.Err(), "delete file/dir") } return nil diff --git a/pkg/runutil/runutil_test.go b/pkg/runutil/runutil_test.go index 908a3398a3..a50efbd356 100644 --- a/pkg/runutil/runutil_test.go +++ b/pkg/runutil/runutil_test.go @@ -125,35 +125,43 @@ func TestCloseMoreThanOnce(t *testing.T) { testutil.Equals(t, true, lc.WasCalled) } -func TestClearsDirectoriesFilesProperly(t *testing.T) { +func TestDeleteAll(t *testing.T) { dir, err := ioutil.TempDir("", "example") testutil.Ok(t, err) t.Cleanup(func() { - os.RemoveAll(dir) + testutil.Ok(t, os.RemoveAll(dir)) }) - f, err := os.Create(filepath.Join(dir, "test123")) + f, err := os.Create(filepath.Join(dir, "file1")) testutil.Ok(t, err) testutil.Ok(t, f.Close()) - testutil.Ok(t, os.MkdirAll(filepath.Join(dir, "01EHBQRN4RF0HSRR1772KW0TN8"), os.ModePerm)) - testutil.Ok(t, os.MkdirAll(filepath.Join(dir, "01EHBQRN4RF0HSRR1772KW1TN8"), os.ModePerm)) - f, err = os.Create(filepath.Join(dir, "01EHBQRN4RF0HSRR1772KW0TN9")) + testutil.Ok(t, os.MkdirAll(filepath.Join(dir, "a"), os.ModePerm)) + testutil.Ok(t, os.MkdirAll(filepath.Join(dir, "b"), os.ModePerm)) + testutil.Ok(t, os.MkdirAll(filepath.Join(dir, "c", "innerc"), os.ModePerm)) + f, err = os.Create(filepath.Join(dir, "a", "file2")) + testutil.Ok(t, err) + testutil.Ok(t, f.Close()) + f, err = os.Create(filepath.Join(dir, "c", "file3")) testutil.Ok(t, err) testutil.Ok(t, f.Close()) - testutil.Ok(t, DeleteAll(dir, "01EHBQRN4RF0HSRR1772KW0TN9", "01EHBQRN4RF0HSRR1772KW0TN8")) + testutil.Ok(t, DeleteAll(dir, "file1", "a", filepath.Join("c", "innerc"))) - _, err = os.Stat(filepath.Join(dir, "test123")) + // Deleted. + _, err = os.Stat(filepath.Join(dir, "file1")) testutil.Assert(t, os.IsNotExist(err)) - - _, err = os.Stat(filepath.Join(dir, "01EHBQRN4RF0HSRR1772KW0TN9")) + _, err = os.Stat(filepath.Join(dir, "b/")) testutil.Assert(t, os.IsNotExist(err)) - - _, err = os.Stat(filepath.Join(dir, "01EHBQRN4RF0HSRR1772KW1TN8/")) + _, err = os.Stat(filepath.Join(dir, "file3")) testutil.Assert(t, os.IsNotExist(err)) - _, err = os.Stat(filepath.Join(dir, "01EHBQRN4RF0HSRR1772KW0TN8/")) + // Exists. + _, err = os.Stat(filepath.Join(dir, "a", "file2")) + testutil.Ok(t, err) + _, err = os.Stat(filepath.Join(dir, "a/")) + testutil.Ok(t, err) + _, err = os.Stat(filepath.Join(dir, "c", "innerc")) testutil.Ok(t, err) } diff --git a/test/e2e/compact_test.go b/test/e2e/compact_test.go index a1e206f08b..3697fedccf 100644 --- a/test/e2e/compact_test.go +++ b/test/e2e/compact_test.go @@ -539,8 +539,26 @@ func TestCompactWithStoreGateway(t *testing.T) { // No replica label with overlaps should halt compactor. This test is sequential // because we do not want two Thanos Compact instances deleting the same partially - // uploaded blocks and blocks with deletion marks. + // uploaded blocks and blocks with deletion marks. We also check that Thanos Compactor + // deletes directories inside of a compaction group that do not belong there. { + // Precreate a directory. It should be deleted. + // In a hypothetical scenario, the directory could be a left-over from + // a compaction that had crashed. + p := filepath.Join(s.SharedDir(), "data", "compact", "expect-to-halt", "compact") + + testutil.Assert(t, len(blocksWithHashes) > 0) + + m, err := block.DownloadMeta(ctx, logger, bkt, blocksWithHashes[0]) + testutil.Ok(t, err) + + randBlockDir := filepath.Join(p, compact.DefaultGroupKey(m.Thanos), "ITISAVERYRANDULIDFORTESTS0") + testutil.Ok(t, os.MkdirAll(randBlockDir, os.ModePerm)) + + f, err := os.Create(filepath.Join(randBlockDir, "index")) + testutil.Ok(t, err) + testutil.Ok(t, f.Close()) + c, err := e2ethanos.NewCompactor(s.SharedDir(), "expect-to-halt", svcConfig, nil) testutil.Ok(t, err) testutil.Ok(t, s.StartAndWaitReady(c)) @@ -578,6 +596,10 @@ func TestCompactWithStoreGateway(t *testing.T) { ensureGETStatusCode(t, http.StatusOK, "http://"+path.Join(c.HTTPEndpoint(), "loaded")) testutil.Ok(t, s.Stop(c)) + + _, err = os.Stat(randBlockDir) + testutil.NotOk(t, err) + testutil.Assert(t, os.IsNotExist(err)) } // Sequential because we want to check that Thanos Compactor does not From 700260940e9b61e9181cec9f18d99b42c35c5a5f Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 24 Mar 2021 19:18:29 +0100 Subject: [PATCH 11/15] Added benchmark, Moved minio-deps to fork without race fix we don't need. (#3968) Fixes: https://github.com/thanos-io/thanos/issues/3917 Long term fix: https://github.com/thanos-io/thanos/issues/3967 Signed-off-by: Bartlomiej Plotka --- docs/storage.md | 2 +- go.mod | 3 ++ go.sum | 6 ++-- pkg/objstore/s3/s3.go | 6 ++-- pkg/objstore/s3/s3_e2e_test.go | 55 ++++++++++++++++++++++++++++++++++ pkg/objstore/s3/s3_test.go | 2 +- test/e2e/e2ethanos/helpers.go | 2 +- 7 files changed, 65 insertions(+), 11 deletions(-) create mode 100644 pkg/objstore/s3/s3_e2e_test.go diff --git a/docs/storage.md b/docs/storage.md index 989767feb4..8b62d29067 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -92,7 +92,7 @@ config: trace: enable: false list_objects_version: "" - part_size: 134217728 + part_size: 67108864 sse_config: type: "" kms_key_id: "" diff --git a/go.mod b/go.mod index e3ce8199d3..926e451a46 100644 --- a/go.mod +++ b/go.mod @@ -76,6 +76,9 @@ replace ( github.com/bradfitz/gomemcache => github.com/themihai/gomemcache v0.0.0-20180902122335-24332e2d58ab // Update to v1.1.1 to make sure windows CI pass. github.com/elastic/go-sysinfo => github.com/elastic/go-sysinfo v1.1.1 + + // TODO: Remove this: https://github.com/thanos-io/thanos/issues/3967. + github.com/minio/minio-go/v7 => github.com/bwplotka/minio-go/v7 v7.0.11-0.20210324165441-f9927e5255a6 // Make sure Prometheus version is pinned as Prometheus semver does not include Go APIs. github.com/prometheus/prometheus => github.com/prometheus/prometheus v1.8.2-0.20210215121130-6f488061dfb4 github.com/sercand/kuberesolver => github.com/sercand/kuberesolver v2.4.0+incompatible diff --git a/go.sum b/go.sum index f530ac84c6..bb70670be0 100644 --- a/go.sum +++ b/go.sum @@ -195,6 +195,8 @@ github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnweb github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4= github.com/bmizerany/pat v0.0.0-20170815010413-6226ea591a40/go.mod h1:8rLXio+WjiTceGBHIoTvn60HIbs7Hm7bcHjyrSqYB9c= github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps= +github.com/bwplotka/minio-go/v7 v7.0.11-0.20210324165441-f9927e5255a6 h1:h9SZ0jmAKjtrZF6iZ77/jdXdHr+Usn29itI669SVRp4= +github.com/bwplotka/minio-go/v7 v7.0.11-0.20210324165441-f9927e5255a6/go.mod h1:td4gW1ldOsj1PbSNS+WYK43j+P1XVhX/8W8awaYlBFo= github.com/c-bata/go-prompt v0.2.2/go.mod h1:VzqtzE2ksDBcdln8G7mk2RX9QyGjH+OVqOCSiVIqS34= github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ= github.com/cenkalti/backoff v0.0.0-20181003080854-62661b46c409/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM= @@ -900,10 +902,6 @@ github.com/minio/md5-simd v1.1.0 h1:QPfiOqlZH+Cj9teu0t9b1nTBfPbyTl16Of5MeuShdK4= github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw= github.com/minio/minio-go/v6 v6.0.44/go.mod h1:qD0lajrGW49lKZLtXKtCB4X/qkMf0a5tBvN2PaZg7Gg= github.com/minio/minio-go/v6 v6.0.56/go.mod h1:KQMM+/44DSlSGSQWSfRrAZ12FVMmpWNuX37i2AX0jfI= -github.com/minio/minio-go/v7 v7.0.2 h1:P/7wFd4KrRBHVo7AKdcqO+9ReoS+XpMjfRFoE5quH0E= -github.com/minio/minio-go/v7 v7.0.2/go.mod h1:dJ80Mv2HeGkYLH1sqS/ksz07ON6csH3S6JUMSQ2zAns= -github.com/minio/minio-go/v7 v7.0.10 h1:1oUKe4EOPUEhw2qnPQaPsJ0lmVTYLFu03SiItauXs94= -github.com/minio/minio-go/v7 v7.0.10/go.mod h1:td4gW1ldOsj1PbSNS+WYK43j+P1XVhX/8W8awaYlBFo= github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU= github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc= diff --git a/pkg/objstore/s3/s3.go b/pkg/objstore/s3/s3.go index 8814412e89..6a490fd5eb 100644 --- a/pkg/objstore/s3/s3.go +++ b/pkg/objstore/s3/s3.go @@ -66,9 +66,7 @@ var DefaultConfig = Config{ MaxIdleConnsPerHost: 100, MaxConnsPerHost: 0, }, - // Minimum file size after which an HTTP multipart request should be used to upload objects to storage. - // Set to 128 MiB as in the minio client. - PartSize: 1024 * 1024 * 128, + PartSize: 1024 * 1024 * 64, // 64MB. } // Config stores the configuration for s3 bucket. @@ -85,6 +83,7 @@ type Config struct { TraceConfig TraceConfig `yaml:"trace"` ListObjectsVersion string `yaml:"list_objects_version"` // PartSize used for multipart upload. Only used if uploaded object size is known and larger than configured PartSize. + // NOTE we need to make sure this number does not produce more parts than 10 000. PartSize uint64 `yaml:"part_size"` SSEConfig SSEConfig `yaml:"sse_config"` } @@ -449,7 +448,6 @@ func (b *Bucket) Upload(ctx context.Context, name string, r io.Reader) error { size = -1 } - // partSize cannot be larger than object size. partSize := b.partSize if size < int64(partSize) { partSize = 0 diff --git a/pkg/objstore/s3/s3_e2e_test.go b/pkg/objstore/s3/s3_e2e_test.go new file mode 100644 index 0000000000..e837b9baa5 --- /dev/null +++ b/pkg/objstore/s3/s3_e2e_test.go @@ -0,0 +1,55 @@ +// Copyright (c) The Thanos Authors. +// Licensed under the Apache License 2.0. + +package s3_test + +import ( + "bytes" + "context" + "strings" + "testing" + + "github.com/cortexproject/cortex/integration/e2e" + e2edb "github.com/cortexproject/cortex/integration/e2e/db" + "github.com/go-kit/kit/log" + "github.com/thanos-io/thanos/pkg/objstore/s3" + "github.com/thanos-io/thanos/test/e2e/e2ethanos" + + "github.com/thanos-io/thanos/pkg/testutil" +) + +// Regression benchmark for https://github.com/thanos-io/thanos/issues/3917. +func BenchmarkUpload(b *testing.B) { + b.ReportAllocs() + ctx := context.Background() + + s, err := e2e.NewScenario("e2e_bench_mino_client") + testutil.Ok(b, err) + b.Cleanup(e2ethanos.CleanScenario(b, s)) + + const bucket = "test" + m := e2edb.NewMinio(8080, bucket) + testutil.Ok(b, s.StartAndWaitReady(m)) + + bkt, err := s3.NewBucketWithConfig(log.NewNopLogger(), s3.Config{ + Bucket: bucket, + AccessKey: e2edb.MinioAccessKey, + SecretKey: e2edb.MinioSecretKey, + Endpoint: m.HTTPEndpoint(), + Insecure: true, + }, "test-feed") + testutil.Ok(b, err) + + buf := bytes.Buffer{} + buf.Grow(1028 * 1028 * 100) // 100MB. + word := "abcdefghij" + for i := 0; i < buf.Cap()/len(word); i++ { + _, _ = buf.WriteString(word) + } + str := buf.String() + + b.ResetTimer() + for i := 0; i < b.N; i++ { + testutil.Ok(b, bkt.Upload(ctx, "test", strings.NewReader(str))) + } +} diff --git a/pkg/objstore/s3/s3_test.go b/pkg/objstore/s3/s3_test.go index 2cac107cae..b084473e26 100644 --- a/pkg/objstore/s3/s3_test.go +++ b/pkg/objstore/s3/s3_test.go @@ -208,7 +208,7 @@ http_config: cfg, err := parseConfig(input) testutil.Ok(t, err) - testutil.Assert(t, cfg.PartSize == 1024*1024*128, "when part size not set it should default to 128MiB") + testutil.Assert(t, cfg.PartSize == 1024*1024*64, "when part size not set it should default to 128MiB") input2 := []byte(`bucket: "bucket-name" endpoint: "s3-endpoint" diff --git a/test/e2e/e2ethanos/helpers.go b/test/e2e/e2ethanos/helpers.go index 714bdbb468..0bcd21c912 100644 --- a/test/e2e/e2ethanos/helpers.go +++ b/test/e2e/e2ethanos/helpers.go @@ -16,7 +16,7 @@ import ( "github.com/thanos-io/thanos/pkg/testutil" ) -func CleanScenario(t *testing.T, s *e2e.Scenario) func() { +func CleanScenario(t testing.TB, s *e2e.Scenario) func() { return func() { // Make sure Clean can properly delete everything. testutil.Ok(t, exec.Command("chmod", "-R", "777", s.SharedDir()).Run()) From 5620c0955104526dd9cfc7aa4481bfda9030e66c Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 24 Mar 2021 19:30:52 +0100 Subject: [PATCH 12/15] Cut v0.19.0-rc.2 (#3969) Signed-off-by: Bartlomiej Plotka --- CHANGELOG.md | 3 +-- VERSION | 2 +- tutorials/katacoda/thanos/1-globalview/courseBase.sh | 2 +- tutorials/katacoda/thanos/1-globalview/step2.md | 8 ++++---- tutorials/katacoda/thanos/1-globalview/step3.md | 2 +- tutorials/katacoda/thanos/2-lts/courseBase.sh | 2 +- tutorials/katacoda/thanos/2-lts/step1.md | 4 ++-- tutorials/katacoda/thanos/2-lts/step2.md | 2 +- tutorials/katacoda/thanos/2-lts/step3.md | 6 +++--- tutorials/katacoda/thanos/2-lts/step4.md | 2 +- .../katacoda/thanos/6-query-caching/courseBase.sh | 2 +- tutorials/katacoda/thanos/6-query-caching/step1.md | 4 ++-- tutorials/katacoda/thanos/6-query-caching/step2.md | 2 +- .../katacoda/thanos/7-multi-tenancy/courseBase.sh | 2 +- tutorials/katacoda/thanos/7-multi-tenancy/step1.md | 10 +++++----- tutorials/katacoda/thanos/7-multi-tenancy/step2.md | 2 +- tutorials/katacoda/thanos/x-playground/courseBase.sh | 2 +- tutorials/katacoda/thanos/x-playground/step1.md | 10 +++++----- tutorials/katacoda/thanos/x-playground/step2.md | 12 ++++++------ 19 files changed, 39 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 980959d992..e2c5e8aa1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ NOTE: As semantic versioning states all 0.y.z releases can contain breaking chan We use _breaking :warning:_ to mark changes that are not backward compatible (relates only to v0.y.z releases.) -## [v0.19.0-rc.1](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.2) - 2021.03.22 +## [v0.19.0-rc.2](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.2) - 2021.03.24 ### Added @@ -24,7 +24,6 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re - [#3686](https://github.com/thanos-io/thanos/pull/3686) Query/Sidecar: Added metric metadata API support. You can now configure you Querier to fetch Prometheus metrics metadata from leaf Prometheus-es! - [#3031](https://github.com/thanos-io/thanos/pull/3031) Compact/Sidecar/Receive/Rule: Added `--hash-func`. If some function has been specified, writers calculate hashes using that function of each file in a block before uploading them. If those hashes exist in the `meta.json` file then Compact does not download the files if they already exist on disk and with the same hash. This also means that the data directory passed to Thanos Compact is only *cleared once at boot* or *if everything succeeds*. So, if you, for example, use persistent volumes on k8s and your Thanos Compact crashes or fails to make an iteration properly then the last downloaded files are not wiped from the disk. The directories that were created the last time are only wiped again after a successful iteration or if the previously picked up blocks have disappeared. - ### Fixed - [#3705](https://github.com/thanos-io/thanos/pull/3705) Store: Fix race condition leading to failing queries or possibly incorrect query results. diff --git a/VERSION b/VERSION index 196f1821a1..2ea21d0904 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.19.0-rc.1 \ No newline at end of file +0.19.0-rc.2 \ No newline at end of file diff --git a/tutorials/katacoda/thanos/1-globalview/courseBase.sh b/tutorials/katacoda/thanos/1-globalview/courseBase.sh index 6c77da4e83..46a04f83a1 100644 --- a/tutorials/katacoda/thanos/1-globalview/courseBase.sh +++ b/tutorials/katacoda/thanos/1-globalview/courseBase.sh @@ -1,4 +1,4 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.16.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.1 +docker pull quay.io/thanos/thanos:v0.19.0-rc.2 diff --git a/tutorials/katacoda/thanos/1-globalview/step2.md b/tutorials/katacoda/thanos/1-globalview/step2.md index 0d6feef8d0..913410d747 100644 --- a/tutorials/katacoda/thanos/1-globalview/step2.md +++ b/tutorials/katacoda/thanos/1-globalview/step2.md @@ -10,7 +10,7 @@ component and can be invoked in a single command. Let's take a look at all the Thanos commands: ``` -docker run --rm quay.io/thanos/thanos:v0.19.0-rc.1 --help +docker run --rm quay.io/thanos/thanos:v0.19.0-rc.2 --help ```{{execute}} You should see multiple commands that solves different purposes. @@ -53,7 +53,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-eu1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -68,7 +68,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -81,7 +81,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus1_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-1-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ diff --git a/tutorials/katacoda/thanos/1-globalview/step3.md b/tutorials/katacoda/thanos/1-globalview/step3.md index 116f24c081..95a0329d76 100644 --- a/tutorials/katacoda/thanos/1-globalview/step3.md +++ b/tutorials/katacoda/thanos/1-globalview/step3.md @@ -28,7 +28,7 @@ Click below snippet to start the Querier. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:29090 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/courseBase.sh b/tutorials/katacoda/thanos/2-lts/courseBase.sh index f740592076..4f3a417af1 100644 --- a/tutorials/katacoda/thanos/2-lts/courseBase.sh +++ b/tutorials/katacoda/thanos/2-lts/courseBase.sh @@ -2,7 +2,7 @@ docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.1 +docker pull quay.io/thanos/thanos:v0.19.0-rc.2 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 mkdir /root/editor diff --git a/tutorials/katacoda/thanos/2-lts/step1.md b/tutorials/katacoda/thanos/2-lts/step1.md index fba61a8d70..cfc2a82762 100644 --- a/tutorials/katacoda/thanos/2-lts/step1.md +++ b/tutorials/katacoda/thanos/2-lts/step1.md @@ -117,7 +117,7 @@ Similar to previous course, let's setup global view querying with sidecar: docker run -d --net=host --rm \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -130,7 +130,7 @@ so we will make sure we point the Querier to the gRPC endpoints of the sidecar: ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step2.md b/tutorials/katacoda/thanos/2-lts/step2.md index 561a72f2b9..f94f223217 100644 --- a/tutorials/katacoda/thanos/2-lts/step2.md +++ b/tutorials/katacoda/thanos/2-lts/step2.md @@ -79,7 +79,7 @@ docker run -d --net=host --rm \ -v /root/prom-eu1:/prometheus \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ diff --git a/tutorials/katacoda/thanos/2-lts/step3.md b/tutorials/katacoda/thanos/2-lts/step3.md index a0c69d9ccf..6f394d3e21 100644 --- a/tutorials/katacoda/thanos/2-lts/step3.md +++ b/tutorials/katacoda/thanos/2-lts/step3.md @@ -6,7 +6,7 @@ In this step, we will learn about Thanos Store Gateway and how to deploy it. Let's take a look at all the Thanos commands: -```docker run --rm quay.io/thanos/thanos:v0.19.0-rc.1 --help```{{execute}} +```docker run --rm quay.io/thanos/thanos:v0.19.0-rc.2 --help```{{execute}} You should see multiple commands that solve different purposes, block storage based long-term storage for Prometheus. @@ -32,7 +32,7 @@ You can read more about [Store](https://thanos.io/tip/components/store.md/) here docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19091 \ @@ -49,7 +49,7 @@ Currently querier does not know about store yet. Let's change it by adding Store docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step4.md b/tutorials/katacoda/thanos/2-lts/step4.md index 8b78709b7a..d2cda01fe7 100644 --- a/tutorials/katacoda/thanos/2-lts/step4.md +++ b/tutorials/katacoda/thanos/2-lts/step4.md @@ -25,7 +25,7 @@ Click below snippet to start the Compactor. docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name thanos-compact \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ diff --git a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh index b611da79ed..9d19da1017 100644 --- a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh +++ b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.22.2 -docker pull quay.io/thanos/thanos:v0.19.0-rc.1 +docker pull quay.io/thanos/thanos:v0.19.0-rc.2 docker pull yannrobert/docker-nginx diff --git a/tutorials/katacoda/thanos/6-query-caching/step1.md b/tutorials/katacoda/thanos/6-query-caching/step1.md index cc2e6e3b6f..2554486cfb 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step1.md +++ b/tutorials/katacoda/thanos/6-query-caching/step1.md @@ -103,7 +103,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \ --name prometheus-sidecar"${i}" \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address=0.0.0.0:1909"${i}" \ --grpc-address=0.0.0.0:1919"${i}" \ @@ -129,7 +129,7 @@ And now, let's deploy Thanos Querier to have a global overview on our services. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:10912 \ --grpc-address 0.0.0.0:10901 \ diff --git a/tutorials/katacoda/thanos/6-query-caching/step2.md b/tutorials/katacoda/thanos/6-query-caching/step2.md index 1b521ca2eb..3d2df4aa79 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step2.md +++ b/tutorials/katacoda/thanos/6-query-caching/step2.md @@ -62,7 +62,7 @@ And deploy Query Frontend: docker run -d --net=host --rm \ -v $(pwd)/frontend.yml:/etc/thanos/frontend.yml \ --name query-frontend \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query-frontend \ --http-address 0.0.0.0:20902 \ --query-frontend.compress-responses \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh index 2c1a02a1cd..db5d262e46 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh +++ b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.1 +docker pull quay.io/thanos/thanos:v0.19.0-rc.2 docker pull quay.io/thanos/prom-label-proxy:v0.3.0-rc.0-ext1 docker pull caddy:2.2.1 diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md index 4c1979851f..71e1d71ba2 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md @@ -88,7 +88,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_fruit.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-fruit \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -120,7 +120,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -152,7 +152,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus1_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-01-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -170,7 +170,7 @@ Fruit: ``` docker run -d --net=host --rm \ --name querier-fruit \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:29091 \ --grpc-address 0.0.0.0:29191 \ @@ -183,7 +183,7 @@ Veggie: ``` docker run -d --net=host --rm \ --name querier-veggie \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:29092 \ --grpc-address 0.0.0.0:29192 \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md index 3e3ed83628..955904ba12 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md @@ -11,7 +11,7 @@ docker stop querier-fruit && docker stop querier-veggie ``` docker run -d --net=host --rm \ --name querier-multi \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:29090 \ --grpc-address 0.0.0.0:29190 \ diff --git a/tutorials/katacoda/thanos/x-playground/courseBase.sh b/tutorials/katacoda/thanos/x-playground/courseBase.sh index f5db1b0183..b433c45ae3 100644 --- a/tutorials/katacoda/thanos/x-playground/courseBase.sh +++ b/tutorials/katacoda/thanos/x-playground/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.1 +docker pull quay.io/thanos/thanos:v0.19.0-rc.2 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z diff --git a/tutorials/katacoda/thanos/x-playground/step1.md b/tutorials/katacoda/thanos/x-playground/step1.md index 44883dd871..eac662fdae 100644 --- a/tutorials/katacoda/thanos/x-playground/step1.md +++ b/tutorials/katacoda/thanos/x-playground/step1.md @@ -169,7 +169,7 @@ docker run -d --net=host --rm \ ### Step1: Sidecar ``` -docker run -it --rm quay.io/thanos/thanos:v0.19.0-rc.1 --help +docker run -it --rm quay.io/thanos/thanos:v0.19.0-rc.2 --help ```{{execute}} @@ -180,7 +180,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -195,7 +195,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -210,7 +210,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --http-address 0.0.0.0:19093 \ --grpc-address 0.0.0.0:19193 \ @@ -223,7 +223,7 @@ docker run -d --net=host --rm \ ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ diff --git a/tutorials/katacoda/thanos/x-playground/step2.md b/tutorials/katacoda/thanos/x-playground/step2.md index e8956fa1e1..443c0d3bee 100644 --- a/tutorials/katacoda/thanos/x-playground/step2.md +++ b/tutorials/katacoda/thanos/x-playground/step2.md @@ -65,7 +65,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0:/prometheus \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -85,7 +85,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1:/prometheus \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -105,7 +105,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0:/prometheus \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -130,7 +130,7 @@ Let's run Store Gateway server: docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19094 \ @@ -143,7 +143,7 @@ docker run -d --net=host --rm \ docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ @@ -162,7 +162,7 @@ Visit https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.co docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name compactor \ - quay.io/thanos/thanos:v0.19.0-rc.1 \ + quay.io/thanos/thanos:v0.19.0-rc.2 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ From faf1d041e3798bf13aeb5a62cd4c3ac8fcd1e278 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 31 Mar 2021 12:59:13 +0200 Subject: [PATCH 13/15] pkg/rules: fix deduplication of equal alerts with different labels (#3960) (#3999) Currently, if an alerting rule having the same name with different severity labels is being returned from different replicas then they are being treated as separate alerts. Given the following alerts a1,a2 with severities s1,s2 returned from replicas r1,2: a1[s1,r1] a1[s2,r1] a1[s1,r2] a1[s2,r2] Then, currently, the algorithm deduplicates to: a1[s1] a1[s2] a1[s1] a1[s2] Instead of the intendet result: a1[s1] a1[s2] This fixes it by removing replica labels before sorting labels for deduplication. Signed-off-by: Sergiusz Urbaniak # Conflicts: # CHANGELOG.md Co-authored-by: Sergiusz Urbaniak --- CHANGELOG.md | 3 +- pkg/rules/rules.go | 9 ++-- pkg/rules/rules_test.go | 108 +++++++++++++++++++++++++++++++++++++--- 3 files changed, 106 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e2c5e8aa1f..d55e080ba8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,7 +35,8 @@ We use _breaking :warning:_ to mark changes that are not backward compatible (re - [#3815](https://github.com/thanos-io/thanos/pull/3815) Receive: Improve handling of empty time series from clients - [#3795](https://github.com/thanos-io/thanos/pull/3795) s3: A truncated "get object" response is reported as error. - [#3899](https://github.com/thanos-io/thanos/pull/3899) Receive: Correct the inference of client gRPC configuration. -- [#3943](https://github.com/thanos-io/thanos/pull/3943): Receive: Fixed memory regression introduced in v0.17.0. +- [#3943](https://github.com/thanos-io/thanos/pull/3943) Receive: Fixed memory regression introduced in v0.17.0. +- [#3960](https://github.com/thanos-io/thanos/pull/3960) Query: Fixed deduplication of equal alerts with different labels. ### Changed diff --git a/pkg/rules/rules.go b/pkg/rules/rules.go index 9d40ee2403..422d03297c 100644 --- a/pkg/rules/rules.go +++ b/pkg/rules/rules.go @@ -68,23 +68,22 @@ func dedupRules(rules []*rulespb.Rule, replicaLabels map[string]struct{}) []*rul return rules } - // Sort each rule's label names such that they are comparable. + // Remove replica labels and sort each rule's label names such that they are comparable. for _, r := range rules { + removeReplicaLabels(r, replicaLabels) sort.Slice(r.GetLabels(), func(i, j int) bool { return r.GetLabels()[i].Name < r.GetLabels()[j].Name }) } - // Sort rules globally based on synthesized deduplication labels, also considering replica labels and their values. + // Sort rules globally. sort.Slice(rules, func(i, j int) bool { return rules[i].Compare(rules[j]) < 0 }) - // Remove rules based on synthesized deduplication labels, this time ignoring replica labels and last evaluation. + // Remove rules based on synthesized deduplication labels. i := 0 - removeReplicaLabels(rules[i], replicaLabels) for j := 1; j < len(rules); j++ { - removeReplicaLabels(rules[j], replicaLabels) if rules[i].Compare(rules[j]) != 0 { // Effectively retain rules[j] in the resulting slice. i++ diff --git a/pkg/rules/rules_test.go b/pkg/rules/rules_test.go index ada7bf8ebf..b6e67be19e 100644 --- a/pkg/rules/rules_test.go +++ b/pkg/rules/rules_test.go @@ -448,14 +448,14 @@ func TestDedupRules(t *testing.T) { rulespb.NewAlertingRule(&rulespb.Alert{ Name: "a1", Query: "up", - DurationSeconds: 2.0, + DurationSeconds: 1.0, Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ {Name: "a", Value: "1"}, }}}), rulespb.NewAlertingRule(&rulespb.Alert{ Name: "a1", Query: "up", - DurationSeconds: 1.0, + DurationSeconds: 2.0, Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ {Name: "a", Value: "1"}, }}}), @@ -505,13 +505,13 @@ func TestDedupRules(t *testing.T) { }}}), }, want: []*rulespb.Rule{ + rulespb.NewRecordingRule(&rulespb.RecordingRule{Name: "a1"}), rulespb.NewRecordingRule(&rulespb.RecordingRule{Name: "a1", Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ {Name: "a", Value: "1"}, }}}), rulespb.NewRecordingRule(&rulespb.RecordingRule{Name: "a1", Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ {Name: "a", Value: "2"}, }}}), - rulespb.NewRecordingRule(&rulespb.RecordingRule{Name: "a1"}), }, replicaLabels: []string{"replica"}, }, @@ -613,6 +613,11 @@ func TestDedupRules(t *testing.T) { }), }, want: []*rulespb.Rule{ + rulespb.NewAlertingRule(&rulespb.Alert{ + State: rulespb.AlertState_FIRING, + Name: "a1", + LastEvaluation: time.Unix(2, 0), + }), rulespb.NewAlertingRule(&rulespb.Alert{ State: rulespb.AlertState_FIRING, Name: "a1", @@ -621,11 +626,6 @@ func TestDedupRules(t *testing.T) { }}, LastEvaluation: time.Unix(1, 0), }), - rulespb.NewAlertingRule(&rulespb.Alert{ - State: rulespb.AlertState_FIRING, - Name: "a1", - LastEvaluation: time.Unix(2, 0), - }), rulespb.NewAlertingRule(&rulespb.Alert{ State: rulespb.AlertState_PENDING, Name: "a2", @@ -634,6 +634,98 @@ func TestDedupRules(t *testing.T) { }, replicaLabels: []string{"replica"}, }, + { + name: "alerts with different severity", + rules: []*rulespb.Rule{ + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "replica", Value: "1"}, + {Name: "severity", Value: "warning"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "replica", Value: "1"}, + {Name: "severity", Value: "critical"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "replica", Value: "2"}, + {Name: "severity", Value: "warning"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "replica", Value: "2"}, + {Name: "severity", Value: "critical"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + }, + want: []*rulespb.Rule{ + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "severity", Value: "critical"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "severity", Value: "warning"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + }, + replicaLabels: []string{"replica"}, + }, + { + name: "alerts with missing replica labels", + rules: []*rulespb.Rule{ + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "replica", Value: "1"}, + {Name: "label", Value: "foo"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "replica", Value: "2"}, + {Name: "label", Value: "foo"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "label", Value: "foo"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + }, + want: []*rulespb.Rule{ + rulespb.NewAlertingRule(&rulespb.Alert{ + Name: "a1", + Labels: labelpb.ZLabelSet{Labels: []labelpb.ZLabel{ + {Name: "label", Value: "foo"}, + }}, + LastEvaluation: time.Unix(1, 0), + }), + }, + replicaLabels: []string{"replica"}, + }, } { t.Run(tc.name, func(t *testing.T) { replicaLabels := make(map[string]struct{}) From 18c0dbe9e5d4b944d1f786ba8a5ca3d654033343 Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 31 Mar 2021 14:01:15 +0200 Subject: [PATCH 14/15] Cut v0.19.0 (#3998) Signed-off-by: Bartlomiej Plotka --- CHANGELOG.md | 2 +- tutorials/katacoda/thanos/1-globalview/courseBase.sh | 2 +- tutorials/katacoda/thanos/1-globalview/step2.md | 8 ++++---- tutorials/katacoda/thanos/1-globalview/step3.md | 2 +- tutorials/katacoda/thanos/2-lts/courseBase.sh | 2 +- tutorials/katacoda/thanos/2-lts/step1.md | 4 ++-- tutorials/katacoda/thanos/2-lts/step2.md | 2 +- tutorials/katacoda/thanos/2-lts/step3.md | 6 +++--- tutorials/katacoda/thanos/2-lts/step4.md | 2 +- .../katacoda/thanos/6-query-caching/courseBase.sh | 2 +- tutorials/katacoda/thanos/6-query-caching/step1.md | 4 ++-- tutorials/katacoda/thanos/6-query-caching/step2.md | 2 +- .../katacoda/thanos/7-multi-tenancy/courseBase.sh | 2 +- tutorials/katacoda/thanos/7-multi-tenancy/step1.md | 10 +++++----- tutorials/katacoda/thanos/7-multi-tenancy/step2.md | 2 +- tutorials/katacoda/thanos/x-playground/courseBase.sh | 2 +- tutorials/katacoda/thanos/x-playground/step1.md | 10 +++++----- tutorials/katacoda/thanos/x-playground/step2.md | 12 ++++++------ 18 files changed, 38 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d55e080ba8..9ecc404b3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ NOTE: As semantic versioning states all 0.y.z releases can contain breaking chan We use _breaking :warning:_ to mark changes that are not backward compatible (relates only to v0.y.z releases.) -## [v0.19.0-rc.2](https://github.com/thanos-io/thanos/releases/tag/v0.19.0-rc.2) - 2021.03.24 +## [v0.19.0](https://github.com/thanos-io/thanos/releases/tag/v0.19.0) - 2021.03.31 ### Added diff --git a/tutorials/katacoda/thanos/1-globalview/courseBase.sh b/tutorials/katacoda/thanos/1-globalview/courseBase.sh index 46a04f83a1..14a23d8254 100644 --- a/tutorials/katacoda/thanos/1-globalview/courseBase.sh +++ b/tutorials/katacoda/thanos/1-globalview/courseBase.sh @@ -1,4 +1,4 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.16.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.2 +docker pull quay.io/thanos/thanos:v0.19.0 diff --git a/tutorials/katacoda/thanos/1-globalview/step2.md b/tutorials/katacoda/thanos/1-globalview/step2.md index 913410d747..6967d6ebc9 100644 --- a/tutorials/katacoda/thanos/1-globalview/step2.md +++ b/tutorials/katacoda/thanos/1-globalview/step2.md @@ -10,7 +10,7 @@ component and can be invoked in a single command. Let's take a look at all the Thanos commands: ``` -docker run --rm quay.io/thanos/thanos:v0.19.0-rc.2 --help +docker run --rm quay.io/thanos/thanos:v0.19.0 --help ```{{execute}} You should see multiple commands that solves different purposes. @@ -53,7 +53,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-eu1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -68,7 +68,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus0_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -81,7 +81,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus1_us1.yml:/etc/prometheus/prometheus.yml \ --name prometheus-1-sidecar-us1 \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ diff --git a/tutorials/katacoda/thanos/1-globalview/step3.md b/tutorials/katacoda/thanos/1-globalview/step3.md index 95a0329d76..ce3938b320 100644 --- a/tutorials/katacoda/thanos/1-globalview/step3.md +++ b/tutorials/katacoda/thanos/1-globalview/step3.md @@ -28,7 +28,7 @@ Click below snippet to start the Querier. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:29090 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/courseBase.sh b/tutorials/katacoda/thanos/2-lts/courseBase.sh index 4f3a417af1..74fc448ad6 100644 --- a/tutorials/katacoda/thanos/2-lts/courseBase.sh +++ b/tutorials/katacoda/thanos/2-lts/courseBase.sh @@ -2,7 +2,7 @@ docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.2 +docker pull quay.io/thanos/thanos:v0.19.0 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 mkdir /root/editor diff --git a/tutorials/katacoda/thanos/2-lts/step1.md b/tutorials/katacoda/thanos/2-lts/step1.md index cfc2a82762..a1b8f4d98b 100644 --- a/tutorials/katacoda/thanos/2-lts/step1.md +++ b/tutorials/katacoda/thanos/2-lts/step1.md @@ -117,7 +117,7 @@ Similar to previous course, let's setup global view querying with sidecar: docker run -d --net=host --rm \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -130,7 +130,7 @@ so we will make sure we point the Querier to the gRPC endpoints of the sidecar: ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step2.md b/tutorials/katacoda/thanos/2-lts/step2.md index f94f223217..281d27bb7e 100644 --- a/tutorials/katacoda/thanos/2-lts/step2.md +++ b/tutorials/katacoda/thanos/2-lts/step2.md @@ -79,7 +79,7 @@ docker run -d --net=host --rm \ -v /root/prom-eu1:/prometheus \ --name prometheus-0-eu1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ diff --git a/tutorials/katacoda/thanos/2-lts/step3.md b/tutorials/katacoda/thanos/2-lts/step3.md index 6f394d3e21..c383423037 100644 --- a/tutorials/katacoda/thanos/2-lts/step3.md +++ b/tutorials/katacoda/thanos/2-lts/step3.md @@ -6,7 +6,7 @@ In this step, we will learn about Thanos Store Gateway and how to deploy it. Let's take a look at all the Thanos commands: -```docker run --rm quay.io/thanos/thanos:v0.19.0-rc.2 --help```{{execute}} +```docker run --rm quay.io/thanos/thanos:v0.19.0 --help```{{execute}} You should see multiple commands that solve different purposes, block storage based long-term storage for Prometheus. @@ -32,7 +32,7 @@ You can read more about [Store](https://thanos.io/tip/components/store.md/) here docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19091 \ @@ -49,7 +49,7 @@ Currently querier does not know about store yet. Let's change it by adding Store docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:9091 \ --query.replica-label replica \ diff --git a/tutorials/katacoda/thanos/2-lts/step4.md b/tutorials/katacoda/thanos/2-lts/step4.md index d2cda01fe7..7320dff011 100644 --- a/tutorials/katacoda/thanos/2-lts/step4.md +++ b/tutorials/katacoda/thanos/2-lts/step4.md @@ -25,7 +25,7 @@ Click below snippet to start the Compactor. docker run -d --net=host --rm \ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \ --name thanos-compact \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ diff --git a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh index 9d19da1017..7fc0465d4c 100644 --- a/tutorials/katacoda/thanos/6-query-caching/courseBase.sh +++ b/tutorials/katacoda/thanos/6-query-caching/courseBase.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.22.2 -docker pull quay.io/thanos/thanos:v0.19.0-rc.2 +docker pull quay.io/thanos/thanos:v0.19.0 docker pull yannrobert/docker-nginx diff --git a/tutorials/katacoda/thanos/6-query-caching/step1.md b/tutorials/katacoda/thanos/6-query-caching/step1.md index 2554486cfb..021f99a3c9 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step1.md +++ b/tutorials/katacoda/thanos/6-query-caching/step1.md @@ -103,7 +103,7 @@ docker run -d --net=host --rm \ -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \ --name prometheus-sidecar"${i}" \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address=0.0.0.0:1909"${i}" \ --grpc-address=0.0.0.0:1919"${i}" \ @@ -129,7 +129,7 @@ And now, let's deploy Thanos Querier to have a global overview on our services. ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:10912 \ --grpc-address 0.0.0.0:10901 \ diff --git a/tutorials/katacoda/thanos/6-query-caching/step2.md b/tutorials/katacoda/thanos/6-query-caching/step2.md index 3d2df4aa79..4ee1a602a4 100644 --- a/tutorials/katacoda/thanos/6-query-caching/step2.md +++ b/tutorials/katacoda/thanos/6-query-caching/step2.md @@ -62,7 +62,7 @@ And deploy Query Frontend: docker run -d --net=host --rm \ -v $(pwd)/frontend.yml:/etc/thanos/frontend.yml \ --name query-frontend \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query-frontend \ --http-address 0.0.0.0:20902 \ --query-frontend.compress-responses \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh index db5d262e46..c7c24cd8b2 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh +++ b/tutorials/katacoda/thanos/7-multi-tenancy/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.2 +docker pull quay.io/thanos/thanos:v0.19.0 docker pull quay.io/thanos/prom-label-proxy:v0.3.0-rc.0-ext1 docker pull caddy:2.2.1 diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md index 71e1d71ba2..dbd12b05e3 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step1.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step1.md @@ -88,7 +88,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_fruit.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-fruit \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19090 \ --grpc-address 0.0.0.0:19190 \ @@ -120,7 +120,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus0_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-0-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -152,7 +152,7 @@ docker run -d --net=host --rm \ -v $(pwd)/editor/prometheus1_veggie.yml:/etc/prometheus/prometheus.yml \ --name prometheus-01-sidecar-veggie \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -170,7 +170,7 @@ Fruit: ``` docker run -d --net=host --rm \ --name querier-fruit \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:29091 \ --grpc-address 0.0.0.0:29191 \ @@ -183,7 +183,7 @@ Veggie: ``` docker run -d --net=host --rm \ --name querier-veggie \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:29092 \ --grpc-address 0.0.0.0:29192 \ diff --git a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md index 955904ba12..fc33d48568 100644 --- a/tutorials/katacoda/thanos/7-multi-tenancy/step2.md +++ b/tutorials/katacoda/thanos/7-multi-tenancy/step2.md @@ -11,7 +11,7 @@ docker stop querier-fruit && docker stop querier-veggie ``` docker run -d --net=host --rm \ --name querier-multi \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:29090 \ --grpc-address 0.0.0.0:29190 \ diff --git a/tutorials/katacoda/thanos/x-playground/courseBase.sh b/tutorials/katacoda/thanos/x-playground/courseBase.sh index b433c45ae3..2315b1cc54 100644 --- a/tutorials/katacoda/thanos/x-playground/courseBase.sh +++ b/tutorials/katacoda/thanos/x-playground/courseBase.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash docker pull quay.io/prometheus/prometheus:v2.20.0 -docker pull quay.io/thanos/thanos:v0.19.0-rc.2 +docker pull quay.io/thanos/thanos:v0.19.0 docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1 docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z diff --git a/tutorials/katacoda/thanos/x-playground/step1.md b/tutorials/katacoda/thanos/x-playground/step1.md index eac662fdae..15b075ca7f 100644 --- a/tutorials/katacoda/thanos/x-playground/step1.md +++ b/tutorials/katacoda/thanos/x-playground/step1.md @@ -169,7 +169,7 @@ docker run -d --net=host --rm \ ### Step1: Sidecar ``` -docker run -it --rm quay.io/thanos/thanos:v0.19.0-rc.2 --help +docker run -it --rm quay.io/thanos/thanos:v0.19.0 --help ```{{execute}} @@ -180,7 +180,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19091 \ --grpc-address 0.0.0.0:19191 \ @@ -195,7 +195,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19092 \ --grpc-address 0.0.0.0:19192 \ @@ -210,7 +210,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0-config.yaml:/etc/prometheus/prometheus.yml \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --http-address 0.0.0.0:19093 \ --grpc-address 0.0.0.0:19193 \ @@ -223,7 +223,7 @@ docker run -d --net=host --rm \ ``` docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ diff --git a/tutorials/katacoda/thanos/x-playground/step2.md b/tutorials/katacoda/thanos/x-playground/step2.md index 443c0d3bee..29ab1e4b12 100644 --- a/tutorials/katacoda/thanos/x-playground/step2.md +++ b/tutorials/katacoda/thanos/x-playground/step2.md @@ -65,7 +65,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica0:/prometheus \ --name prom-eu1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -85,7 +85,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-eu1-replica1:/prometheus \ --name prom-eu1-1-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -105,7 +105,7 @@ docker run -d --net=host --rm \ -v ${CURR_DIR}/prom-us1-replica0:/prometheus \ --name prom-us1-0-sidecar \ -u root \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ sidecar \ --tsdb.path /prometheus \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ @@ -130,7 +130,7 @@ Let's run Store Gateway server: docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name store-gateway \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ store \ --objstore.config-file /etc/thanos/minio-bucket.yaml \ --http-address 0.0.0.0:19094 \ @@ -143,7 +143,7 @@ docker run -d --net=host --rm \ docker stop querier && \ docker run -d --net=host --rm \ --name querier \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ query \ --http-address 0.0.0.0:9090 \ --grpc-address 0.0.0.0:19190 \ @@ -162,7 +162,7 @@ Visit https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.co docker run -d --net=host --rm \ -v ${CURR_DIR}/minio-bucket.yaml:/etc/thanos/minio-bucket.yaml \ --name compactor \ - quay.io/thanos/thanos:v0.19.0-rc.2 \ + quay.io/thanos/thanos:v0.19.0 \ compact \ --wait --wait-interval 30s \ --consistency-delay 0s \ From ba6c5c4726ff52807c7383c68f2159b1af7980bb Mon Sep 17 00:00:00 2001 From: Bartlomiej Plotka Date: Wed, 31 Mar 2021 14:02:36 +0200 Subject: [PATCH 15/15] Fixed VERSION file. Signed-off-by: Bartlomiej Plotka --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 2ea21d0904..1cf0537c34 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.19.0-rc.2 \ No newline at end of file +0.19.0