.*: Prometheus library updated to v2.8.1 #1009

vitkovskii · 2019-04-04T14:03:59Z

Changes

Updated github.com/prometheus/prometheus to 2.8.1.
Updated github.com/prometheus/tsdb to 0.6.1
Added prometheus 2.8.1 to e2e tests.

Verification

make test runs successfully.

cmd/thanos/compact.go

povilasv

Could you also do more testing locally or on your dev cluster ? It's a bit of a scary change :D

vitkovskii · 2019-04-05T15:05:23Z

Yes, I'll do it next week. Actually, I have tested the query component and it works well.

pkg/testutil/prometheus.go

bwplotka · 2019-04-09T13:15:00Z

I think we should delay this change until we need this. Is there anything in particular that we look for in upgraded prom or tsdb code? It's changing massive amount of things in hidden way so we need to be careful.

bwplotka · 2019-04-09T13:15:55Z

If the aim is to add 2.8.1 to e2e then we don't need dep change

vitkovskii · 2019-04-09T13:19:10Z

Could you also do more testing locally or on your dev cluster ? It's a bit of a scary change :D

@povilasv Hey. I've tested sidecar, store and query component locally with about 2Gb of data. And it runs well. I've also successfully tested query with 11 production stores. New prometheus features like the explanation of many-to-many error work fine too.

vitkovskii · 2019-04-09T13:27:57Z

Is there anything in particular that we look for in this change?

@bwplotka Prometheus 2.8.0 has cool feature explaining why you have "many-to-many" error when you dealing with metrics matching. It's cool to have this explanation in Thanos too. We are using Thanos as "single endpoint" for all our metrics. And it's hard to debug these errors. An expression may run well on particular Prometheus instance, but sometimes you have "many-to-many" error from Thanos. And it's hard to figure out why this happens without explanation.

vitkovskii · 2019-04-09T14:28:35Z

bwplotka · 2019-04-09T14:34:52Z

Cool, that's a valid request, but we need to double check all, including FULL change log of what changed in TSDB and Prometheus in relation to what packages we use in Thanos like we did here: https://github.com/improbable-eng/thanos/blob/master/CHANGELOG.md#changed-1

I will review more carefully as well.

bwplotka · 2019-04-16T21:24:44Z

Ok, looks like this will help with couple of things e.g:

ruler: prometheus_rule_group_last_duration_seconds is not working. #1027

I will help you a bit and will do the CHANGELOG part. I will add this as part of: #1017

So let's focus here on rebasing and making the changes right. Is that ok @vitkovskii @d-ulyanov so we can fit into 0.4.0 release.

bwplotka

Those changes looks legit, not much of an API changes, so it should be fine 👍

Let's rebase.

bwplotka · 2019-04-16T21:28:24Z

pkg/testutil/prometheus.go

-		}, extra...)...,
-	)
+	args := append([]string{
+		"--storage.tsdb.retention=2d", // Pass retention cause prometheus since 2.8.0 don't show default value for that flags in web/api: https://github.com/prometheus/prometheus/pull/5433


Just FYI: We need better handling for compatibility testing. Ideas are here

Feedback and help wanted (:

bwplotka

I think this version change is safe, but just curious why not grabbing newest. Is there a reason?

bwplotka · 2019-04-16T21:32:24Z

go.mod

@@ -34,8 +33,8 @@ require (
 	github.com/pkg/errors v0.8.1
 	github.com/prometheus/client_golang v0.9.2
 	github.com/prometheus/common v0.0.0-20181218105931-67670fe90761
-	github.com/prometheus/prometheus v0.0.0-20190118110214-3bd41cc92c78
-	github.com/prometheus/tsdb v0.4.0
+	github.com/prometheus/prometheus v0.0.0-20190328180107-4d60eb36dcbe


Why this commit not some release tag?

v2.8.1 essentially

bwplotka · 2019-04-16T21:32:36Z

go.mod

-	github.com/prometheus/prometheus v0.0.0-20190118110214-3bd41cc92c78
-	github.com/prometheus/tsdb v0.4.0
+	github.com/prometheus/prometheus v0.0.0-20190328180107-4d60eb36dcbe
+	github.com/prometheus/tsdb v0.6.1


Why this version not newest?

I guess it's because 2.8.1 uses this? Then it's fine

yes, you're right

bwplotka · 2019-04-16T21:39:34Z

Disabled by default and can be enabled via AllowOverlappingBlock option.

Can we add that explicitly in ruler TSDB? And we need to double check compactor for this as well.

bwplotka · 2019-04-16T21:56:29Z

Updated #1017 with this:

- [1009](https://github.com/improbable-eng/thanos/pull/1009): Upgraded Prometheus (~v2.7.0-rc.0 to v2.8.1)  and TSDB (`v0.4.0` to `v0.6.1`) deps.
  
  Changes that affects Thanos:
   * query: 
     * [ENHANCEMENT] In histogram_quantile merge buckets with equivalent le values. #5158.   
     * [ENHANCEMENT] Show list of offending labels in the error message in many-to-many scenarios. #5189   
     * [BUGFIX] Fix panic when aggregator param is not a literal. #5290
   * ruler: 
     * [ENHANCEMENT] Reduce time that Alertmanagers are in flux when reloaded. #5126
     * [BUGFIX] prometheus_rule_group_last_evaluation_timestamp_seconds is now a unix timestamp. #5186
     * [BUGFIX] prometheus_rule_group_last_duration_seconds now reports seconds instead of nanoseconds. Fixes our [issue #1027](https://github.com/improbable-eng/thanos/issues/1027)
     * [BUGFIX] Fix sorting of rule groups. #5260
   * store: [ENHANCEMENT] Fast path for EmptyPostings cases in Merge, Intersect and Without.
   * tooling: [FEATURE] New dump command to tsdb tool to dump all samples.
   * compactor: [ENHANCEMENT] When closing the db any running compaction will be cancelled so it doesn't block.
  
  For ruler essentially whole TSDB CHANGELOG applies beween v0.4.0-v0.6.1: https://github.com/prometheus/tsdb/blob/master/CHANGELOG.md
  
  Note that this was added on TSDB and Prometheus: [FEATURE] Time-ovelapping blocks are now allowed. #370
  Whoever due to nature of Thanos compaction (distributed systems), for safety reason this is disabled for Thanos compactor for now.

LGTM IMO

vitkovskii · 2019-04-17T11:28:03Z

@bwplotka @povilasv Guys, thanks a lot :)

* query: cleanup store statuses as they come and go (thanos-io#910) Signed-off-by: Adrien Fillon <adrien.fillon@cdiscount.com> * [docs] Example of using official prometheus helm chart to deploy server with sidecar (thanos-io#1003) * update documentation with an example of using official prometheus helm chart Signed-off-by: Ivan Kiselev <ivan@messagebird.com> * a little formatting to values Signed-off-by: Ivan Kiselev <ivan@messagebird.com> * satisfy PR comments Signed-off-by: Ivan Kiselev <ivan@messagebird.com> * Compact: group concurrency (thanos-io#1010) * compact: add concurrency to group compact * add flag to controll the number of goroutines to use when compacting group * update compact.md for group-compact-concurrency * fixed: miss wg.Add() * address CR * regenerate docs * use err group * fix typo in flag description * handle context * set up workers in main loop * move var initialisation * remove debug log * validate concurrency * move comment * warn -> error * remove extra newline * fix typo * dns: Added miekgdns resolver as a hidden option to query and ruler. (thanos-io#1016) Fixes: thanos-io#1015 Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * query: set default evaluation interval (thanos-io#1028) Subqueries allows request with no [specified resolution](https://prometheus.io/blog/2019/01/28/subquery-support/). Set it up and allow to configure default evaluation interval. * store+compactor: pre-compute index cache during compaction (thanos-io#986) Fixes first part of thanos-io#942 This changes allow to safe some startup & sync time in store gateway as it is no longer is needed to compute index-cache from block index on its own. For compatibility store GW still can do it, but it first checks bucket if there is index-cached uploaded already. In the same time, compactor precomputes the index cache file on every compaction. To allow quicker addition of index cache files we added `--index.generate-missing-cache-file` flag, that if enabled precompute missing files on compactor startup. Note that it will take time and it's only one-off step per bucket. Signed-off-by: Aleksei Semiglazov <xjewer@gmail.com> * Added website for Thanos' docs using Hugo. (thanos-io#807) Hosted in github pages. Signed-off-by: adrien-f <adrien.fillon@gmail.com> Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * gcs: Fixed scopes for inline ServiceAccount option. (thanos-io#1033) Without this that option was unusable. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * Fixed root docs and liche is now checking root dir as well. (thanos-io#1040) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * storage docs: add detail about GCS policies and testing (thanos-io#1037) * add more details about GCS policies and testing * remove fixed names from exec command * Prometheus library updated to v2.8.1 (thanos-io#1009) * compact: group concurrency improvements (thanos-io#1029) * group concurrency improvements * remove unnecessary error check * add to wg in main goroutine * receive: Add block shipping (thanos-io#1011) * receive: Add retention flag for local tsdb storage (thanos-io#1046) * querier: Add /api/v1/labels support (thanos-io#905) * Feature: add /api/v1/labels support Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Disabled gossip by default, marked all flags as deprecated. (thanos-io#1055) + changed small label. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * ruler: Fixed Chunk going out or Max Uint16. (thanos-io#1041) Fixes thanos-io#1038 Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * store: azure: allow passing an endpoint parameter for specific regions (thanos-io#980) Fix thanos-io#968 Signed-off-by: Adrien Fillon <adrien.fillon@cdiscount.com> * feature: support POST method for series endpoint (thanos-io#1021) Signed-off-by: Joseph Lee <joseph.t.lee@outlook.com> * bucket verify: repair out of order labels (thanos-io#964) * bucket verify: repair out of order labels * verify repair: correctly order series in the index on rewrite When we have label sets that are not in the correct order, fixing that changes the order of the series in the index. So the index must be rewritten in that new order. This makes this repair tool take up a bunch more memory, but produces blocks that verify correctly. * Fix the TSDB block safe-delete function The directory name must be the block ID name exactly to verify. A temp directory or random name will not work here. * verify repair: fix duplicate chunk detection Pointer/reference logic error was eliminating all chunks for a series in a given TSDB block that wasn't the first chunk. Chunks are now referenced correctly via pointers. * PR feedback: use errors.Errorf() instead of fmt.Errorf() * Use errors.New() Some linters catch errors.Errorf() as its not really part of the errors package. * Liberally comment this for loop We're comparing items by pointers, using Go's range variables is misleading here and we need not fall into the same trap. * Take advantage of sort.Interface This prevents us from having to re-implement label sorting. * PR Feedback: Comments are full sentences. * Cut release 0.4.0-rc.0 (thanos-io#1017) * Cut release 0.4.0-rc.0 🎉 🎉 NOTE: This is last release that has gossip. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> Co-Authored-By: povilasv <p.versockas@gmail.com> * Fixed crossbuild. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * ci: Env fixes. (thanos-io#1058) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * Removed bzr requirement for make crossbuild. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * Bump github.com/hashicorp/golang-lru from 0.5.0 to 0.5.1 (thanos-io#1051) Bumps [github.com/hashicorp/golang-lru](https://github.com/hashicorp/golang-lru) from 0.5.0 to 0.5.1. - [Release notes](https://github.com/hashicorp/golang-lru/releases) - [Commits](hashicorp/golang-lru@v0.5.0...v0.5.1) Signed-off-by: dependabot[bot] <support@dependabot.com> * Initialze and correctly register all index cache metrics. (thanos-io#1069) * store/cache: add more tests (thanos-io#1071) * Fixed Downsampling process; Fixed `runutil.CloseAndCaptureErr` (thanos-io#1070) * runutil. Simplified CloseWithErrCapture. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * Fixed Downsampling process; Fixed runutil.CloseAndCaptureErr Fixes thanos-io#1065 Root cause: * runutil defered capture error function was not passing error properly so unit tests were passing, event though there was bug * streamed block write index cache requires index file which was not closed (saved) properly yet. Closers need to be closed to perform this. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * objstore: Expose S3 region attribute (thanos-io#1060) Minio is able to autodetect the region for cloud providers like AWS but the logic fails with Scaleway Object Storage solution. Related issue on Minio: minio/mc#2570 * Fixed fetching go-bindata failed (thanos-io#1074) * Fixed bug: - fetching go-bindata failed. - change the repo of go-bindata to github.com/go-bindata/go-bindata, because old repo has been archived. - pin the go-bindata as v3.3.1. Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Add CHANGELOG Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Remove CHANGELOG Signed-off-by: jojohappy <sarahdj0917@gmail.com> * add compare flags func to compare flags between prometheus and sidecar (thanos-io#838) Original message: * update documentation for a max/min block duration add compare flags func to compare flags between prom and sidecar * fix some nits Functional change: now we check the configured flags (if possible) and error out if MinTime != MaxTime. We need to check this always since if that is not true then we will get overlapping blocks. Additionally, an error message is printed out if it is not equal to 2h (the recommended value). * Ensured index cache is best effort, refactored tests, validated edge cases. (thanos-io#1073) Fixes thanos-io#651 Current size also includes slice header. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * website: Moved to netlify. (thanos-io#1078) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * website: Fixing netlify. (thanos-io#1080) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * website: Added "founded by" footer. (thanos-io#1081) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * store/proxy: properly check if context has ended (thanos-io#1082) How the code was before it could happen that we might receive some series from the stream however by the time we'd send them back to the reader, it would not read it anymore since the deadline would have been exceeded. Properly use a `select` here to get out of the goroutine if the deadline has been exceeded. Might potentially fix a problem where we see one goroutine hanging constantly (and thus blocking from work being done): ``` goroutine profile: total 126 25 @ 0x42f62f 0x40502b 0x405001 0x404de5 0xe7435b 0x45cc41 0xe7435a github.com/improbable-eng/thanos/pkg/store.startStreamSeriesSet.func1+0x18a /go/src/github.com/improbable-eng/thanos/pkg/store/proxy.go:318 ``` * Cut release v0.4.0-rc.1 (thanos-io#1088) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * website: Removed ghpages handling; fixed docs; and status badge. (thanos-io#1084) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * Fix readme (thanos-io#1090) * store: Compose indexCache properly allowing injection for testing purposes. (thanos-io#1098) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * website: add sponsor section on homepage (thanos-io#1062) * website: Adjusted logos sizing and responsiveness. (thanos-io#1105) Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * Add Monzo to "Used by" section 🎉 (thanos-io#1106) * Compactor: remove malformed blocks after delay (thanos-io#1053) * compactor removes malformed blocks after delay * compactor removes malformed blocks after delay * include missing file * reuse existing freshness check * fix comment * remove unused var * fix comment * syncDelay -> consistencyDelay * fix comment * update flag description * address cr * fix dupliacte error handling * minimum value for --consistency-delay * update * docs * add test case * move test to inmem bucket * Add Utility Warehouse to "used by" section (thanos-io#1108) * Add Utility Warehouse logo * Make logo smaller * website: add Adform as users (thanos-io#1109) We use Thanos extensively as well so I have added Adform. * Cut release v0.4.0 (thanos-io#1107) Signed-off-by: Bartek Plotka <bwplotka@gmail.com>

povilasv reviewed Apr 4, 2019

View reviewed changes

cmd/thanos/compact.go Show resolved Hide resolved

povilasv reviewed Apr 5, 2019

View reviewed changes

GiedriusS reviewed Apr 5, 2019

View reviewed changes

pkg/testutil/prometheus.go Outdated Show resolved Hide resolved

vitkovskii force-pushed the udpate-prom-lib-version branch from 13c0591 to 8e41578 Compare April 9, 2019 12:58

Prometheus library updated to v2.8.1

39de070

vitkovskii force-pushed the udpate-prom-lib-version branch from 8e41578 to 39de070 Compare April 9, 2019 13:11

bwplotka approved these changes Apr 16, 2019

View reviewed changes

bwplotka reviewed Apr 16, 2019

View reviewed changes

povilasv approved these changes Apr 17, 2019

View reviewed changes

Merge branch 'master' into udpate-prom-lib-version

f63d4e7

bwplotka merged commit 9fad981 into thanos-io:master Apr 17, 2019

wbh1 mentioned this pull request May 10, 2019

Remove references to 'bzr' as it is no longer required #1137

Merged

jojohappy mentioned this pull request Jul 17, 2019

ruler: prometheus_rule_group_last_duration_seconds is not working. #1027

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.*: Prometheus library updated to v2.8.1 #1009

.*: Prometheus library updated to v2.8.1 #1009

vitkovskii commented Apr 4, 2019 •

edited

Loading

povilasv left a comment

vitkovskii commented Apr 5, 2019

bwplotka commented Apr 9, 2019 •

edited

Loading

bwplotka commented Apr 9, 2019

vitkovskii commented Apr 9, 2019

vitkovskii commented Apr 9, 2019 •

edited

Loading

vitkovskii commented Apr 9, 2019

bwplotka commented Apr 9, 2019

bwplotka commented Apr 16, 2019 •

edited

Loading

bwplotka left a comment •

edited

Loading

bwplotka Apr 16, 2019

bwplotka left a comment

bwplotka Apr 16, 2019

bwplotka Apr 16, 2019

bwplotka Apr 16, 2019

bwplotka Apr 16, 2019

d-ulyanov Apr 17, 2019

bwplotka commented Apr 16, 2019

bwplotka commented Apr 16, 2019

vitkovskii commented Apr 17, 2019

.*: Prometheus library updated to v2.8.1 #1009

.*: Prometheus library updated to v2.8.1 #1009

Conversation

vitkovskii commented Apr 4, 2019 • edited Loading

Changes

Verification

povilasv left a comment

Choose a reason for hiding this comment

vitkovskii commented Apr 5, 2019

bwplotka commented Apr 9, 2019 • edited Loading

bwplotka commented Apr 9, 2019

vitkovskii commented Apr 9, 2019

vitkovskii commented Apr 9, 2019 • edited Loading

vitkovskii commented Apr 9, 2019

bwplotka commented Apr 9, 2019

bwplotka commented Apr 16, 2019 • edited Loading

bwplotka left a comment • edited Loading

Choose a reason for hiding this comment

bwplotka Apr 16, 2019

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka Apr 16, 2019

Choose a reason for hiding this comment

bwplotka Apr 16, 2019

Choose a reason for hiding this comment

bwplotka Apr 16, 2019

Choose a reason for hiding this comment

bwplotka Apr 16, 2019

Choose a reason for hiding this comment

d-ulyanov Apr 17, 2019

Choose a reason for hiding this comment

bwplotka commented Apr 16, 2019

bwplotka commented Apr 16, 2019

vitkovskii commented Apr 17, 2019

vitkovskii commented Apr 4, 2019 •

edited

Loading

bwplotka commented Apr 9, 2019 •

edited

Loading

vitkovskii commented Apr 9, 2019 •

edited

Loading

bwplotka commented Apr 16, 2019 •

edited

Loading

bwplotka left a comment •

edited

Loading