compact+store+tools: Filter for duplicate block data only within a compaction group and in parallel #4963

nberkley · 2021-12-16T17:09:46Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Changes DeduplicateFilter to only look for duplicate block data within a single compaction group. This both provides a speedup by a factor of the number of group and allows the deduplication to be parallelized over groups. Concurrency level is taken from block-meta-fetch-concurrency as it is for other filters. In our deployment this dropped meta fetch time in the compactor from 10 hours to 6 minutes.

This does mean that, if you're using a replica label in the compactor, stores may load unnecessary blocks since they won't spot label change that will put the duplicate data in two different streams. This resolves after ignore-deletion-marks-delay when the store acknowledges that the old blocks were deleted. This seems like a good tradeoff since it only impacts a small section of data and provides a massive speedup for large buckets. It could be resolved by making the store aware of replica labels, but this diff is already getting rather large.

Also moves group key computation from DefaultGroupKey in the compact package to GroupKey on the Thanos object in the metadata package as now both the compact and block packages need to compute these.

Verification

local tests pass, and we're actively using a version of the compactor with this change (though based off the 0.22 release, before the rebase)

…ompaction groups and in parallel. Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley · 2021-12-21T20:38:23Z

As far as I can tell the e2e test failures here are not related to my changes. Those tests pass consistently when they're run in isolation locally (full runs time out, but so does a clean checkout of main).

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

…ctions groups during deduplication Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

…els before deduplication. Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

hanjm · 2021-12-26T03:27:56Z

Awesome performance improvement.🚀

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

hanjm · 2022-01-05T13:35:19Z

ping @yeya24

nberkley · 2022-01-10T20:04:50Z

@bwplotka You might care about this one (you did the review for the PR that introduced this logic)

cmd/thanos/compact.go

cmd/thanos/tools_bucket.go

pkg/block/fetcher.go

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

GiedriusS · 2022-01-13T08:03:25Z

Seems like CI failure is related.

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley · 2022-01-13T18:08:58Z

Yeah, someone added some new calls to NewStoreGW on main since last run. I'll have a fixed version up in a few minutes.

nberkley · 2022-01-13T19:17:13Z

Should be set now if someone can approve running tests.

yeya24

LGTM

yeya24

LGTM

wiardvanrij · 2022-01-18T17:32:57Z

I only did a basic check, I would recommend already adding the Changelog so this can move forward asap :)

nberkley · 2022-01-18T17:58:41Z

I only did a basic check, I would recommend already adding the Changelog so this can move forward asap :)

I thought the guideline was changelog only for user impacting changes? Happy to add one if that's not correct.

wiardvanrij · 2022-01-18T19:58:53Z

I thought the guideline was changelog only for user impacting changes? Happy to add one if that's not correct.

Hm I can imagine that is listed somewhere (if you know where, please give me a link :) )
However the top of the readme states

All notable changes to this project will be documented in this file

So, just from my perspective, I think it adds value to add this change to the readme. Let's say I have an edge-case when upgrading my Thanos version. I will search for changes related to my issue/component. It's nice to see this in the changelog.

Basically any change in components can have user impact, if something isn't correct ;p (I know, maybe a lame answer but it has nothing to do with the quality here, just in general!)

nberkley · 2022-01-18T23:08:21Z

Hm I can imagine that is listed somewhere (if you know where, please give me a link :) )

From the Changelog and Review Procedure in CONTRIBUTING.md:

If your change affects users (adds or removes feature) consider adding the item to the CHANGELOG.

Also the PR template.

But point taken. I'll add a changelog entry. Better to over communicate than under communicate.

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

bwplotka

Great job! Just minor nits, otherwise LGTM!

Thanks and sorry for delay. 🤗

pkg/block/fetcher.go

bwplotka · 2022-01-20T12:02:35Z

test/e2e/compact_test.go

-	str, err := e2ethanos.NewStoreGW(e, "1", svcConfig, "")
+
+	// Crank down the deletion mark delay since deduplication can miss blocks in the presence of replica labels it doesn't know about.
+	str, err := e2ethanos.NewStoreGW(e, "1", svcConfig, "", []string{"--ignore-deletion-marks-delay=2s"})


Do we now depend on this test to run faster than 2s?

No, it now cannot run faster than 2s.

To be more clear, this is working around the bit mentioned in the description where store gateways load duplicates for a bit in the presence of replica labels. It cranks the window during which we ignore deletion marks down so that the test can pass once we recognize the deletion marks.

pkg/block/fetcher.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley · 2022-01-27T17:42:47Z

@bwplotka Can you take another look? Think all the nits are addressed.

GiedriusS

Solid work! Thank you 💪

…mpaction group and in parallel (thanos-io#4963) * Move group key function to the metadata package. Deduplicate within compaction groups and in parallel. Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * cleanup after rebase Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * formatting Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * Reduce mutex operations in DeduplicateFilter Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * fix tests after last update from main Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * formatting Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * Move fetcher modifiers before filters so we can compute correct compactions groups during deduplication Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * Merge the notion of filters and modifiers so we can strip replica labels before deduplication. Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * formatting Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * fix tests Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * remove duplicate filter list entry Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * FIx new calls to NewStoreGW Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * add CHANGELOG Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * Apply suggestions from code review Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> * comment suggested in review Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Nicholaswang <wzhever@gmail.com>

nberkley added 2 commits December 15, 2021 17:42

Move group key function to the metadata package. Deduplicate within c…

3e370cc

…ompaction groups and in parallel. Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

cleanup after rebase

2a5605d

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

pull-request-size bot added the size/L label Dec 16, 2021

formatting

d8249fe

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley force-pushed the dedup-fix branch from 0255d35 to d8249fe Compare December 16, 2021 17:50

Reduce mutex operations in DeduplicateFilter

5a5165f

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley force-pushed the dedup-fix branch from 9303a0c to 5a5165f Compare December 17, 2021 13:57

nberkley added 2 commits December 17, 2021 08:59

Merge branch 'main' into dedup-fix

2c28c94

fix tests after last update from main

42b1f14

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley force-pushed the dedup-fix branch from c096982 to 42b1f14 Compare December 17, 2021 14:52

formatting

8391be8

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley added 5 commits December 22, 2021 06:05

Update from main for test improvements

fb3c3b3

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Move fetcher modifiers before filters so we can compute correct compa…

6de5809

…ctions groups during deduplication Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Merge the notion of filters and modifiers so we can strip replica lab…

7556d1d

…els before deduplication. Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

formatting

0a91aa2

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

fix tests

cd93e0d

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

hanjm added the component: compact label Dec 26, 2021

Update from main to pick up dock fix/fix merge conflicts

ebd0fd2

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

GiedriusS reviewed Jan 11, 2022

View reviewed changes

cmd/thanos/compact.go Outdated Show resolved Hide resolved

cmd/thanos/tools_bucket.go Show resolved Hide resolved

cmd/thanos/tools_bucket.go Show resolved Hide resolved

pkg/block/fetcher.go Show resolved Hide resolved

remove duplicate filter list entry

ab33691

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley force-pushed the dedup-fix branch from 7a486e9 to ab33691 Compare January 11, 2022 21:27

nberkley added 2 commits January 13, 2022 12:58

Merge branch 'main' into dedup-fix

9ab1573

FIx new calls to NewStoreGW

f8dac08

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

yeya24 approved these changes Jan 15, 2022

View reviewed changes

yeya24 previously approved these changes Jan 15, 2022

View reviewed changes

nberkley added 2 commits January 18, 2022 18:23

add CHANGELOG

81dd0a0

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Merge branch 'main' into dedup-fix

1455910

nberkley dismissed yeya24’s stale review via 1455910 January 18, 2022 23:57

bwplotka previously approved these changes Jan 20, 2022

View reviewed changes

nberkley dismissed bwplotka’s stale review via 928b533 January 20, 2022 18:03

Apply suggestions from code review

eb692d3

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

nberkley force-pushed the dedup-fix branch from 928b533 to eb692d3 Compare January 20, 2022 18:05

nberkley added 2 commits January 20, 2022 13:11

comment suggested in review

2ecfb52

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

update from main to resolve merge conflicts

fd60a06

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

GiedriusS approved these changes Jan 28, 2022

View reviewed changes

GiedriusS merged commit 78e923d into thanos-io:main Jan 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

compact+store+tools: Filter for duplicate block data only within a compaction group and in parallel #4963

compact+store+tools: Filter for duplicate block data only within a compaction group and in parallel #4963

nberkley commented Dec 16, 2021 •

edited

nberkley commented Dec 21, 2021

hanjm commented Dec 26, 2021 •

edited

hanjm commented Jan 5, 2022

nberkley commented Jan 10, 2022

GiedriusS commented Jan 13, 2022

nberkley commented Jan 13, 2022

nberkley commented Jan 13, 2022

yeya24 left a comment

yeya24 left a comment

wiardvanrij commented Jan 18, 2022

nberkley commented Jan 18, 2022

wiardvanrij commented Jan 18, 2022

nberkley commented Jan 18, 2022

bwplotka left a comment

bwplotka Jan 20, 2022

nberkley Jan 20, 2022

nberkley Jan 20, 2022

nberkley commented Jan 27, 2022

GiedriusS left a comment

compact+store+tools: Filter for duplicate block data only within a compaction group and in parallel #4963

compact+store+tools: Filter for duplicate block data only within a compaction group and in parallel #4963

Conversation

nberkley commented Dec 16, 2021 • edited

Changes

Verification

nberkley commented Dec 21, 2021

hanjm commented Dec 26, 2021 • edited

hanjm commented Jan 5, 2022

nberkley commented Jan 10, 2022

GiedriusS commented Jan 13, 2022

nberkley commented Jan 13, 2022

nberkley commented Jan 13, 2022

yeya24 left a comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

wiardvanrij commented Jan 18, 2022

nberkley commented Jan 18, 2022

wiardvanrij commented Jan 18, 2022

nberkley commented Jan 18, 2022

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka Jan 20, 2022

Choose a reason for hiding this comment

nberkley Jan 20, 2022

Choose a reason for hiding this comment

nberkley Jan 20, 2022

Choose a reason for hiding this comment

nberkley commented Jan 27, 2022

GiedriusS left a comment

Choose a reason for hiding this comment

nberkley commented Dec 16, 2021 •

edited

hanjm commented Dec 26, 2021 •

edited