skip blocks with out-of-order chunk during compaction #4469

huyan0 · 2021-07-22T15:59:23Z

Signed-off-by: Yang Hu yhuz@amazon.com

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

add a check for OOO chunks in block index, and mark such block for no compaction, and continue on with the compaction
add testutil to generate an index with OOO chunks

Verification

verified the exepected behavior with compact_e2e_test.go

Addresses: #3442
cc: @yeya24 @alvinlin123 @roystchiang

yeya24

Love it! Super high-quality code. Thanks for the contribution @huyan0

cmd/thanos/compact.go

alvinlin123 · 2021-07-22T18:48:21Z

pkg/block/index.go

-
-	if i.OutOfOrderSeries > 0 {
-		errMsg = append(errMsg, fmt.Sprintf(
+func (i HealthStats) OutOfOrderChunksErr() error {


I am wondering, do you think we can just skip compaction for all the CriticalErr; instead of just OutOfOrderChunksErr?

I think it's okay on the compact side given we cannot generate more blocks than not compacting...I don't think there's much implication on the retrieval side either but would like to hear what @yeya24 thinks : )
@yeya24 do you have some context on why the project decided to halt compaction in the first place?

@bwplotka do you have some context on the discussion? :)

huyan0 · 2021-07-27T16:16:16Z

I have updated the metric counter, will add to changelog if there's no further comments for now @yeya24

yeya24

This pr LGTM. But it changes the default behavior and might be critical so it would be good to hear from other maintainers.

cc @thanos-io/thanos-maintainers

wiardvanrij · 2021-07-29T07:36:33Z

The code looks solid and you referenced an issue. If I understand correctly, we now mark it as corrupted and skip it. That still leaves a block corrupted. Are there any follow-up steps that a user could do? Basically I would really love to have some documentation on:

How could one find corrupted blocks?
What is the impact for the user if that happens?
Could one still fix a corrupted block and let it get compacted after the fact?
anything else that provides context to users

This would IMO be the cherry on top :)

bwplotka · 2021-07-29T17:46:12Z

TODO: Can we check if those blocks are readable by Thanos Store/Cortex Store

bwplotka · 2021-07-29T17:48:29Z

TODO: Review & check for consistency with other issues and how we handle them (:

huyan0 · 2021-07-30T23:43:13Z

@bwplotka regarding your two comments:

the blocks are still readable because the mark that we add to the block is for compact only, and the store API has no requirement on whether the chunks are sorted: link
We handle this error differently than the other errors encountered during compaction; for those we halt the compaction. Other than OOO chunks, the two other specific errors compact check for are these , for which we attempt to repair or compacts if a feature flag is set, so I don't see an overarching consistent approach to adhere to, other than logging the occurrence to provide context for user. This is understandable since these errors are unexpected and specific. I agree a deep dive into the root cause is needed to address this in long term to come up with a fix for the root cause.

huyan0 · 2021-08-09T16:19:26Z

PR still pending review :) Any updates needed?

cmd/thanos/compact.go

yeya24 · 2021-08-09T19:57:27Z

Btw let's update the changelog and we are ready to go I think

yeya24 · 2021-08-11T07:14:33Z

@huyan0 Sorry for the late review. I will merge the pr after you resolve the conflict.

huyan0 · 2021-08-11T15:01:19Z

hello, I have rebased : )

yeya24

LGTM

yeya24 · 2021-08-11T22:16:09Z

Looks like the E2E is still failing and caused by this pr.

21:58:23 Killing sidecar-remote-and-sidecar
21:58:23 compact-expect-to-halt: panic: inconsistent label cardinality: expected 2 label values but got 1 in []string{"deletion-mark.json"}
21:58:23 compact-expect-to-halt: goroutine 1 [running]:
21:58:23 compact-expect-to-halt: github.com/prometheus/client_golang/prometheus.(*CounterVec).WithLabelValues(0xc00000f9b8, 0xc000887840, 0x1, 0x1, 0x2035af0, 0xc00019f320)
21:58:23 compact-expect-to-halt: /home/runner/go/pkg/mod/github.com/prometheus/client_golang@v1.10.0/prometheus/counter.go:248 +0xff
21:58:23 compact-expect-to-halt: main.newCompactMetrics(0xc0001a0640, 0x9d29229e0000, 0x1ba7500)
21:58:23 compact-expect-to-halt: /home/runner/work/thanos/thanos/cmd/thanos/compact.go:152 +0x5cd
21:58:23 compact-expect-to-halt: main.runCompact(0xc00049d5a8, 0x1ff4040, 0xc000889410, 0x201c490, 0x2e02e80, 0xc0001a0640, 0x1ff8d80, 0xc0008876b0, 0x1, 0x4, ...)
21:58:23 compact-expect-to-halt: /home/runner/work/thanos/thanos/cmd/thanos/compact.go:171 +0xc5
21:58:23 compact-expect-to-halt: main.registerCompact.func1(0xc00049d5a8, 0x1ff4040, 0xc000889410, 0xc0001a0640, 0x201c490, 0x2e02e80, 0xc000083e60, 0x0, 0x4169fb, 0x7f5f11773100)

yeya24

There might be other panics caused by that deletion marker metric. Please check and fix them. Thanks for the patience.

cmd/thanos/compact.go

pkg/compact/compact.go

yeya24 · 2021-08-12T00:18:45Z

Can you do a rebase on latest main rather than including changes from another pr.

CHANGELOG.md

pkg/compact/compact.go

pkg/testutil/testutil.go

yeya24

LGTM

huyan0 · 2021-08-12T17:33:05Z

looking... Sorry for the back and forth. Will try to address it

Signed-off-by: Yang Hu <yhuz@amazon.com>

yeya24

LGTM! Thanks for the contribution.

huyan0 force-pushed the main branch from e5a0a29 to 6ab2ca7 Compare July 22, 2021 17:44

yeya24 reviewed Jul 22, 2021

View reviewed changes

cmd/thanos/compact.go Outdated Show resolved Hide resolved

alvinlin123 reviewed Jul 22, 2021

View reviewed changes

huyan0 force-pushed the main branch 2 times, most recently from 887afae to 0532597 Compare July 23, 2021 14:20

huyan0 marked this pull request as ready for review July 23, 2021 19:19

huyan0 requested a review from yeya24 July 23, 2021 19:19

huyan0 force-pushed the main branch from 0532597 to f8b52a4 Compare July 27, 2021 16:14

yeya24 previously approved these changes Jul 28, 2021

View reviewed changes

huyan0 dismissed yeya24’s stale review via 42740f4 July 30, 2021 23:19

huyan0 force-pushed the main branch from f8b52a4 to 42740f4 Compare July 30, 2021 23:19

yeya24 reviewed Aug 9, 2021

View reviewed changes

cmd/thanos/compact.go Outdated Show resolved Hide resolved

huyan0 force-pushed the main branch from 42740f4 to 05aa833 Compare August 10, 2021 17:29

huyan0 force-pushed the main branch from 05aa833 to 903e40f Compare August 11, 2021 15:00

huyan0 force-pushed the main branch from 903e40f to 6197376 Compare August 11, 2021 18:13

yeya24 enabled auto-merge (squash) August 11, 2021 18:32

yeya24 approved these changes Aug 11, 2021

View reviewed changes

yeya24 requested changes Aug 11, 2021

View reviewed changes

cmd/thanos/compact.go Show resolved Hide resolved

cmd/thanos/compact.go Outdated Show resolved Hide resolved

cmd/thanos/compact.go Outdated Show resolved Hide resolved

pkg/compact/compact.go Outdated Show resolved Hide resolved

yeya24 disabled auto-merge August 11, 2021 22:36

huyan0 force-pushed the main branch from 6197376 to 5760c0b Compare August 12, 2021 00:11

yeya24 reviewed Aug 12, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

huyan0 force-pushed the main branch 2 times, most recently from b4405a2 to 18efebe Compare August 12, 2021 00:31

yeya24 reviewed Aug 12, 2021

View reviewed changes

pkg/compact/compact.go Outdated Show resolved Hide resolved

pkg/testutil/testutil.go Outdated Show resolved Hide resolved

pkg/testutil/testutil.go Outdated Show resolved Hide resolved

huyan0 force-pushed the main branch 2 times, most recently from 2abf49b to b0fbe2d Compare August 12, 2021 16:36

huyan0 requested a review from yeya24 August 12, 2021 16:58

yeya24 enabled auto-merge (squash) August 12, 2021 17:26

yeya24 previously approved these changes Aug 12, 2021

View reviewed changes

skip blocks with out-of-order chunk during compaction

d1da61c

Signed-off-by: Yang Hu <yhuz@amazon.com>

auto-merge was automatically disabled August 12, 2021 17:36
Head branch was pushed to by a user without write access

huyan0 dismissed yeya24’s stale review via d1da61c August 12, 2021 17:36

huyan0 force-pushed the main branch from b0fbe2d to d1da61c Compare August 12, 2021 17:36

huyan0 requested a review from yeya24 August 12, 2021 19:28

yeya24 enabled auto-merge (squash) August 13, 2021 03:58

yeya24 approved these changes Aug 13, 2021

View reviewed changes

yeya24 merged commit 07b377f into thanos-io:main Aug 13, 2021

This was referenced Aug 13, 2021

Fix -compact.skip-block-with-out-of-order-chunks #4566

Merged

Upgrade Thanos and simplify store-gateway metrics grafana/mimir#142

Merged

huyan0 mentioned this pull request Aug 16, 2021

update thanos to include skip out of order chunks in compactor cortexproject/cortex#4423

Closed

3 tasks

huyan0 mentioned this pull request Sep 1, 2021

upgrade thanos that include skip bad blocks during compaction cortexproject/cortex#4453

Closed

3 tasks

AndrewNiven1 mentioned this pull request Nov 30, 2021

Improve handling of out-of-order chunks #4916

Closed

shybbko mentioned this pull request Mar 30, 2022

Compactor is unable to properly compact blocks cortexproject/cortex#4677

Closed

alanprot mentioned this pull request Apr 7, 2022

Add shuffle-sharding for the compactor cortexproject/cortex#4433

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

skip blocks with out-of-order chunk during compaction #4469

skip blocks with out-of-order chunk during compaction #4469

huyan0 commented Jul 22, 2021 •

edited

yeya24 left a comment

alvinlin123 Jul 22, 2021

huyan0 Jul 23, 2021 •

edited

huyan0 Jul 30, 2021

huyan0 commented Jul 27, 2021 •

edited

yeya24 left a comment

wiardvanrij commented Jul 29, 2021

bwplotka commented Jul 29, 2021

bwplotka commented Jul 29, 2021

huyan0 commented Jul 30, 2021 •

edited

huyan0 commented Aug 9, 2021

yeya24 commented Aug 9, 2021

yeya24 commented Aug 11, 2021

huyan0 commented Aug 11, 2021

yeya24 left a comment

yeya24 commented Aug 11, 2021

yeya24 left a comment

yeya24 commented Aug 12, 2021

yeya24 left a comment

huyan0 commented Aug 12, 2021

yeya24 left a comment

skip blocks with out-of-order chunk during compaction #4469

skip blocks with out-of-order chunk during compaction #4469

Conversation

huyan0 commented Jul 22, 2021 • edited

Changes

Verification

yeya24 left a comment

Choose a reason for hiding this comment

alvinlin123 Jul 22, 2021

Choose a reason for hiding this comment

huyan0 Jul 23, 2021 • edited

Choose a reason for hiding this comment

huyan0 Jul 30, 2021

Choose a reason for hiding this comment

huyan0 commented Jul 27, 2021 • edited

yeya24 left a comment

Choose a reason for hiding this comment

wiardvanrij commented Jul 29, 2021

bwplotka commented Jul 29, 2021

bwplotka commented Jul 29, 2021

huyan0 commented Jul 30, 2021 • edited

huyan0 commented Aug 9, 2021

yeya24 commented Aug 9, 2021

yeya24 commented Aug 11, 2021

huyan0 commented Aug 11, 2021

yeya24 left a comment

Choose a reason for hiding this comment

yeya24 commented Aug 11, 2021

yeya24 left a comment

Choose a reason for hiding this comment

yeya24 commented Aug 12, 2021

yeya24 left a comment

Choose a reason for hiding this comment

huyan0 commented Aug 12, 2021

yeya24 left a comment

Choose a reason for hiding this comment

huyan0 commented Jul 22, 2021 •

edited

huyan0 Jul 23, 2021 •

edited

huyan0 commented Jul 27, 2021 •

edited

huyan0 commented Jul 30, 2021 •

edited