Compact: skip compaction for blocks with no samples #904

mjd95 · 2019-03-11T14:54:26Z

Changes

During the Thanos group compactor, in the case when the Prometheus compactor has found that the resulting block would have no data, delete the source blocks (after checking they are indeed empty) and exit early. Introduce a sentinel error to mark this case, to force the caller (bucket compactor) to continue compacting the rest of that group.

It is necessary to handle the empty compaction case separately because the later parts of the group compactor result in an error if we attempt to continue.

Note that the Prometheus compactor uses the empty ULID for the "block with no samples" case:

    // When resulting Block has 0 samples
    //  * No block is written.
    //  * The source dirs are marked Deletable.
    //  * Returns empty ulid.ULID{}.
    Compact(dest string, dirs []string, open []*Block) (ulid.ULID, error)

Verification

Ran the compactor against a bucket where some of the compacted blocks would end up empty.

devnev

IIUC, the case we were observing was essentially a corrupted block, where we expect data based on the block but see none in the chunks. As we were discussing, this probably shouldn't be silenced entirely, rather the error reporting should not affect the processing of independent blocks. @bwplotka ?

bwplotka · 2019-03-11T15:24:46Z

Wow, ok I think we have 2-3 separate cases here.

When resulting Block has 0 samples case for compactor - we probably don't handle it well on Thanos. We should.
We use empty ID as "sentinel" of something totally separate - "nothing more to compact in the group", rather than compaction resulted in 0 samples block.
@devnev mention about corrupted block. How this is relevant to this issue? Because compaction of your corrupted block results in 0 samples block, right? If yes we should think.. if compactor should do that and treat them in same way as valid 0 sample blocks. I am not entirely when to check that then. The only validation we do is index validation, but we never check if chunks are already there. I really expect compaction to fail for such block, so maybe your block is not corrupted.. it is just actually empty? meta.json.. how many samples it reports?

…ndle from bucket compactor

bwplotka

Generally OK, just some suggestions.

pkg/compact/compact.go

bwplotka · 2019-03-11T17:41:33Z

pkg/compact/compact.go

@@ -34,6 +34,7 @@ const (
 )

 var blockTooFreshSentinelError = errors.New("Block too fresh")
+var emptyBlockSentinelULID = ulid.MustNew(123, nil)


This sentinel is for noMoreCompactionForGroup I guess? can we name it like this or something similar?

OK so I think there is misformation here , I would prefer something opposite like noMoreCompactionForGroup instead of this? What do you think? OR even bool flag return argument to make it more explicit (:

I used a separate bool flag as I think that's the clearest what's happening. The naming was a little awkward but hopefully it makes sense, let me know what you think :)

bwplotka · 2019-03-12T15:23:33Z

pkg/compact/compact.go

@@ -513,23 +513,23 @@ func (cg *Group) Resolution() int64 {

 // Compact plans and runs a single compaction against the group. The compacted result
 // is uploaded into the bucket the blocks were retrieved from.
-func (cg *Group) Compact(ctx context.Context, dir string, comp tsdb.Compactor) (ulid.ULID, error) {
+func (cg *Group) Compact(ctx context.Context, dir string, comp tsdb.Compactor) (bool, ulid.ULID, error) {


Do we need ulid.ULID now?

I originally tried to remove the ulid.ULID but had to add it back in as we need it for compact_e2e_test.go (as written right now). In the case when the compactor actually does something, the test asserts that the new block was actually uploaded, and it needs the id for that.

I thought about changing the test, but it seemed worthwhile keeping the coverage for the upload stage.

sure, let's not end in rabbit whole here.

bwplotka

I think we can stick to (bool, error) return args now, right @mjd95 ?

bwplotka

LGTM, thanks.

One small nit, but we can fix it later on, not a blocker

bwplotka · 2019-03-12T16:53:59Z

pkg/compact/compact.go

@@ -513,23 +513,23 @@ func (cg *Group) Resolution() int64 {

 // Compact plans and runs a single compaction against the group. The compacted result
 // is uploaded into the bucket the blocks were retrieved from.
-func (cg *Group) Compact(ctx context.Context, dir string, comp tsdb.Compactor) (ulid.ULID, error) {
+func (cg *Group) Compact(ctx context.Context, dir string, comp tsdb.Compactor) (bool, ulid.ULID, error) {


sure, let's not end in rabbit whole here.

bwplotka · 2019-03-12T16:54:46Z

pkg/compact/compact.go

 	}

-	compID, err := cg.compact(ctx, subDir, comp)
+	shouldRerun, compID, err := cg.compact(ctx, subDir, comp)


I don't like this as this indicate that something wrong happen and we need to retry.. non empty ulid was actually more sensible maybe? Or at least we should change var name... Not a blocker.

Initial review

* skip compaction for blocks with no samples * update to actually delete the empty input blocks, and to correctly handle from bucket compactor * warn on error deleting empty block * use ULID instead of error for emptyBlockSentinel * don't use a global variable * full stop at end of comment * use boolean to indicate whether there is more compaction work * rename variables * fix test

skip compaction for blocks with no samples

c632532

devnev previously requested changes Mar 11, 2019

View reviewed changes

Martin Dickson added 2 commits March 11, 2019 16:48

update to actually delete the empty input blocks, and to correctly ha…

79b98ed

…ndle from bucket compactor

warn on error deleting empty block

eb2b5df

bwplotka reviewed Mar 11, 2019

View reviewed changes

pkg/compact/compact.go Outdated Show resolved Hide resolved

pkg/compact/compact.go Outdated Show resolved Hide resolved

Martin Dickson added 3 commits March 11, 2019 17:37

use ULID instead of error for emptyBlockSentinel

05bd7fd

don't use a global variable

7453588

full stop at end of comment

b0ac1b7

bwplotka reviewed Mar 11, 2019

View reviewed changes

Martin Dickson added 3 commits March 12, 2019 11:27

use boolean to indicate whether there is more compaction work

385a991

rename variables

a731d85

fix test

aedea91

bwplotka reviewed Mar 12, 2019

View reviewed changes

bwplotka requested changes Mar 12, 2019

View reviewed changes

bwplotka approved these changes Mar 12, 2019

View reviewed changes

bwplotka merged commit bc3aaab into thanos-io:master Mar 12, 2019

GiedriusS mentioned this pull request Mar 14, 2019

compact: failure on empty block #869

Closed

mjd95 deleted the skip-compaction-for-blocks-with-no-samples branch April 12, 2019 10:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compact: skip compaction for blocks with no samples #904

Compact: skip compaction for blocks with no samples #904

mjd95 commented Mar 11, 2019 •

edited

Loading

devnev left a comment

bwplotka commented Mar 11, 2019 •

edited

Loading

bwplotka left a comment

bwplotka Mar 11, 2019

bwplotka Mar 11, 2019

mjd95 Mar 12, 2019

bwplotka Mar 12, 2019

mjd95 Mar 12, 2019

bwplotka Mar 12, 2019

bwplotka left a comment

bwplotka left a comment

bwplotka Mar 12, 2019

bwplotka Mar 12, 2019

Compact: skip compaction for blocks with no samples #904

Compact: skip compaction for blocks with no samples #904

Conversation

mjd95 commented Mar 11, 2019 • edited Loading

Changes

Verification

devnev left a comment

Choose a reason for hiding this comment

bwplotka commented Mar 11, 2019 • edited Loading

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjd95 commented Mar 11, 2019 •

edited

Loading

bwplotka commented Mar 11, 2019 •

edited

Loading