Fixes a bug in the block cache code. #4302
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Because chunks can overlap and that can causes multiple processing of the same chunk within batches, we decided to cache the evaluation of it. However there was a bug here causing incorrect query result, specially for metric queries. The bug would occur when we would only partially consume a block (overlaping block between multiple batches) and then try to replay it. In which case it wouldn't actually replay it but keep consuming the underlaying iterator. This took us 2 days to figure out, thank Danny for helping along. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
dannykopping
approved these changes
Sep 9, 2021
pkg/iter/cache.go
Outdated
| if it.base == nil { | ||
| return false | ||
| } | ||
| ok := it.base.Next() |
There was a problem hiding this comment.
Should we not be using the Base() function instead of these direct member accesses?
There was a problem hiding this comment.
Might give us some leverage later, since we have this layer of indirection.
pkg/iter/cache.go
Outdated
| return it.base | ||
| } | ||
|
|
||
| func (it *cachedIterator) consumeNext() bool { |
There was a problem hiding this comment.
I wonder if we might find a better name for this; it's not super clear how this is different from Next() at a casual glance.
Perhaps consumeBase/consumeWrapped?
I'd personally love it if we renamed base and Base() to wrapped and Wrapped(); might be more clear?
owen-d
approved these changes
Sep 9, 2021
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because chunks can overlap and that can causes multiple processing of the same chunk within batches, we decided to cache the
evaluation of it.
However there was a bug here causing incorrect query result, specially for metric queries. The bug would occur when we would only partially consume a block (overlapping block between multiple batches) and then try to replay it.
In which case it wouldn't actually replay it but keep consuming the underlaying iterator.
I guess I assumed that blocks would only be fully consumed, but that's not correct for metric queries (forward). I added new tests that were failing with the previous code to ensure we're good now.
This took us 2 days to figure out, thank @dannykopping for helping along.
Signed-off-by: Cyril Tovena cyril.tovena@gmail.com