storage: fix missing logs with batched chunk iterator #1299
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
owen-d
reviewed
Nov 25, 2019
pkg/storage/iterator_test.go
Outdated
| for name, tt := range tests { | ||
| tt := tt | ||
| t.Run(fmt.Sprintf("large-batchsize/%s", name), func(t *testing.T) { | ||
| it := newBatchChunkIterator(context.Background(), tt.chunks, 1000, newMatchers(tt.matchers), nil, newQuery("", tt.start, tt.end, tt.direction)) |
There was a problem hiding this comment.
Could you explain the benefit of the large batchsize tests? Additionally, would it be better to use len(tt.chunks) instead of 1000 here for extensibility to ensure the the batch is always large enough to hold all the chunks?
There was a problem hiding this comment.
To be honest I don't know whether these tests are needed or not. I added these tests for debugging purposes since the bug described in #1187 don't happen when the batchsize is big enough to hold all of the chunks. I can remove those if deemed unnecessary.
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>
owen-d
approved these changes
Nov 26, 2019
|
Nice work @putrasattvika! |
cyriltovena
approved these changes
Nov 26, 2019
There was a problem hiding this comment.
Impressive @putrasattvika.
Thank you for effort of diving into it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR addresses bugs that cause missing logs issue described in #1187:
__name__label matcher for queries against the chunk storage. The__name__chunk label is removed from all chunks that are passed intobuildHeapIterator(). Overlapping chunks (according tobatchSize) will go through that function multiple times, and they will be filtered out on the second call since they no longer have the chunk label.[from, through)to(from, through]. This fixes an issue where the last log entry of the first chunk of a batch is not retrieved.from/throughfor batched iterator with more than two batchesWhich issue(s) this PR fixes:
Fixes #1187
Checklist