Enhance VarByteChunkSVForwardIndexReader to directly read from data buffer for uncompressed data #5816
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
snleee
approved these changes
Aug 5, 2020
Yes, |
| int nextRowOffset; | ||
| private int getValueEndOffset(int rowId, ByteBuffer chunkBuffer) { | ||
| if (rowId == _numDocsPerChunk - 1) { | ||
| // Last row in the trunk |
There was a problem hiding this comment.
typo; trunk -> chunk
| /** | ||
| * Helper method to compute the end offset of the value in the data buffer. | ||
| */ | ||
| private long getValueEndOffset(int chunkId, int chunkRowId, long chunkStartOffset) { |
There was a problem hiding this comment.
Why the algorithm for getting endoffset or startOffset for next row is different for uncompressed?
There was a problem hiding this comment.
Algorithm is slightly different because with the chunkBuffer we can directly get the chunkEndOffset via chunkBuffer.limit(), which is not the case for the uncompressed one. That is why we have a branch on the last chunk.
…uffer for uncompressed data
Jackie-Jiang
force-pushed
the
var_byte_chunk_reader
branch
from
d6deb16
to
2d12ff6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently for var-byte raw index, we always pre-allocate the chunk buffer even for uncompressed data (the buffer could be huge if the index contains large size value). When reading values, we first copy the data into the chunk buffer, then read from the buffer. This could cause unnecessary overhead on copying the data as well as allocating the direct memory, and can even cause OOM if the buffer size is too big. The chunk buffer is needed for compressed data in order to decompress it, but not necessary for uncompressed data as we can directly read from the data buffer.
This PR enhances the VarByteChunkSVForwardIndexReader to directly read from the data buffer for uncompressed data, and avoid the overhead of the chunk buffer.