feat: Implement merge reader #225
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
evenyag
force-pushed
the
feat/merge-iter
branch
2 times, most recently
from
63f536a
to
d3b425d
Compare
evenyag
force-pushed
the
feat/merge-iter
branch
from
d3b425d
to
220688f
Compare
Now the ChunkReaderImpl use MergeReader as default reader. Also add more tests to MergeReader.
evenyag
force-pushed
the
feat/merge-iter
branch
from
66483b8
to
4b6f56d
Compare
Ports the doc comments from kudu to merge.rs to describe the idea of the merge algorithm we used.
evenyag
force-pushed
the
feat/merge-iter
branch
from
4b6f56d
to
ff5b9f6
Compare
Codecov Report
@@ Coverage Diff @@
## develop #225 +/- ##
===========================================
+ Coverage 82.97% 83.32% +0.34%
===========================================
Files 274 275 +1
Lines 22099 22580 +481
===========================================
+ Hits 18336 18814 +478
- Misses 3763 3766 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Panics if number of columns in batch is not equal to `BatchBuilder`'s
MichaelScofield
approved these changes
Sep 8, 2022
killme2008
reviewed
Sep 8, 2022
paomian
pushed a commit
to paomian/greptimedb
that referenced
this pull request
Oct 19, 2023
* feat: Check columns when constructing Batch * feat: Merge reader skeleton * test: Add tests for MergeReader * feat: Use get_ref to compare row * feat: Implement MergeReader * test: Add more tests * feat: Use MergeReader to implement ChunkReader Now the ChunkReaderImpl use MergeReader as default reader. Also add more tests to MergeReader. * docs: Describe the merge algo in merge.rs Ports the doc comments from kudu to merge.rs to describe the idea of the merge algorithm we used. * test: Fix unit tests * chore: Address CR comments Panics if number of columns in batch is not equal to `BatchBuilder`'s * chore: Address CR comments * chore: Implement Debug and add test for Node
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR implements the
MergeReaderwhich supports sort and merge rows from multiple source readers/iterators by the row key, timestamp, sequence and op_typeMergeReaderimplementation are ported from kudu's MergeIterator and CeresDB's MergeIteratorMergeReadercould be constructed via theMergeReaderBuildercompare_row_of_batch()toStoreSchemato compare key and internal columns between two rowsBatchBuilderto buildBatchfrom parts of anotherBatch, based on prior works in feat: Adds push_value_ref and extend_slice_of to MutableVector #215ChunkReaderImplused to useConcatReadernow is changed to useMergeReaderto combine inputsRelated Issues