[feat](be) add PLAIN_ENCODING_V3 binary plain page with contiguous data + lengths trailer#63570
Open
csun5285 wants to merge 1 commit into
Open
[feat](be) add PLAIN_ENCODING_V3 binary plain page with contiguous data + lengths trailer#63570csun5285 wants to merge 1 commit into
csun5285 wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
…ta + lengths trailer V3 layout: |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)| Compared to V2 (length and data interleaved per entry), V3 lets the pre-decoder memcpy the entire binary payload in a single shot and walk the contiguous varuint length block once to fill the V1 offsets array, with no data-pointer-vs-length-pointer dependency between the two passes. Backward-compat: V3 is registered as a new EncodingTypePB (= 9). Existing segments persist their per-column encoding meta (V1=2, V2=8, V3=9), so the read path dispatches to the matching pre-decoder. Old V2 segments continue to be served by BinaryPlainPageV2PreDecoder. Wired through the entire write path: - encoding_info.cpp: Hook 1b mirrors the V2 hook, rewriting PLAIN_ENCODING to PLAIN_ENCODING_V3 when the schema preference is BINARY_PLAIN_ENCODING_V3. - segment_writer.cpp + vertical_segment_writer.cpp: row-store-column path switch-ifies over BinaryPlainEncodingTypePB to pick PLAIN_ENCODING_V3. - binary_dict_page.cpp: dict word page and fallback binary page use a small shared helper to map the preference to the on-disk encoding. - tablet_meta.cpp: new TStorageFormat::V3 tablets default to BINARY_PLAIN_ENCODING_V3 for both data schema and row binlog schema. Tests: 15 BinaryPlainPageV3Test cases covering encode/decode roundtrip, seek, read_by_rowids, empty page, page_full, large N, mixed lengths (including unicode), reset, varint length boundaries (127/128/16383/16384 byte values across 1/2/3-byte varint bands), raw trailer layout assertions, and two corruption-rejection cases. All pass under ASAN. Benchmark (15-rep x 2s median, V3 / V2 speedup at 256 KiB page): 8B: 3.56x 16B: 3.07x 32B: 2.46x 64B: 2.63x 128B: 2.25x 256B: 1.39x 512B: 1.22x 1024B: 1.11x 4096B: 1.01x V3 strictly does not lose to V2 across the tested grid. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 31523 ms |
Contributor
TPC-H: Total hot run time: 31963 ms |
Contributor
TPC-DS: Total hot run time: 172821 ms |
Contributor
TPC-DS: Total hot run time: 173713 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
V3 layout: |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)|
Compared to V2 (length and data interleaved per entry), V3 lets the pre-decoder memcpy the entire binary payload in a single shot and walk the contiguous varuint length block once to fill the V1 offsets array, with no data-pointer-vs-length-pointer dependency between the two passes.
Benchmark (15-rep x 2s median, V3 / V2 speedup at 256 KiB page):
8B: 3.56x 16B: 3.07x 32B: 2.46x 64B: 2.63x
128B: 2.25x 256B: 1.39x 512B: 1.22x 1024B: 1.11x 4096B: 1.01x
V3 strictly does not lose to V2 across the tested grid.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)