[enhancement](scan) Optimize Parquet V2 predicate filtering and fixed-binary decimal decoding - #66396
Open
Gabriel39 wants to merge 2 commits into
Open
[enhancement](scan) Optimize Parquet V2 predicate filtering and fixed-binary decimal decoding#66396Gabriel39 wants to merge 2 commits into
Gabriel39 wants to merge 2 commits into
Conversation
…pache#66360) Backport the selected Parquet V2 direct-predicate filtering changes from on this branch. - keep identity selection-vector state implicit and compact selected rows in bulk - retain the selection scratch high-water mark across scanner batches and specialize first compaction from implicit identity - refresh late runtime-filter requests at safe row-group boundaries - re-run footer-statistics pruning and reset adaptive predicate state for unopened row groups after a refresh - preserve real COUNT(*) carrier values while runtime filters are pending - initialize refreshed JNI predicates and attribute refresh work to TableReader/FileReader/Parquet profiles - preserve Hudi/Paimon child-reader predicate state - remove query-scoped dictionary-filter cache state - share immutable `VDirectInPredicate` pruning materialization across split-local expression clones - add correctness-checked selection and direct-IN lifecycle microbenchmarks - `./run-be-ut.sh --run --filter='FileScannerV2Test.*:*Parquet*:*TableReaderTest.*:Hudi*ReaderTest.*:Paimon*ReaderTest.*:SelectionVectorTest.*:DictionaryFilterCostTest.*' -j48` - 639 tests from 47 test suites passed under ASAN - targeted late-RF, COUNT(*), dictionary-snapshot, shared-IN-state, and SelectionVector tests - 19 tests from 5 test suites passed under ASAN - Release benchmark build and smoke run - expected registrations: 228 decoder, 92 kernel, 25 selection, 167 reader, and 8 expression-lifecycle cases - all 25 selection and 8 expression-lifecycle cases executed with zero benchmark errors - `git diff --check` The final benchmark source validates every surviving original row ID after the timed region. Base, pre-fix PR, and final binaries use the same benchmark source and Clang `-O3 -DNDEBUG -mavx2` on the same host. Each comparison uses one pinned CPU, three warmups, eight adjacent A-B-B-A quartets, and at least 0.3 seconds per invocation. The table reports median paired CPU-time ratios; negative values are improvements. | Operation | Final selectivity | Final vs pre-fix PR | Final vs branch base | |---|---:|---:|---:| | Identity initialization | 100% | -15.23% | -99.12% | | Row filter | 1% | -24.23% | -23.76% | | Row filter | 50% | -16.10% | -34.50% | | Row filter | 90% | -45.91% | -45.95% | | Row filter | 100% | -31.72% | -17.32% | | Successive filters | 1% | -33.25% | -35.79% | | Successive filters | 50% | -29.80% | -35.10% | | Successive filters | 90% | -25.27% | -25.16% | | Successive filters | 100% | -24.93% | -23.72% | All final-vs-base paired-ratio CVs are at most 5.85%. The previous 16.43%/59.94% dense row-filter regressions and 27.92%-61.05% successive-filter regressions are no longer present. Retaining `_owned` avoids repeated value initialization; the implicit-identity specialization removes the remaining source/coordinate branches from the first compaction. `FileScannerExpr/direct_in_clone_prepare_open` isolates deep-clone, prepare, and open for an already prepared direct-IN runtime filter. Set construction and the original fragment prepare/open are outside the timed region. Shared and forced-rematerialization implementations run in the same Release binary on one pinned CPU, with 10 repetitions and at least 0.5 seconds per repetition. | IN values | Rematerialize median | Shared median | Speedup | |---:|---:|---:|---:| | 128 | 207.470 us | 1.634 us | 126.9x | | 1,024 | 1.674 ms | 1.642 us | 1,019.5x | | 8,192 | 13.514 ms | 1.672 us | 8,082.2x | | 65,536 | 108.337 ms | 1.650 us | 65,663.1x | The shared path remains approximately constant because split clones reuse the immutable, fragment-originated pruning state; the rematerialization path scales linearly with runtime-filter cardinality. The earlier reader-level comparison used an identical Parquet-only Release benchmark binary on the same host, with one pinned CPU, warm cache, three warmups, A-B-B-A order, 10 repetitions, and at least one second per repetition. It covers nullable INT32 predicate scans with a lazy payload for PLAIN and dictionary encoding. | Encoding | Selectivity | CPU time vs base | Wall time vs base | |---|---:|---:|---:| | Dictionary | 1% | +0.44% | +0.46% | | Dictionary | 10% | +1.13% | +1.22% | | Dictionary | 50% | +1.41% | +1.51% | | Dictionary | 90% | +0.41% | +0.53% | | PLAIN | 1% | -0.65% | -0.66% | | PLAIN | 10% | +0.32% | +0.38% | | PLAIN | 50% | -1.21% | -1.18% | | PLAIN | 90% | -1.34% | -1.29% | The reader-level point estimates span -1.34% to +1.41% CPU time with mixed signs, so this benchmark did not detect a material aggregate regression. It starts at `format::parquet::ParquetReader`; it does not cover scanner scheduling or end-to-end SQL execution.
## Proposed changes - Decode same-scale Parquet `FIXED_LEN_BYTE_ARRAY` decimals with an `int32_t`, `int64_t`, or `Int128` source selected from the physical width instead of always using `Int256`. - Use unaligned full-width big-endian loads for 4-, 8-, and 16-byte values, while preserving sign extension for shorter widths. - Validate the target decimal precision before narrowing, preserve permissive/strict conversion behavior, and keep rescaling and wider values on the existing `Int256` path. - Add regression coverage for positive and negative precision boundaries, shorter signed inputs, strict rollback, permissive null marking, and the complete int32 source domain. ## Validation - ASAN BE unit tests: 8 tests from `DataTypeSerDeParquetTest` passed. - Release microbenchmark: 65,536 values per iteration through `DataTypeDecimalSerDe::read_column_from_parquet`, pinned to one CPU. Each stage used 3 warmups followed by 10 repetitions in ABBA order. | Target / physical width | Before median CPU | After median CPU | Speedup | CPU reduction | | --- | ---: | ---: | ---: | ---: | | Decimal32 / 4 bytes | 1,359,514 ns | 88,527 ns | 15.36x | 93.49% | | Decimal64 / 8 bytes | 1,634,757 ns | 90,609 ns | 18.04x | 94.46% | | Decimal128 / 16 bytes | 2,206,272 ns | 152,823 ns | 14.44x | 93.07% | The benchmark host was heavily loaded and CPU frequency scaling was enabled, so the exact ratios are noisy. However, the before/after median ranges did not overlap in any ABBA stage. A final optimized-build smoke run measured median CPU times of 95,077 ns, 94,635 ns, and 145,417 ns with CPU CVs of 0.41%, 1.76%, and 0.64%, respectively.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Gabriel39
marked this pull request as ready for review
August 3, 2026 14:06
Gabriel39
changed the title
Contributor
TPC-H: Total hot run time: 28910 ms |
Contributor
TPC-DS: Total hot run time: 168998 ms |
Contributor
ClickBench: Total hot run time: 23.86 s |
Contributor
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #66360, #66379
Problem Summary:
This PR brings two complementary Parquet V2 optimizations from
branch-4.1tomaster:Int256when their physical width permits a narrower native type.Parquet V2 predicate filtering (#66360)
COUNT(*)carrier values while runtime filters are pending.VDirectInPredicatepruning materialization across split-local expression clones.Fixed-binary decimal decoding (#66379)
FIXED_LEN_BYTE_ARRAYdecimals withint32_t,int64_t, orInt128, selected from the physical width, instead of always usingInt256.Int256path.Performance
The results below come from the original
branch-4.1PR benchmarks in #66360 and #66379. This PR applies the same implementation tomaster; the performance benchmarks were not re-run as part of this forward-port.Selection-vector processing
The benchmark validates every surviving original row ID after the timed region. It used the same Clang
-O3 -DNDEBUG -mavx2benchmark source for the branch base, the pre-fix PR, and the final implementation, with one pinned CPU, three warmups, eight adjacent A-B-B-A quartets, and at least 0.3 seconds per invocation. Negative percentages are improvements.Compared with the branch base, identity initialization improved by 99.12%, row filtering improved by 17.32%-45.95%, and successive filtering improved by 23.72%-35.79%. All final-vs-base paired-ratio CVs were at most 5.85%.
Direct-IN expression lifecycle
FileScannerExpr/direct_in_clone_prepare_openisolates deep-clone, prepare, and open for an already prepared direct-IN runtime filter. Set construction and the original fragment prepare/open are outside the timed region. The shared and forced-rematerialization implementations ran in the same Release binary on one pinned CPU, with 10 repetitions and at least 0.5 seconds per repetition.The shared path remains approximately constant because split-local clones reuse immutable pruning state instead of rebuilding it for every split.
Reader-level regression guardrail
The Parquet reader benchmark covered nullable INT32 predicate scans with a lazy payload for both PLAIN and dictionary encoding. Across 1%, 10%, 50%, and 90% selectivity, CPU-time changes ranged from -1.34% to +1.41% with mixed signs, so it did not detect a material aggregate reader-level regression.
Fixed-binary decimal decoding
The benchmark decoded 65,536 values per iteration through
DataTypeDecimalSerDe::read_column_from_parquet, pinned to one CPU, with three warmups followed by 10 repetitions in A-B-B-A order.The optimized path was 14.44x-18.04x faster in this benchmark, reducing CPU time by 93.07%-94.46%. The benchmark host was heavily loaded and CPU frequency scaling was enabled, so the exact ratios are noisy; however, the before/after median ranges did not overlap in any A-B-B-A stage.
Validation on master
clang-formatchecks passed.git diff --checkpassed.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)