[improvement](be) Optimize IN filter pruning with native HybridSet operations - #66455
Draft
mrhhsg wants to merge 7 commits into
Draft
[improvement](be) Optimize IN filter pruning with native HybridSet operations#66455mrhhsg wants to merge 7 commits into
mrhhsg wants to merge 7 commits into
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: IN-predicate metadata pruning converted each HybridSet value through TExprNode/VLiteral/Field and repeated that work across expression clones. Replace it with typed one-pass materialization, bounded zonemap point snapshots, shared owning caches, direct dictionary probing, transparent StringRef lookup, and safe runtime-filter publication. In a local Release microbenchmark, median CPU time improved by 70.99x/355.25x/455.04x/510.41x for INT32 sets of 64/65/1024/40960 values and by 119.18x/143.67x/47.53x for long-string sets of 65/1024/40960 values; cache hits were 147-151 ns.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:RuntimeFilterMergerTest.*:NewOrcReaderTest.SargDirectIn*' -j 16` (57 tests passed)
- Local Release microbenchmark with 5 repetitions
- Behavior changed: No
- Does this need documentation: No
(cherry picked from commit 77f7f42ddae3a560347ab7bfe57f763dff422811)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Zonemap pruning retained small IN-list candidates as Fields and made direct IN rewrites depend on that materialization. Keep zonemap snapshots min/max-only, convert zonemap bounds to native values for direct HybridSet range checks, restore the iterator-based direct IN rewrite, and isolate full Bloom-filter values behind lazy materialization.
### Release note
None
### Check List (For Author)
- Test:
- Unit Test
- Regression test
- Manual benchmark
- Behavior changed: No
- Does this need documentation: No
(cherry picked from commit 7d3cf548bf310458427d1a74a3e6e9bc9888ef41)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: IN-filter metadata pruning retained a generic cache that materialized every candidate as an owning Field vector for file Bloom-filter evaluation. Remove that cache, probe Bloom filters directly from native HybridSet values with early exit, and keep zonemap snapshots limited to exactly two min/max Fields. This reduces Bloom evaluation from N temporary owning Fields to zero while retaining a separate digest memoization cache because digest calculation sorts the complete set for each seed.
### Release note
None
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:HybridSetTest.*:RuntimeFilterMergerTest.*:ParquetBloomFilterPruningTest.*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload:RuntimeFilterExprSamplingTest.deep_clone_clones_impl_tree:NewOrcReaderTest.SargDirectIn*:ParquetExprTest.test_in' -j 16` (79 tests passed)
- `./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1` (1 suite passed)
- Focused `ContainsAnyRaw` benchmark run (18 cases passed)
- ASAN BE and benchmark binaries compiled and linked; post-build packaging stopped at the pre-existing Paimon S3Loader service check
- `./build-support/check-format.sh`
- `CLANG_TIDY_BINARY=/tmp/doris-clang-tidy-with-resource-dir ./build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The native HybridSet matcher exposed a function pointer and an untyped context argument at each call site, which obscured its any-match semantics. Rename the API to any_match_raw and accept a capturing predicate through a public template facade. Keep the virtual traversal boundary allocation-free with a protected function-pointer trampoline, and express the Bloom-filter probe directly as a lambda.
### Release note
None
### Check List (For Author)
- Test: Unit Test / Manual test
- `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:HybridSetTest.*:RuntimeFilterMergerTest.*:ParquetBloomFilterPruningTest.*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload:RuntimeFilterExprSamplingTest.deep_clone_clones_impl_tree:NewOrcReaderTest.SargDirectIn*:ParquetExprTest.test_in' -j 16` (79 tests passed)
- Focused `AnyMatchRaw` benchmark run (18 cases passed)
- `./build-support/check-format.sh`
- `CLANG_TIDY_BINARY=/tmp/doris-clang-tidy-with-resource-dir ./build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: HybridSetBase exposed a generic materialize API even though zonemap pruning only retains owning non-null minimum and maximum Fields. Replace it with an explicit get_min_max API, rename the snapshot type and related zonemap state, and remove redundant status and readiness state while preserving typed range, Bloom, dictionary, clone, and borrowed-string ownership semantics.
### Release note
None
### Check List (For Author)
- Test: Unit Test and Regression Test
- Focused BE unit tests: 79 tests passed
- test_expr_zonemap_pruning_p1: 1 suite passed
- ASAN BE and FE build passed with unrelated Paimon/preload extensions excluded
- Focused HybridSet benchmark smoke tests: 32 cases completed
- Behavior changed: No
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: A producer-local IN filter may be published to an instance-local consumer while the same input seeds a merger that receives more producers. Make that source usage explicit so only dual-publication inputs detach their HybridSet, while remote-only and RPC temporary inputs retain the zero-copy first-accumulator path. Add focused coverage for local versus merged publication, merger-only reuse, disabled and single-producer paths, and IN_OR_BLOOM conversion.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:RuntimeFilterMergerTest.*:RuntimeFilterProducerTest.*:NewOrcReaderTest.SargDirectIn*' -j 16`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Runtime-filter merger ownership is independent of the IN metadata pruning optimization. Remove the merger ownership changes and wrapper-level digest memoization from this change, restore digest computation from the current HybridSet payload, and remove the clone-only HybridSet API and tests. Keep the typed min/max, range lookup, and native Bloom probing improvements focused on metadata pruning.
### Release note
None
### Check List (For Author)
- Test: Unit Test, regression test, ASAN BE/FE build, formatting, and clang-tidy
- 67 focused BE unit tests passed
- test_expr_zonemap_pruning, test_expr_zonemap_pruning_p1, and set_operator_in_filter regression suites passed
- Behavior changed: No
- Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
IN-filter metadata pruning previously converted
HybridSetvalues through generic literal andFieldmaterialization paths. This duplicated the IN set for zonemap pruning and made Bloom pruning allocate owningFieldobjects for every candidate.This PR:
HybridSetoperations for owning min/max bounds, exact range membership, and native-value callback traversal for Bloom probes;Fieldvalues and performs exact checks against the source typed set;HybridSetvalues with early exit, without building aFieldvector;Empty and NULL-only IN sets, embedded-NUL strings, borrowed string storage, prepared min/max reuse across expression clones, and native zonemap, dictionary, and Bloom pruning are covered by focused tests.
Release note
None
Check List (For Author)
Test
./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p0/expr_zonemap -s test_expr_zonemap_pruning(1/1 suite passed)./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1(1/1 suite passed)./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p0/runtime_filter -s set_operator_in_filter(1/1 suite passed)./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:HybridSetTest.IntegerMinMaxAndRangeLookup:HybridSetTest.SignedBitSetRangeLookup:HybridSetTest.StringRangeLookupPreservesEmbeddedNull:RuntimeFilterMergerTest.*:RuntimeFilterProducerTest.*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload:ParquetBloomFilterPruningTest.NativeUint32BloomUsesPhysicalInt32Hash:ParquetBloomFilterPruningTest.NativeRowGroupKeepsPresentUint32AboveInt32Max:NewOrcReaderTest.SargDirectIn*' -j 32(67/67 tests passed)BUILD_TYPE=ASAN ./build.sh --be --fe --be-extension-ignore paimon-scanner,preload-extensions -j 32./build-support/clang-format.sh./build-support/check-format.shCLANG_TIDY_BINARY=/tmp/doris-clang-tidy-with-resource-dir ./build-support/run-clang-tidy.sh --base 6ddb9af90571d5ceb41d13caacd80ebbf753b913 --build-dir be/ut_build_ASANgit diff --check 6ddb9af90571d5ceb41d13caacd80ebbf753b913...HEAD(GetMinMax|LegacyMaterialize|AnyMatchRaw)(32/32 cases completed; benchmarked code is unchanged by the final scope cleanup).Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)