Skip to content

[improvement](be) Optimize IN filter pruning with native HybridSet operations - #66455

Draft
mrhhsg wants to merge 7 commits into
apache:masterfrom
mrhhsg:improvement/hybrid-set-min-max-pruning
Draft

[improvement](be) Optimize IN filter pruning with native HybridSet operations#66455
mrhhsg wants to merge 7 commits into
apache:masterfrom
mrhhsg:improvement/hybrid-set-min-max-pruning

Conversation

@mrhhsg

@mrhhsg mrhhsg commented Aug 5, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

IN-filter metadata pruning previously converted HybridSet values through generic literal and Field materialization paths. This duplicated the IN set for zonemap pruning and made Bloom pruning allocate owning Field objects for every candidate.

This PR:

  • exposes typed HybridSet operations for owning min/max bounds, exact range membership, and native-value callback traversal for Bloom probes;
  • limits each zonemap snapshot to two owning non-null min/max Field values and performs exact checks against the source typed set;
  • probes file Bloom filters directly from native HybridSet values with early exit, without building a Field vector;
  • maintains fast min/max and range summaries for tinyint/smallint bitset containers;
  • keeps dictionary, ORC, and Parquet pruning behavior conservative for unsupported or mismatched types.

Empty and NULL-only IN sets, embedded-NUL strings, borrowed string storage, prepared min/max reuse across expression clones, and native zonemap, dictionary, and Bloom pruning are covered by focused tests.

Release note

None

Check List (For Author)

  • Test

    • Regression test
      • ./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p0/expr_zonemap -s test_expr_zonemap_pruning (1/1 suite passed)
      • ./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1 (1/1 suite passed)
      • ./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p0/runtime_filter -s set_operator_in_filter (1/1 suite passed)
    • Unit Test
      • ./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:HybridSetTest.IntegerMinMaxAndRangeLookup:HybridSetTest.SignedBitSetRangeLookup:HybridSetTest.StringRangeLookupPreservesEmbeddedNull:RuntimeFilterMergerTest.*:RuntimeFilterProducerTest.*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload:ParquetBloomFilterPruningTest.NativeUint32BloomUsesPhysicalInt32Hash:ParquetBloomFilterPruningTest.NativeRowGroupKeepsPresentUint32AboveInt32Max:NewOrcReaderTest.SargDirectIn*' -j 32 (67/67 tests passed)
    • Manual test
      • BUILD_TYPE=ASAN ./build.sh --be --fe --be-extension-ignore paimon-scanner,preload-extensions -j 32
      • ./build-support/clang-format.sh
      • ./build-support/check-format.sh
      • CLANG_TIDY_BINARY=/tmp/doris-clang-tidy-with-resource-dir ./build-support/run-clang-tidy.sh --base 6ddb9af90571d5ceb41d13caacd80ebbf753b913 --build-dir be/ut_build_ASAN
      • git diff --check 6ddb9af90571d5ceb41d13caacd80ebbf753b913...HEAD
      • Focused benchmark list/run for (GetMinMax|LegacyMaterialize|AnyMatchRaw) (32/32 cases completed; benchmarked code is unchanged by the final scope cleanup).
    • No need to test or manual test. Explain why:
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

mrhhsg added 5 commits August 5, 2026 10:14
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: IN-predicate metadata pruning converted each HybridSet value through TExprNode/VLiteral/Field and repeated that work across expression clones. Replace it with typed one-pass materialization, bounded zonemap point snapshots, shared owning caches, direct dictionary probing, transparent StringRef lookup, and safe runtime-filter publication. In a local Release microbenchmark, median CPU time improved by 70.99x/355.25x/455.04x/510.41x for INT32 sets of 64/65/1024/40960 values and by 119.18x/143.67x/47.53x for long-string sets of 65/1024/40960 values; cache hits were 147-151 ns.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:RuntimeFilterMergerTest.*:NewOrcReaderTest.SargDirectIn*' -j 16` (57 tests passed)
    - Local Release microbenchmark with 5 repetitions
- Behavior changed: No
- Does this need documentation: No

(cherry picked from commit 77f7f42ddae3a560347ab7bfe57f763dff422811)
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Zonemap pruning retained small IN-list candidates as Fields and made direct IN rewrites depend on that materialization. Keep zonemap snapshots min/max-only, convert zonemap bounds to native values for direct HybridSet range checks, restore the iterator-based direct IN rewrite, and isolate full Bloom-filter values behind lazy materialization.

### Release note

None

### Check List (For Author)

- Test:
    - Unit Test
    - Regression test
    - Manual benchmark
- Behavior changed: No
- Does this need documentation: No

(cherry picked from commit 7d3cf548bf310458427d1a74a3e6e9bc9888ef41)
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: IN-filter metadata pruning retained a generic cache that materialized every candidate as an owning Field vector for file Bloom-filter evaluation. Remove that cache, probe Bloom filters directly from native HybridSet values with early exit, and keep zonemap snapshots limited to exactly two min/max Fields. This reduces Bloom evaluation from N temporary owning Fields to zero while retaining a separate digest memoization cache because digest calculation sorts the complete set for each seed.

### Release note

None

### Check List (For Author)

- Test: Unit Test / Regression test / Manual test
    - `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:HybridSetTest.*:RuntimeFilterMergerTest.*:ParquetBloomFilterPruningTest.*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload:RuntimeFilterExprSamplingTest.deep_clone_clones_impl_tree:NewOrcReaderTest.SargDirectIn*:ParquetExprTest.test_in' -j 16` (79 tests passed)
    - `./run-regression-test.sh --conf output/local-regression/regression-conf-19700.groovy --run -d query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1` (1 suite passed)
    - Focused `ContainsAnyRaw` benchmark run (18 cases passed)
    - ASAN BE and benchmark binaries compiled and linked; post-build packaging stopped at the pre-existing Paimon S3Loader service check
    - `./build-support/check-format.sh`
    - `CLANG_TIDY_BINARY=/tmp/doris-clang-tidy-with-resource-dir ./build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: The native HybridSet matcher exposed a function pointer and an untyped context argument at each call site, which obscured its any-match semantics. Rename the API to any_match_raw and accept a capturing predicate through a public template facade. Keep the virtual traversal boundary allocation-free with a protected function-pointer trampoline, and express the Bloom-filter probe directly as a lambda.

### Release note

None

### Check List (For Author)

- Test: Unit Test / Manual test
    - `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:HybridSetTest.*:RuntimeFilterMergerTest.*:ParquetBloomFilterPruningTest.*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload:RuntimeFilterExprSamplingTest.deep_clone_clones_impl_tree:NewOrcReaderTest.SargDirectIn*:ParquetExprTest.test_in' -j 16` (79 tests passed)
    - Focused `AnyMatchRaw` benchmark run (18 cases passed)
    - `./build-support/check-format.sh`
    - `CLANG_TIDY_BINARY=/tmp/doris-clang-tidy-with-resource-dir ./build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: HybridSetBase exposed a generic materialize API even though zonemap pruning only retains owning non-null minimum and maximum Fields. Replace it with an explicit get_min_max API, rename the snapshot type and related zonemap state, and remove redundant status and readiness state while preserving typed range, Bloom, dictionary, clone, and borrowed-string ownership semantics.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Regression Test
    - Focused BE unit tests: 79 tests passed
    - test_expr_zonemap_pruning_p1: 1 suite passed
    - ASAN BE and FE build passed with unrelated Paimon/preload extensions excluded
    - Focused HybridSet benchmark smoke tests: 32 cases completed
- Behavior changed: No
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

mrhhsg added 2 commits August 5, 2026 18:34
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: A producer-local IN filter may be published to an instance-local consumer while the same input seeds a merger that receives more producers. Make that source usage explicit so only dual-publication inputs detach their HybridSet, while remote-only and RPC temporary inputs retain the zero-copy first-accumulator path. Add focused coverage for local versus merged publication, merger-only reuse, disabled and single-producer paths, and IN_OR_BLOOM conversion.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `./run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:RuntimeFilterMergerTest.*:RuntimeFilterProducerTest.*:NewOrcReaderTest.SargDirectIn*' -j 16`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Runtime-filter merger ownership is independent of the IN metadata pruning optimization. Remove the merger ownership changes and wrapper-level digest memoization from this change, restore digest computation from the current HybridSet payload, and remove the clone-only HybridSet API and tests. Keep the typed min/max, range lookup, and native Bloom probing improvements focused on metadata pruning.

### Release note

None

### Check List (For Author)

- Test: Unit Test, regression test, ASAN BE/FE build, formatting, and clang-tidy
    - 67 focused BE unit tests passed
    - test_expr_zonemap_pruning, test_expr_zonemap_pruning_p1, and set_operator_in_filter regression suites passed
- Behavior changed: No
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants