Skip to content

[improvement](parquet) Fuse dictionary selection planning - #66508

Open
Gabriel39 wants to merge 6 commits into
apache:branch-4.1from
Gabriel39:dev/parquet-v2-dictionary-selection-plan-4.1
Open

[improvement](parquet) Fuse dictionary selection planning#66508
Gabriel39 wants to merge 6 commits into
apache:branch-4.1from
Gabriel39:dev/parquet-v2-dictionary-selection-plan-4.1

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Predicate-only dictionary filtering currently expands definition levels and row selection into ColumnSelectVector, then scans that row-oriented state again to rebuild physical selection ranges and the selected NULL layout. Fragmented nullable filters spend significant CPU in these two planning passes before dictionary IDs are decoded.

Same-column conjuncts are combined into one dictionary-id bitmap before row decoding. Their incoming row selection therefore remains identity, which previously kept this common shape on the two-pass planner.

What is changed?

  • Build physical dictionary selection ranges, selected NULL layout, and the filtered-row count directly while merging definition-level runs with the input filter.
  • Pass the prepared plan into dictionary-index filtering instead of expanding and rescanning ColumnSelectVector.
  • Plan identity-input dictionary selections directly when definition levels are materially fragmented.
  • Use a bounded run-search fast path for no-NULL clustered filters, then switch to one linear scan after 32 runs.
  • Keep small batches, dense nullable inputs, compact identity inputs, and sparse clustered nullable inputs on the existing planner.
  • Add a self-validating 220-case dictionary-selection benchmark matrix, including 20 identity-input cases.

This PR does not change predicate decomposition, supported data types, decode strategy, or merge-read behavior. It is stacked on #66504.

Performance

Release build, 65,536 rows, CPU-pinned runs with 7 repetitions. The table reports median CPU time; lower is better.

Input selection NULL layout NULL rate Existing planner Selected planner Change
Identity Alternating 1% 11,066 ns 8,383 ns -24.24%
Identity Alternating 10% 98,943 ns 60,185 ns -39.17%
Identity Alternating 50% 580,955 ns 319,267 ns -45.04%
Identity Alternating 90% 87,733 ns 56,741 ns -35.33%
Identity No NULL 0% 1,451 ns 1,458 ns +0.47%
Identity Clustered 1% 1,488 ns 1,497 ns +0.59%

The last two negative controls are routed to the existing planner; their sub-1% differences are measurement noise from identical code paths.

Validation

  • 7 targeted ASAN Parquet dictionary and nullable-selection tests passed.
  • All 220 dictionary-selection scenarios completed and self-validated identical physical ranges, selected NULL layout, and filtered-row counts.
  • All affected C/C++ files pass clang-format 16.

@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner August 5, 2026 15:28
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31020463260

Please trigger /review again after that time.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31065828575

Please trigger /review again after that time.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 65.23% (362/555) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.33% (24737/42411)
Line Coverage 42.59% (249504/585801)
Region Coverage 38.64% (197096/510130)
Branch Coverage 40.03% (90279/225521)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants