Skip to content

[improvement](parquet) Decode fragmented dictionary indices in one pass - #66504

Open
Gabriel39 wants to merge 4 commits into
apache:branch-4.1from
Gabriel39:dev/parquet-v2-fused-dictionary-selection-4.1
Open

[improvement](parquet) Decode fragmented dictionary indices in one pass#66504
Gabriel39 wants to merge 4 commits into
apache:branch-4.1from
Gabriel39:dev/parquet-v2-fused-dictionary-selection-4.1

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

File Scanner V2 can produce hundreds of one-row selection ranges after compound raw filtering. Dictionary decoding then repeatedly enters skip, decode, and validation for every range. The roughly 5% fragmented Decimal predicate shape in TPC-DS Q28 misses the existing dense-selection threshold and spends significant CPU on dictionary-index transitions.

What is changed?

  • Preserve the existing dense-selection policy.
  • Decode dictionary IDs sequentially once when the estimated range-transition cost exceeds one full pass and the index scratch buffer fits in L2 cache.
  • Keep clustered and very sparse selections on range decode.
  • Add a 5% decoder benchmark axis and a regression test for the Q28-shaped selection.

Performance

Release build, same-host CPU-pinned A-B-B-A comparison, 9 repetitions per run. Values are average paired median CPU time; lower is better.

Scenario Before (ns) After (ns) Change
Dictionary INT32, 5% alternating 191,996 25,664 -86.63%
Dictionary INT32, 10% alternating 375,928 34,112 -90.93%
Dictionary INT64, 5% alternating 234,943 31,844 -86.45%
Dictionary BYTE_ARRAY, 5% alternating 230,975 30,649 -86.73%
Dictionary INT32, 1% alternating (control) 68,290 44,670 -34.59%
Dictionary INT32, 5% clustered (control) 26,223 18,698 -28.70%
Plain INT32, 5% alternating (control) 11,344 10,781 -4.96%

No regressions were observed in the control matrix. Existing paths remain unchanged unless fragmented-selection transition cost crosses the new cache-aware threshold.

Validation

  • 132 targeted Parquet native-decoder and benchmark-scenario tests passed.
  • Release microbenchmark completed with the matrix above.
  • Release BE build completed successfully.

This PR is stacked on #66485; the final commit contains this change.

@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner August 5, 2026 12:34
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31006232914

Please trigger /review again after that time.

@Gabriel39
Gabriel39 force-pushed the dev/parquet-v2-fused-dictionary-selection-4.1 branch from 79d397c to 0eb752d Compare August 5, 2026 15:23
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31020182365

Please trigger /review again after that time.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.27% (24674/42347)
Line Coverage 42.51% (248528/584601)
Region Coverage 38.55% (196233/508970)
Branch Coverage 39.95% (89832/224852)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.75% (30441/41275)
Line Coverage 57.68% (334695/580226)
Region Coverage 54.60% (278650/510336)
Branch Coverage 55.50% (124656/224610)

@Gabriel39
Gabriel39 force-pushed the dev/parquet-v2-fused-dictionary-selection-4.1 branch from 79d397c to 0eb752d Compare August 6, 2026 01:54
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31064118892

Please trigger /review again after that time.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 66.13% (248/375) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.35% (24745/42410)
Line Coverage 42.60% (249540/585763)
Region Coverage 38.63% (197036/510019)
Branch Coverage 40.03% (90258/225468)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants