Skip to content

[fix](parquet) Isolate Variant planning from ordinary scans - #66441

Merged
Gabriel39 merged 1 commit into
apache:branch-4.1from
Gabriel39:fix/isolate-variant-planning-4.1
Aug 4, 2026
Merged

[fix](parquet) Isolate Variant planning from ordinary scans#66441
Gabriel39 merged 1 commit into
apache:branch-4.1from
Gabriel39:fix/isolate-variant-planning-4.1

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

What

  • Cache whether a Parquet schema subtree contains Variant data.
  • Build Variant physical types and materialization plans only for Variant-containing columns.
  • Keep ordinary Parquet scans on their existing logical-type path.

Why

Variant planning was performed for every projected column and row group, even when the schema contained no Variant data. This added repeated schema traversal and allocation overhead to ordinary scans.

Validation

  • Cherry-picked cleanly onto the latest branch-4.1.
  • The resulting patch is equivalent to the original isolated fix.
  • Local tests were not run for this cherry-pick-only PR.

@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner August 4, 2026 11:14
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review: changes requested. One P2 performance regression remains: a partial complex projection that excludes all Variant descendants is still treated as Variant-bearing because the new gates test plan-pointer presence rather than the projection-sensitive plan content. Schema propagation, logical/physical type ownership, file-context/footer-cache lifecycle, row-group and shredded-statistics conservatism, row shape/cursors, and external compatibility otherwise checked out. The added tests validate the cached booleans but do not exercise the Variant-excluding projection. Review was static as required; repository builds/tests were not run. No additional user focus was provided, so the full PR received the normal review. Review-cycle status: all full-coverage and risk-focused agents converged on this frozen candidate set; landed-state verification follows submission.

_native_reader->reset_filter_map_index();
const bool materialize_variant =
!dictionary_ids && _variant_plan->contains_variant && output_type->equals(*_type);
!dictionary_ids && _variant_plan != nullptr && output_type->equals(*_type);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the Variant gate projection-sensitive

contains_variant describes the complete schema subtree, not the selected projection. For a reachable STRUCT<label, info VARIANT> scan that projects only label, build_variant_plan() follows the projection and returns contains_variant == false, but the plan pointer is still non-null. This check therefore sends every read/select and accumulated lazy-skip batch through _variant_physical_column, recursive wrapper transformation/shape validation, and replacement scratch allocation even though no Variant node was requested; the base path decoded directly because it checked the plan's content. Please discard a plan whose contains_variant is false (or preserve the content-sensitive gates), and cover a Variant-excluding partial projection.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2026
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (43/43) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.29% (24579/42164)
Line Coverage 42.47% (246903/581382)
Region Coverage 38.56% (195076/505857)
Branch Coverage 39.86% (89086/223512)

@Gabriel39
Gabriel39 merged commit 7d9cf5d into apache:branch-4.1 Aug 4, 2026
29 of 33 checks passed
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (43/43) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.50% (30265/41178)
Line Coverage 57.42% (331985/578177)
Region Coverage 54.33% (276426/508758)
Branch Coverage 55.25% (123544/223615)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/5.0.x

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants