[fix](parquet) Isolate Variant planning from ordinary scans - #66441
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Automated review: changes requested. One P2 performance regression remains: a partial complex projection that excludes all Variant descendants is still treated as Variant-bearing because the new gates test plan-pointer presence rather than the projection-sensitive plan content. Schema propagation, logical/physical type ownership, file-context/footer-cache lifecycle, row-group and shredded-statistics conservatism, row shape/cursors, and external compatibility otherwise checked out. The added tests validate the cached booleans but do not exercise the Variant-excluding projection. Review was static as required; repository builds/tests were not run. No additional user focus was provided, so the full PR received the normal review. Review-cycle status: all full-coverage and risk-focused agents converged on this frozen candidate set; landed-state verification follows submission.
| _native_reader->reset_filter_map_index(); | ||
| const bool materialize_variant = | ||
| !dictionary_ids && _variant_plan->contains_variant && output_type->equals(*_type); | ||
| !dictionary_ids && _variant_plan != nullptr && output_type->equals(*_type); |
There was a problem hiding this comment.
[P2] Keep the Variant gate projection-sensitive
contains_variant describes the complete schema subtree, not the selected projection. For a reachable STRUCT<label, info VARIANT> scan that projects only label, build_variant_plan() follows the projection and returns contains_variant == false, but the plan pointer is still non-null. This check therefore sends every read/select and accumulated lazy-skip batch through _variant_physical_column, recursive wrapper transformation/shape validation, and replacement scratch allocation even though no Variant node was requested; the base path decoded directly because it checked the plan's content. Please discard a plan whose contains_variant is false (or preserve the content-sensitive gates), and cover a Variant-excluding partial projection.
|
PR approved by at least one committer and no changes requested. |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
What
Why
Variant planning was performed for every projected column and row group, even when the schema contained no Variant data. This added repeated schema traversal and allocation overhead to ordinary scans.
Validation
branch-4.1.