Search before asking
Description
The ORC and Parquet readers evaluate every filter conjunct over all rows of a block and AND the results together, regardless of how cheap/selective each predicate is or the order the planner emitted them. An expensive predicate (e.g. a string function like split_by_string(col)[n]='x') is therefore run over the full block even when a cheap, highly-selective predicate on the same block would have filtered almost everything first.
Proposal
Add three opt-in (default off) session variables that only change the order/method of filter execution, never which rows pass:
enable_scan_conjunct_reorder — static cost-based conjunct reorder (cheap predicates first).enable_scan_selective_filter — selection-vector execution: a later predicate is evaluated only on the rows earlier ones let through, producing the identical full-width result_filter/can_filter_all contract.enable_scan_adaptive_reorder — runtime reorder by measured cost/selectivity, correcting the static estimate for skewed data or unknown UDF cost.
All default off, so with the flags unset the reader path is byte-identical to today. When enabled, they change execution order/method only — results are identical.
Use case
Faster ORC/Parquet scans on queries mixing cheap selective predicates with expensive ones (e.g. string/UDF predicates).
Related issues
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
The ORC and Parquet readers evaluate every filter conjunct over all rows of a block and AND the results together, regardless of how cheap/selective each predicate is or the order the planner emitted them. An expensive predicate (e.g. a string function like
split_by_string(col)[n]='x') is therefore run over the full block even when a cheap, highly-selective predicate on the same block would have filtered almost everything first.Proposal
Add three opt-in (default off) session variables that only change the order/method of filter execution, never which rows pass:
enable_scan_conjunct_reorder— static cost-based conjunct reorder (cheap predicates first).enable_scan_selective_filter— selection-vector execution: a later predicate is evaluated only on the rows earlier ones let through, producing the identical full-widthresult_filter/can_filter_allcontract.enable_scan_adaptive_reorder— runtime reorder by measured cost/selectivity, correcting the static estimate for skewed data or unknown UDF cost.All default off, so with the flags unset the reader path is byte-identical to today. When enabled, they change execution order/method only — results are identical.
Use case
Faster ORC/Parquet scans on queries mixing cheap selective predicates with expensive ones (e.g. string/UDF predicates).
Related issues
No response
Are you willing to submit PR?
Code of Conduct