Is your feature request related to a problem or challenge?
arrow-rs#10141 allows RowSelection to be backed by a BooleanBuffer.
However, ParquetAccessPlan::into_overall_row_selection currently calls RowSelection::iter() and converts each selection into a Vec<RowSelector>. This materializes mask-backed selections as RLE and loses the bitmap backing before it reaches the Parquet reader.
This issue is blocked until DataFusion upgrades to an Arrow/Parquet release containing arrow-rs#10141.
Describe the solution you'd like
- Validate selection length using
row_count() + skipped_row_count() instead of iter().
- When all row-group selections are mask-backed, concatenate their
BooleanBuffers directly.
- Represent scanned row groups as all-set bitmap ranges and omit skipped row groups.
- Keep the existing selector path as the fallback for selector-backed or mixed inputs.
Describe alternatives you've considered
Converting selectors back to a bitmap in the Parquet reader does not help because the large intermediate Vec<RowSelector> has already been allocated.
Additional context
No response
Is your feature request related to a problem or challenge?
arrow-rs#10141 allows
RowSelectionto be backed by aBooleanBuffer.However,
ParquetAccessPlan::into_overall_row_selectioncurrently callsRowSelection::iter()and converts each selection into aVec<RowSelector>. This materializes mask-backed selections as RLE and loses the bitmap backing before it reaches the Parquet reader.This issue is blocked until DataFusion upgrades to an Arrow/Parquet release containing
arrow-rs#10141.Describe the solution you'd like
row_count() + skipped_row_count()instead ofiter().BooleanBuffers directly.Describe alternatives you've considered
Converting selectors back to a bitmap in the Parquet reader does not help because the large intermediate
Vec<RowSelector>has already been allocated.Additional context
No response