[Parquet] How to decode part columns of ColumnIndex

[mapleFU]

Which part is this question about

New parquet metadata decoder and page index. I know we have #8643 , and we can avoid Statistics via https://github.com/apache/arrow-rs/blob/main/parquet/src/file/metadata/options.rs#L91
but column index seems follow a different rule: https://github.com/apache/arrow-rs/blob/main/parquet/src/file/metadata/parser.rs#L244

I may miss the reason here.

Describe your question

How can we avoid decode some columns in column index

Additional context

No

[mapleFU]

cc @alamb @etseidl did I miss some thing? I was busy in the passing few months so may missing some changes here

[etseidl]

Hi @mapleFU 👋. You haven't missed anything. I've not explored this feature much given #7582, but perhaps it's time to kick this again. I definitely see the need for only parsing the column indexes you'll actually use.

One issue I can see is figuring out the I/O bounds. Right now we read the entire page index blob into memory, and then parse bits as needed. I could see keeping that and only parsing the requested column indexes to keep I/O simple. Otherwise we could be doing a lot of small fetches.

[mapleFU]

Avoiding io is possible, but in intuition, perhaps like statistics, it's unlikely to avoid i/o for column indexes. Just avoid decode would help a lot 🤔

Considering Statistics can avoid decode some columns. Page Index would have larger statistics. So I think it's okay to avoid it. And for I/O, in intuition, column index is much smaller than data, and may not benifit from i/o elimination, especially when we're on HDD or cloud storage (maybe optimize little on local NVMe SSD?)