Respect page index policy option for ParquetObjectReader when it's not skip

[zhuqi-lucas]

Which issue does this PR close?

When i try to wrapper ParquetObjectReader and implement our internal metadata cache, and i will pass the option to the inner ParquetObjectReader, but it does not respect the index policy option even it's not skip, and it always be false and will not load page index which i want to prefetch and cache.

• Closes #8856

cc @alamb

Rationale for this change

Support option with page index if it's not skip.

Are these changes tested?

Yes

Are there any user-facing changes?

No

[alamb]

Thank you @zhuqi-lucas -- this makes sense to me

[zhuqi-lucas]

Thank you @zhuqi-lucas -- this makes sense to me

Thank you @alamb for quick review!

[etseidl]

This looks ok to me. I'm not up on all uses of this API so I wonder if down the road we can just get rid of the preloading options and require use of ArrowReaderOptions if one wants control over page index loading.

I think for now we should add a note to with_preload_column_index and with_preload_offset_index that settings in ArrowReaderOptions might override, as this is non-obvious.

[zhuqi-lucas]

This looks ok to me. I'm not up on all uses of this API so I wonder if down the road we can just get rid of the preloading options and require use of ArrowReaderOptions if one wants control over page index loading.

I think for now we should add a note to with_preload_column_index and with_preload_offset_index that settings in ArrowReaderOptions might override, as this is non-obvious.

Thank you @etseidl for review and good suggestion, i added comments in latest commit. And i also created a follow-up ticket, we may can get rid of the reloading options and require use of ArrowReaderOptions.
#8862

[alamb]

This looks ok to me. I'm not up on all uses of this API so I wonder if down the road we can just get rid of the preloading options and require use of ArrowReaderOptions if one wants control over page index loading.

I think for now we should add a note to with_preload_column_index and with_preload_offset_index that settings in ArrowReaderOptions might override, as this is non-obvious.

I agree the current state of the structures is confusing (nothing that you did @zhuqi-lucas).

In my opinion part of the confusion stems from having ParuetObjectReader responsible both for the IO with object_store AND for reading/caching the ParquetMetadata

I think those are separate concerns and in an ideal API they wouldn't be in the same object

[alamb]

Thank you @zhuqi-lucas and @etseidl