[Python] Make CacheOptions configurable from Python

[Tom-Newton]

Describe the enhancement requested

When doing parquet reads from high latency storage e.g. S3 or Azure blob storage its beneficial to use fragment_scan_options=ParquetFragmentScanOptions(pre_buffer=True).

As far as I can tell pre_buffer=True does 2 important things:

1. It reads all the parquet metadata at the beginning then makes all the data block reads.
2. It coalesces adjacent data blocks so that it can read the data in a smaller number of larger reads. Coalescing is done based on CacheOptions, in particular hole_size_limit and range_size_limit.

In my use-case I have found that while point 1 is very beneficial for performance point 2 is actually detrimental. Point 1 though is still enough to make pre_buffer=True a net advantage. I found that setting range_size_limit to a much smaller value (in combination with pyarrow.set_io_thread_count(<large number>)) can give me ~6X performance improvement on my usecase because it increases the parallelism. I would therefore like to make these parameters configurable from python.

If you are interested in my use-case I'm loading a single parquet file from US East Azure blob storage to a workstation in the UK. Probably the current defaults would be sensible if it wasn't for the trans-Atlantic latency. (Pyarrow does not natively support Azure yet but I'm making use of #12914 even though its not in an official release yet)

I can make a PR that implements this if others agree that its a sensible idea.

Component(s)

Python

[pitrou]

cc @westonpace @lidavidm @AlenkaF @jorisvandenbossche

[lidavidm]

It seems reasonable.

[jorisvandenbossche]

The CacheOptions is also already explicitly exposed in the IpcFragmentScanOptions (on the C++ side, this class it not yet exposed in Python).

All the other options of ArrowReaderProperties are exposed in the pyarrow equivalent (in ParquetReader in _parquet.pyx), so indeed seems to make sense to expose the cache options as well.

In the ParquetFragmentScanOptions, currently only pre_buffer is exposed.

[Tom-Newton]

take

[Tom-Newton]

I've opened a PR for review to address this #36627.

I don't have much experience with C++ or Cython and this is my first contribution to arrow so hopefully what I've done makes sense.