Avoid multiple read requests on small parquet files

[raunaqmorarka]

Description

Read files below parquet.small-file-threshold size in a single file system request to avoid multiple small read requests

Additional context and related issues

This is similar to hive.orc.tiny-stripe-threshold in ORC reader

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive, Hudi, Delta, Iceberg
* Reduce read requests for scanning small parquet files. 
  The catalog configuration property `parquet.small-file-threshold` or the corresponding
  catalog session property can be used to change the default size of `3MB` below which files 
  will be read entirely. Setting this configuration to `0B` disables the feature. ({issue}`19127`)

[wendigo]

Do you have any benchmarks @raunaqmorarka ?

[raunaqmorarka]

Iceberg partitioned parquet small read threshold.pdf

[findinpath]

As of now, within the tests of the object storage connectors, we use almost completely for reading Parquet content the memory implementation.
This doesn't reflect however the reality in production.
I'd recommend using the session without small file threshold by default (on the query runner setup) and only when explicitly needed make use of this feature.

[raunaqmorarka]

As of now, within the tests of the object storage connectors, we use almost completely for reading Parquet content the memory implementation.
This doesn't reflect however the reality in production.
I'd recommend using the session without small file threshold by default (on the query runner setup) and only when explicitly needed make use of this feature.

The corresponding orc feature hive.orc.tiny-stripe-threshold has a higher threshold of 8MB and we still don't disable it for tests. The same buffering of small file can be found in LinePageSourceFactory as well and that's not even configurable.
We have unit test coverage for TrinoParquetDataSource in TestParquetDataSource and in other places, so it doesn't become unused in tests due to the change here.
I've also added parquet.small-file-threshold=100kB to the default hive product test config to improve coverage of typical case in TestParquet.