perf(parquet): scope exact metadata reads by storage capability

[JunRuiLee]

Purpose

Linked issue: close #687

Use an explicit range-read capability to choose the Parquet metadata loading strategy:

• Built-in HDFS, local filesystem, and memory readers use the seek-based path: read the 8-byte footer first, then fetch the exact footer metadata and requested page-index ranges.
• Object-store readers and caller-provided filesystem operators retain the 512 KiB suffix prefetch by default, avoiding additional serialized network round trips when their range-read cost is unknown.

OpenDAL's HDFS native service keeps a file-scoped positioned-read handle, so exact range reads reuse the same underlying HDFS file reader. The capability is determined in the IO/storage layer and propagated through InputFileReader; the Parquet layer does not infer the backend from URI strings. Custom FileRead implementations may explicitly opt in, while the default remains conservative.

Performance validation

A local HDFS comparison showed better read performance with exact range reads than with the fixed 512 KiB prefetch. Object stores and caller-provided filesystem operators have not been benchmarked, so this change deliberately preserves their existing prefetch behavior rather than assuming the HDFS result transfers to them.

This PR remains Draft until the reproducible HDFS measurements are added. The benchmark record should include the same dataset/query for both variants, file count, representative footer-metadata sizes, repeated wall-clock results, and bytes-read or request counts when available; #687 tracks that evidence.

Brief change log

• Add a defaulted FileRead::supports_cheap_range_reads capability with a conservative false default.
• Enable exact metadata/page-index reads for built-in HDFS, local filesystem, and memory backends.
• Keep the fixed 512 KiB metadata prefetch for OSS, S3, COS, AzDLS, OBS, GCS, caller-provided filesystem operators, and other readers that do not explicitly opt in.
• Avoid requesting OffsetIndex solely because an external row selection is present but empty.
• Add range-tracking tests for both exact-read and prefetch paths.
• Add storage classification and FileRead capability-propagation tests.

Tests

• cargo fmt --all -- --check
• cargo test --locked -p paimon --lib arrow::format::parquet::tests (49 passed)
• cargo test --locked -p paimon --lib scheme_tests --features storage-all (10 passed)
• cargo test --locked -p paimon --lib custom_fs_operator --features storage-all (3 passed)
• Targeted memory and caller-provided filesystem capability-propagation tests passed.
• cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
• git diff --check

API and Format

Adds a defaulted FileRead::supports_cheap_range_reads method. Existing implementations do not need to implement the new method; they receive the conservative false behavior. No storage-format changes.

Documentation

No user-facing documentation changes required. Benchmark evidence is tracked in #687 before the PR is marked ready.

[JingsongLi]

How about object store?

[JunRuiLee]

How about object store?

Good point. I revised the implementation so exact metadata reads are opt-in through an explicit FileRead capability, rather than inferred from URI or path semantics.

• Built-in HDFS, local filesystem, and memory readers advertise cheap range reads and use exact footer, metadata, and page-index reads.
• OSS, S3, COS, AzDLS, OBS, and GCS keep the existing 512 KiB metadata prefetch.
• Caller-provided filesystem operators (CustomFs) also keep prefetch by default, because filesystem semantics alone do not guarantee cheap range reads.
• Other custom FileRead implementations default to the conservative prefetch behavior, but can explicitly opt in.

The capability is decided in the storage/IO layer and propagated to the Parquet reader. Tests cover backend classification, capability propagation, exact reads, and retained prefetch.

We have only benchmarked HDFS so far, so preserving the existing behavior for object stores and unknown custom backends is intentional.