[datafusion] Honor Paimon read.batch-size

[JingsongLi]

Summary

Honor Java Paimon standard read.batch-size table option in file decoding instead of coupling Paimon readers to DataFusion TaskContext batch size. DataFusion can override the persisted table option per session through its existing Paimon dynamic-option syntax.

Changes

• add a typed CoreOptions::read_batch_size accessor with the Java-compatible default of 1024 and validation for invalid or non-positive values
• expose TableSchema::core_options() as a zero-copy typed view while keeping the serialized options map as the single source of truth
• validate persisted read.batch-size values during table creation and ALTER option application; validate dynamic overrides when the file reader consumes them
• pass the resolved table option into normal, raw incremental, audit, format-table, PK sort-merge input, and data-evolution input file readers
• keep PK sort-merge and data-evolution column-merge output batching independent from the input decode size
• keep TableRead free of a new public batch-size API and keep DataFusion scan execution independent of SessionConfig::batch_size
• verify precedence as session dynamic option > persisted table option > default, using SET paimon.read.batch-size = N and RESET

Testing

• cargo test -p paimon --lib — 1669 passed, 1 ignored
• cargo test -p paimon input_decode_honors_read_batch_size -- --nocapture — 2 passed
• cargo test -p paimon-datafusion batch_size --lib — 2 passed
• cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings
• cargo fmt --all -- --check
• git diff --check

Notes

This follows #543. query-auth.enabled remains a runtime read-boundary guard: an auth-enabled schema is valid, but local readers must fail closed instead of bypassing server-side authorization.

[leaves12138]

There is still one semantics gap in the current implementation: a non-default read.batch-size is not propagated into the composite primary-key and data-evolution readers.

• KeyValueFileReader constructs DataFileReader in crates/paimon/src/table/kv_file_reader.rs without calling with_batch_size.
• DataEvolutionReader does the same for its direct and column-source reads in crates/paimon/src/table/data_evolution_reader.rs.

Those paths therefore pass None to the format reader, so Parquet continues to decode with its default batch size of 1024. The PR note says that PK merge and data-evolution output batching keeps its existing boundaries, but currently the input file-decoder batch size also ignores the table option. Java Paimon injects READ_BATCH_SIZE when constructing the FileFormat, so the underlying format readers receive it regardless of the higher-level merge reader.

Please thread the resolved batch size through KeyValueReadConfig / KeyValueFileReader and the DataEvolutionReader file sources. The final merge output can still keep SortMergeReaderBuilder's 1024 boundary and MERGE_BATCH_SIZE = 1024 if that is intentional. A regression test using a non-default value on a merge-required PK split and a data-evolution read would cover both paths.

[leaves12138]

Re-reviewed the latest revision. The configured read batch size is now propagated into PK sort-merge and data-evolution input file readers while keeping merge output batching independent. The new regression tests cover both composite paths, and the focused tests pass locally. LGTM.