Native Parquet writer ignores Spark's Parquet writer properties (block size, page size, dictionary, writer version, statistics)

[andygrove]

What is the problem the feature request solves?

ParquetWriterExec builds its WriterProperties with compression and nothing else (native/core/src/execution/operators/parquet_writer.rs):

let props = WriterProperties::builder()
    .set_compression(compression)
    .build();

Every other Parquet write knob Spark exposes is therefore silently ignored, and files come out with arrow-rs defaults rather than the ones the user configured:

• parquet.block.size (row group size) — tracked as an unchecked item on [EPIC] Improve Comet Native writer #2967
• parquet.page.size / parquet.page.row.count.limit
• parquet.enable.dictionary and dictionary page size
• spark.sql.parquet.writer.version (PARQUET_1_0 / PARQUET_2_0)
• statistics level, and bloom filter settings
• spark.sql.files.maxRecordsPerFile (needs file rolling, so partly separate)

Row group size is the one with real consequences: the default row group size decides read parallelism and memory for every downstream consumer, and a user who set parquet.block.size gets no error, just a differently shaped file.

This also matters beyond the V1 write path — #4658 splits Iceberg writes into writer + committer operators with the stated goal of writing data files natively next, and Iceberg data files are read by engines other than Spark.

Describe the potential solution

Thread the resolved options through the ParquetWriter protobuf message and set them on WriterProperties. Spark's ParquetOptions / ParquetWriteSupport are the reference for precedence and defaults.

Related: #2814 (compression settings), #3425 (INT96), #3427 (Spark version in footer metadata).