Describe the enhancement requested
The Parquet C++ library already lets you control how much SizeStatistics metadata is written, via WriterProperties::Builder::set_size_statistics_level() (added in #40594). The available levels are:
None — write no size statisticsColumnChunk — write size statistics only at the column-chunk level (ColumnMetaData.size_statistics)PageAndColumnChunk — also write page-level histograms into ColumnIndex / OffsetIndex (this is the C++ default)
However, PyArrow does not currently expose this setting. There is no way from pyarrow.parquet.write_table, ParquetWriter, or the dataset writer to change the size-statistics level, so every file written by PyArrow always includes SizeStatistics at the PageAndColumnChunk level.
We have hit one concrete problem with it being unconditionally written:
Reader compatibility. Some third-party / older Parquet readers do not handle the newer SizeStatistics thrift fields gracefully and fail to read files that PyArrow produces. Being able to write with size_statistics_level="none" produces files those readers can consume, without having to downgrade any other writer settings.
The C++ capability exists and is stable; this request is purely to surface it through the Python binding layer.
Component(s)
Python, Parquet
Describe the enhancement requested
The Parquet C++ library already lets you control how much
SizeStatisticsmetadata is written, viaWriterProperties::Builder::set_size_statistics_level()(added in #40594). The available levels are:None— write no size statisticsColumnChunk— write size statistics only at the column-chunk level (ColumnMetaData.size_statistics)PageAndColumnChunk— also write page-level histograms intoColumnIndex/OffsetIndex(this is the C++ default)However, PyArrow does not currently expose this setting. There is no way from
pyarrow.parquet.write_table,ParquetWriter, or the dataset writer to change the size-statistics level, so every file written by PyArrow always includesSizeStatisticsat thePageAndColumnChunklevel.We have hit one concrete problem with it being unconditionally written:
Reader compatibility. Some third-party / older Parquet readers do not handle the newer
SizeStatisticsthrift fields gracefully and fail to read files that PyArrow produces. Being able to write withsize_statistics_level="none"produces files those readers can consume, without having to downgrade any other writer settings.The C++ capability exists and is stable; this request is purely to surface it through the Python binding layer.
Component(s)
Python, Parquet