feat: write value stats for append data files#477
Conversation
Append data files previously wrote empty value_stats, so manifest-level stats pruning could not skip files produced by Rust append writes. Accumulate min/max/null counts for supported fixed-width value columns across batches, persist dense value_stats_cols, and handle nanosecond timestamp stats. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
In Java Side, the performance of the accumulator is poor, and the only truly useful link is the one that extracts stats from Parquet. Perhaps we should optimize its implementation instead of the accumulator.
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
@JingsongLi Thanks, addressed. I changed the implementation to avoid per-row stats accumulation. Append value stats are now extracted from Parquet writer metadata on close, aligned with the Java ParquetSimpleStatsExtractor path. Added/updated tests for append value stats, stats pruning, and timestamp stats. |
| parquet_metadata: Option<&parquet::file::metadata::ParquetMetaData>, | ||
| ) -> DataFileMeta { | ||
| let (value_stats, value_stats_cols) = | ||
| crate::arrow::format::parquet::extract_value_stats(parquet_metadata, write_fields); |
There was a problem hiding this comment.
Just return stats from parquet? It is not good to expose ParquetMetaData in core.
| .collect() | ||
| } | ||
|
|
||
| pub(crate) fn extract_value_stats( |
There was a problem hiding this comment.
We need to support default stats mode: truncate(16), otherwise the stats are too big.
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2 participants
Purpose
Rust append writes currently emit empty
_VALUE_STATS, so manifest-level stats pruning cannot skip data files written by Rust even when min/max/null-count information is available.Brief change log
DataFileWriter.DataFileMeta.value_statsand densevalue_stats_cols.Tests
cargo fmt --all -- --checkcargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warningscargo test -p paimon --all-targets --features fulltext,vortex,mosaicAPI and Format
Documentation