Context
#5177 makes the native Parquet reader's TIMESTAMP_MILLIS -> microseconds conversion checked for top-level columns, matching Spark's millisToMicros (Math.multiplyExact). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:
- Nested fields (struct children, list/map elements): the conversion inside
parquet_convert_struct_to_struct / list / map recursion is unchecked.
- Scans whose data filters reference nested fields (
GetStructField): the whole scan falls back to the safe cast (SparkParquetOptions.checked_timestamp_overflow = false, set in init_datasource_exec).
Why
Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (ParquetFilters supports nested column predicates). DataFusion currently can neither:
- prune nested-field predicates —
datafusion-pruning says "PruningPredicate does not support pruning on nested fields yet" (pruning_predicate.rs), nor
- evaluate them as Parquet row filters —
can_expr_be_pushed_down_with_schemas in datafusion-datasource-parquet/src/row_filter.rs classifies struct columns as non-pushable.
So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: ts IN (...), ts <=> ..., and s.ts < ... reproducers).
Remaining divergence vs Spark
- A nested-predicate scan that Spark fails to prune (mixed row-group statistics) throws
ArithmeticException in Spark but returns NULL for overflowing top-level values in Comet.
- A direct read of an overflowing nested
TIMESTAMP_MILLIS field throws in Spark but returns NULL in Comet (pre-existing behavior on main).
Proposed work
Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the checked_timestamp_overflow fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in SparkPhysicalExprAdapter to GetStructField-wrapped conversions.
Regression coverage lives in ParquetReadSuite:
- "TIMESTAMP_MILLIS overflow fails in native scan"
- "TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail"
Related: #5517 (error fidelity for the overflow exception).
Context
#5177 makes the native Parquet reader's
TIMESTAMP_MILLIS -> microsecondsconversion checked for top-level columns, matching Spark'smillisToMicros(Math.multiplyExact). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:parquet_convert_struct_to_struct/ list / map recursion is unchecked.GetStructField): the whole scan falls back to the safe cast (SparkParquetOptions.checked_timestamp_overflow = false, set ininit_datasource_exec).Why
Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (
ParquetFilterssupports nested column predicates). DataFusion currently can neither:datafusion-pruningsays "PruningPredicate does not support pruning on nested fields yet" (pruning_predicate.rs), norcan_expr_be_pushed_down_with_schemasindatafusion-datasource-parquet/src/row_filter.rsclassifies struct columns as non-pushable.So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177:
ts IN (...),ts <=> ..., ands.ts < ...reproducers).Remaining divergence vs Spark
ArithmeticExceptionin Spark but returns NULL for overflowing top-level values in Comet.TIMESTAMP_MILLISfield throws in Spark but returns NULL in Comet (pre-existing behavior on main).Proposed work
Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the
checked_timestamp_overflowfallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite inSparkPhysicalExprAdaptertoGetStructField-wrapped conversions.Regression coverage lives in
ParquetReadSuite:Related: #5517 (error fidelity for the overflow exception).