Skip to content

Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning #5553

Description

@peterxcli

Context

#5177 makes the native Parquet reader's TIMESTAMP_MILLIS -> microseconds conversion checked for top-level columns, matching Spark's millisToMicros (Math.multiplyExact). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:

  1. Nested fields (struct children, list/map elements): the conversion inside parquet_convert_struct_to_struct / list / map recursion is unchecked.
  2. Scans whose data filters reference nested fields (GetStructField): the whole scan falls back to the safe cast (SparkParquetOptions.checked_timestamp_overflow = false, set in init_datasource_exec).

Why

Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (ParquetFilters supports nested column predicates). DataFusion currently can neither:

  • prune nested-field predicates — datafusion-pruning says "PruningPredicate does not support pruning on nested fields yet" (pruning_predicate.rs), nor
  • evaluate them as Parquet row filters — can_expr_be_pushed_down_with_schemas in datafusion-datasource-parquet/src/row_filter.rs classifies struct columns as non-pushable.

So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: ts IN (...), ts <=> ..., and s.ts < ... reproducers).

Remaining divergence vs Spark

  • A nested-predicate scan that Spark fails to prune (mixed row-group statistics) throws ArithmeticException in Spark but returns NULL for overflowing top-level values in Comet.
  • A direct read of an overflowing nested TIMESTAMP_MILLIS field throws in Spark but returns NULL in Comet (pre-existing behavior on main).

Proposed work

Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the checked_timestamp_overflow fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in SparkPhysicalExprAdapter to GetStructField-wrapped conversions.

Regression coverage lives in ParquetReadSuite:

  • "TIMESTAMP_MILLIS overflow fails in native scan"
  • "TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail"

Related: #5517 (error fidelity for the overflow exception).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions