Skip to content

feat: add native Parquet datetime rebasing - #5047

Draft
peterxcli wants to merge 2 commits into
apache:mainfrom
peterxcli:feat/native-datetime-rebase
Draft

feat: add native Parquet datetime rebasing#5047
peterxcli wants to merge 2 commits into
apache:mainfrom
peterxcli:feat/native-datetime-rebase

Conversation

@peterxcli

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #5010.

Rationale for this change

Native Parquet scans did not apply Spark's legacy Julian/Gregorian datetime rebasing. Files written by Spark 2.x or with a LEGACY write mode could therefore return shifted ancient DATE and TIMESTAMP values. Rebasing only final output is insufficient because Parquet predicates and native filters or aggregates can consume the encoded values first.

Spark resolves rebasing separately for every file, with footer metadata taking precedence over reader configuration. Comet needs the same behavior to keep native scans correct for legacy, corrected, and mixed-file inputs.

What changes are included in this PR?

  • Propagate the Parquet datetime and INT96 read modes plus the executor JVM timezone to native scans.
  • Resolve the effective rebase specification from each file's Spark version, legacy markers, and timezone metadata.
  • Rebase DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS, and INT96 values before native filtering and aggregation.
  • Use Spark's Julian-to-Gregorian lookup table, with Spark JNI fallback for table misses and BCE timestamps.
  • Raise Spark's normal rebase exception in EXCEPTION mode.
  • Remove the dead datetime-rebase configuration and obsolete native scan limitation.
  • Add focused Rust tests and Spark-derived Parquet tests for mixed per-file modes, configuration precedence, and error propagation.

How are these changes tested?

  • make core
  • cargo fmt --all -- --check
  • cargo clippy --all-targets --workspace -- -D warnings
  • cargo test -p datafusion-comet datetime_rebase --no-default-features
  • ./mvnw test -Dtest=none '-Dsuites=org.apache.comet.parquet.ParquetReadV1Suite native scan rebases legacy' -Dscalastyle.skip=true
  • ./mvnw test -Dtest=none '-Dsuites=org.apache.comet.parquet.ParquetReadV1Suite native scan honors per-file datetime rebase modes' -Dscalastyle.skip=true
  • ./mvnw test -Dtest=none '-Dsuites=org.apache.spark.sql.comet.PlanDataInjectorSuite native scan injector records the executor JVM timezone' -Dscalastyle.skip=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Datetime rebase: track the documented scan limitation, and spark.comet.exceptionOnDatetimeRebase is dead code

1 participant