Skip to content

fix: fall back for Parquet datetime rebasing - #5048

Open
peterxcli wants to merge 6 commits into
apache:mainfrom
peterxcli:fix/fallback-for-datetime-rebase
Open

fix: fall back for Parquet datetime rebasing#5048
peterxcli wants to merge 6 commits into
apache:mainfrom
peterxcli:fix/fallback-for-datetime-rebase

Conversation

@peterxcli

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #5010.

Rationale for this change

Native Parquet scans cannot currently apply Spark's legacy Julian/Gregorian rebasing safely before Parquet pruning, filters, and aggregates consume datetime values. This can produce incorrect results for files written by Spark 2.x or with legacy datetime or INT96 rebasing.

I want to merge this conservative fallback first to restore correctness with a small, low-risk change. The native implementation in #5047 should remain a follow-up and can restore native acceleration after its broader per-file rebasing behavior is reviewed.

What changes are included in this PR?

  • Inspect Parquet footer metadata and configured read modes before converting a scan to Comet.
  • Fall back to Spark when any input file may require legacy datetime or INT96 rebasing, or when metadata inspection fails.
  • Remove the dead datetime-rebase configuration and the unused SparkParquetOptions.use_legacy_date_timestamp_or_ntz field.
  • Update scan compatibility documentation and add regression coverage for the fallback.

How are these changes tested?

  • make core
  • git diff --check
  • ./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadV1Suite fallback for Parquet datetime rebasing" -Dscalastyle.skip=true

@peterxcli
peterxcli marked this pull request as ready for review July 27, 2026 08:04
def ignoreMissingIds(conf: SQLConf): Boolean =
conf.getConfString(IGNORE_MISSING_PARQUET_FIELD_ID, "false").toBoolean

def requiresDatetimeRebase(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Datetime rebase: track the documented scan limitation, and spark.comet.exceptionOnDatetimeRebase is dead code

1 participant