Skip to content

[AURON #2155] Date-part extraction functions missing timezone handling for Timestamp inputs #2156

Open
ShreyeshArangath wants to merge 4 commits intoapache:masterfrom
ShreyeshArangath:bug/ts-aware
Open

[AURON #2155] Date-part extraction functions missing timezone handling for Timestamp inputs #2156
ShreyeshArangath wants to merge 4 commits intoapache:masterfrom
ShreyeshArangath:bug/ts-aware

Conversation

@ShreyeshArangath
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2155

Rationale for this change

Five date-part extraction functions in NativeConverters.scala use buildExtScalarFunction, which does not pass the session timezone to the native Rust implementation:

By contrast, Hour, Minute, Second, and WeekOfYear correctly use buildTimePartExt, which passes sessionLocalTimeZone for TimestampType inputs.

This inconsistency can cause incorrect results for timestamp inputs near date boundaries in non-UTC timezones.

Affected functions:

  • Year (Spark_Year) — not timezone-aware
  • Month (Spark_Month) — not timezone-aware
  • DayOfMonth (Spark_Day) — not timezone-aware
  • DayOfWeek (Spark_DayOfWeek) — not timezone-aware
  • Quarter (Spark_Quarter) — not timezone-aware

What changes are included in this PR?

This PR fixes the bug described above

Are there any user-facing changes?

Correctness issues fixed

How was this patch tested?

Unit tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Date-part extraction functions missing timezone handling for Timestamp inputs

1 participant