[SPARK-57825][SQL] Add +/- ANSI year-month interval support for TIMESTAMP_NTZ/LTZ(p) - #57636
[SPARK-57825][SQL] Add +/- ANSI year-month interval support for TIMESTAMP_NTZ/LTZ(p)#57636stevomitric wants to merge 2 commits into
Conversation
…TAMP_NTZ/LTZ(p) ### What changes were proposed in this pull request? This PR adds support for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) in `+/- ANSI year-month interval` arithmetic. Concretely: - Extends `TimestampAddYMInterval` input typing to accept nanos timestamp types alongside the existing microsecond timestamp types. - Adds nanos-aware execution/codegen paths that apply the month shift on epoch micros while carrying the `nanosWithinMicro` remainder through unchanged. - Routes nanos timestamps into `TimestampAddYMInterval` from `BinaryArithmeticWithDatetimeResolver` for both `Add` and `Subtract` (the year-month branches previously matched only `TimestampType | TimestampNTZType`, so nanos timestamps fell through to an unresolved `Add` / `Subtract` and failed analysis). - Adds a `DateTimeUtils.timestampNanosAddMonths` helper. - Adds catalyst and SQL test coverage for NTZ/LTZ nanos year-month interval arithmetic parity, and regenerates the impacted nanos SQL golden files. ### Why are the changes needed? Spark already supports timestamp +/- ANSI year-month interval for the microsecond timestamp families, and SPARK-57501 added the day-time-interval parity for the nanos families, but year-month intervals were still unsupported for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`). This left valid datetime arithmetic unsupported for nanos types. These changes close that parity gap while preserving nanos precision semantics and existing LTZ/NTZ timezone behavior. ### Does this PR introduce _any_ user-facing change? Yes. `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) now support `+/- ANSI year-month interval` arithmetic. Examples: - `TIMESTAMP_NTZ '2020-01-02 03:04:05.123456789' + INTERVAL '1' YEAR` - `TIMESTAMP_LTZ '2020-01-02 03:04:05.123456789 UTC' - INTERVAL '1-2' YEAR TO MONTH` ### How was this patch tested? - `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.DateExpressionsSuite org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite'` - `SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos" -z "interval"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos"'` - `build/sbt catalyst/scalastyle catalyst/Test/scalastyle` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Co-authored-by: Isaac
uros-b
left a comment
There was a problem hiding this comment.
Is this ready for review @stevomitric?
|
Yes, PTAL when you get a chance. cc @uros-b cc @cloud-fan also PTAL if you get a chance. |
cloud-fan
left a comment
There was a problem hiding this comment.
0 blocking, 0 non-blocking, 0 nits.
The resolver, expression, and datetime-helper changes are consistent with the existing microsecond path, and the focused coverage exercises the newly enabled end-to-end behavior.
Verification
I traced Add and Subtract from BinaryArithmeticWithDatetimeResolver through TimestampAddYMInterval into timestampNanosAddMonths. Both interpreted and generated paths select the same nanos helper, which delegates the calendar adjustment to timestampAddMonths and preserves nanosWithinMicro when rebuilding the result.
uros-b
left a comment
There was a problem hiding this comment.
Thank you @stevomitric and @cloud-fan!
|
Merge Summary:
Posted by |
|
Merge Summary:
Posted by |
…TAMP_NTZ/LTZ(p) ### What changes were proposed in this pull request? This PR adds support for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) in `+/- ANSI year-month interval` arithmetic. Concretely: - Extends `TimestampAddYMInterval` input typing to accept nanos timestamp types alongside the existing microsecond timestamp types. - Adds nanos-aware execution/codegen paths that apply the month shift on epoch micros while carrying the `nanosWithinMicro` remainder through unchanged. - Routes nanos timestamps into `TimestampAddYMInterval` from `BinaryArithmeticWithDatetimeResolver` for both `Add` and `Subtract` (the year-month branches previously matched only `TimestampType | TimestampNTZType`, so nanos timestamps fell through to an unresolved `Add` / `Subtract` and failed analysis). - Adds a `DateTimeUtils.timestampNanosAddMonths` helper. - Adds catalyst and SQL test coverage for NTZ/LTZ nanos year-month interval arithmetic parity, and regenerates the impacted nanos SQL golden files. ### Why are the changes needed? Spark already supports timestamp +/- ANSI year-month interval for the microsecond timestamp families, and SPARK-57501 added the day-time-interval parity for the nanos families, but year-month intervals were still unsupported for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`). This left valid datetime arithmetic unsupported for nanos types. These changes close that parity gap while preserving nanos precision semantics and existing LTZ/NTZ timezone behavior. ### Does this PR introduce _any_ user-facing change? Yes. `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) now support `+/- ANSI year-month interval` arithmetic. Examples: - `TIMESTAMP_NTZ '2020-01-02 03:04:05.123456789' + INTERVAL '1' YEAR` - `TIMESTAMP_LTZ '2020-01-02 03:04:05.123456789 UTC' - INTERVAL '1-2' YEAR TO MONTH` ### How was this patch tested? - `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.DateExpressionsSuite org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite'` - `SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos" -z "interval"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos"'` - `build/sbt catalyst/scalastyle catalyst/Test/scalastyle` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes #57636 from stevomitric/stevomitric/spark-57825-nanos. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit 88697a3) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
What changes were proposed in this pull request?
This PR adds support for
TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p)(p in [7, 9]) in+/- ANSI year-month intervalarithmetic.Concretely:
TimestampAddYMIntervalinput typing to accept nanos timestamp types alongside the existing microsecond timestamp types.nanosWithinMicroremainder through unchanged.TimestampAddYMIntervalfromBinaryArithmeticWithDatetimeResolverfor bothAddandSubtract(the year-month branches previously matched onlyTimestampType | TimestampNTZType, so nanos timestamps fell through to an unresolvedAdd/Subtractand failed analysis).DateTimeUtils.timestampNanosAddMonthshelper.Why are the changes needed?
Spark already supports timestamp +/- ANSI year-month interval for the microsecond timestamp families, and SPARK-57501 added the day-time-interval parity for the nanos families, but year-month intervals were still unsupported for
TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p)(p in [7, 9]). This left valid datetime arithmetic unsupported for nanos types.These changes close that parity gap while preserving nanos precision semantics and existing LTZ/NTZ timezone behavior.
Does this PR introduce any user-facing change?
Yes.
TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p)(p in [7, 9]) now support+/- ANSI year-month intervalarithmetic.Examples:
TIMESTAMP_NTZ '2020-01-02 03:04:05.123456789' + INTERVAL '1' YEARTIMESTAMP_LTZ '2020-01-02 03:04:05.123456789 UTC' - INTERVAL '1-2' YEAR TO MONTHHow was this patch tested?
build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.DateExpressionsSuite org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite'SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos" -z "interval"'build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos"'build/sbt catalyst/scalastyle catalyst/Test/scalastyleWas this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)