Skip to content

[SPARK-57825][SQL] Add +/- ANSI year-month interval support for TIMESTAMP_NTZ/LTZ(p) - #57636

Closed
stevomitric wants to merge 2 commits into
apache:masterfrom
stevomitric:stevomitric/spark-57825-nanos
Closed

[SPARK-57825][SQL] Add +/- ANSI year-month interval support for TIMESTAMP_NTZ/LTZ(p)#57636
stevomitric wants to merge 2 commits into
apache:masterfrom
stevomitric:stevomitric/spark-57825-nanos

Conversation

@stevomitric

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR adds support for TIMESTAMP_NTZ(p) / TIMESTAMP_LTZ(p) (p in [7, 9]) in +/- ANSI year-month interval arithmetic.

Concretely:

  • Extends TimestampAddYMInterval input typing to accept nanos timestamp types alongside the existing microsecond timestamp types.
  • Adds nanos-aware execution/codegen paths that apply the month shift on epoch micros while carrying the nanosWithinMicro remainder through unchanged.
  • Routes nanos timestamps into TimestampAddYMInterval from BinaryArithmeticWithDatetimeResolver for both Add and Subtract (the year-month branches previously matched only TimestampType | TimestampNTZType, so nanos timestamps fell through to an unresolved Add / Subtract and failed analysis).
  • Adds a DateTimeUtils.timestampNanosAddMonths helper.
  • Adds catalyst and SQL test coverage for NTZ/LTZ nanos year-month interval arithmetic parity, and regenerates the impacted nanos SQL golden files.

Why are the changes needed?

Spark already supports timestamp +/- ANSI year-month interval for the microsecond timestamp families, and SPARK-57501 added the day-time-interval parity for the nanos families, but year-month intervals were still unsupported for TIMESTAMP_NTZ(p) / TIMESTAMP_LTZ(p) (p in [7, 9]). This left valid datetime arithmetic unsupported for nanos types.

These changes close that parity gap while preserving nanos precision semantics and existing LTZ/NTZ timezone behavior.

Does this PR introduce any user-facing change?

Yes.

TIMESTAMP_NTZ(p) / TIMESTAMP_LTZ(p) (p in [7, 9]) now support +/- ANSI year-month interval arithmetic.

Examples:

  • TIMESTAMP_NTZ '2020-01-02 03:04:05.123456789' + INTERVAL '1' YEAR
  • TIMESTAMP_LTZ '2020-01-02 03:04:05.123456789 UTC' - INTERVAL '1-2' YEAR TO MONTH

How was this patch tested?

  • build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.DateExpressionsSuite org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite'
  • SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos" -z "interval"'
  • build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos"'
  • build/sbt catalyst/scalastyle catalyst/Test/scalastyle

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

…TAMP_NTZ/LTZ(p)

### What changes were proposed in this pull request?
This PR adds support for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) in
`+/- ANSI year-month interval` arithmetic.

Concretely:
- Extends `TimestampAddYMInterval` input typing to accept nanos timestamp types alongside
  the existing microsecond timestamp types.
- Adds nanos-aware execution/codegen paths that apply the month shift on epoch micros while
  carrying the `nanosWithinMicro` remainder through unchanged.
- Routes nanos timestamps into `TimestampAddYMInterval` from
  `BinaryArithmeticWithDatetimeResolver` for both `Add` and `Subtract` (the year-month
  branches previously matched only `TimestampType | TimestampNTZType`, so nanos timestamps
  fell through to an unresolved `Add` / `Subtract` and failed analysis).
- Adds a `DateTimeUtils.timestampNanosAddMonths` helper.
- Adds catalyst and SQL test coverage for NTZ/LTZ nanos year-month interval arithmetic
  parity, and regenerates the impacted nanos SQL golden files.

### Why are the changes needed?
Spark already supports timestamp +/- ANSI year-month interval for the microsecond timestamp
families, and SPARK-57501 added the day-time-interval parity for the nanos families, but
year-month intervals were still unsupported for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`
(`p in [7, 9]`). This left valid datetime arithmetic unsupported for nanos types.

These changes close that parity gap while preserving nanos precision semantics and existing
LTZ/NTZ timezone behavior.

### Does this PR introduce _any_ user-facing change?
Yes.

`TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) now support `+/- ANSI year-month
interval` arithmetic.

Examples:
- `TIMESTAMP_NTZ '2020-01-02 03:04:05.123456789' + INTERVAL '1' YEAR`
- `TIMESTAMP_LTZ '2020-01-02 03:04:05.123456789 UTC' - INTERVAL '1-2' YEAR TO MONTH`

### How was this patch tested?
- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.DateExpressionsSuite org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite'`
- `SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos" -z "interval"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos"'`
- `build/sbt catalyst/scalastyle catalyst/Test/scalastyle`

### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)

Co-authored-by: Isaac

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this ready for review @stevomitric?

@stevomitric stevomitric changed the title [WIP][SPARK-57825][SQL] Add +/- ANSI year-month interval support for TIMESTAMP_NTZ/LTZ(p) [SPARK-57825][SQL] Add +/- ANSI year-month interval support for TIMESTAMP_NTZ/LTZ(p) Jul 30, 2026
@stevomitric

Copy link
Copy Markdown
Contributor Author

Yes, PTAL when you get a chance. cc @uros-b

cc @cloud-fan also PTAL if you get a chance.

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 0 non-blocking, 0 nits.
The resolver, expression, and datetime-helper changes are consistent with the existing microsecond path, and the focused coverage exercises the newly enabled end-to-end behavior.

Verification

I traced Add and Subtract from BinaryArithmeticWithDatetimeResolver through TimestampAddYMInterval into timestampNanosAddMonths. Both interpreted and generated paths select the same nanos helper, which delegates the calendar adjustment to timestampAddMonths and preserves nanosWithinMicro when rebuilding the result.

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @stevomitric and @cloud-fan!

@uros-b uros-b closed this in 88697a3 Jul 30, 2026
@uros-b

uros-b commented Jul 30, 2026

Copy link
Copy Markdown
Member

Merge Summary:

Posted by merge_spark_pr.py

@uros-b

uros-b commented Jul 30, 2026

Copy link
Copy Markdown
Member

Merge Summary:

Posted by merge_spark_pr.py

uros-b pushed a commit that referenced this pull request Jul 30, 2026
…TAMP_NTZ/LTZ(p)

### What changes were proposed in this pull request?

This PR adds support for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) in `+/- ANSI year-month interval` arithmetic.

Concretely:
- Extends `TimestampAddYMInterval` input typing to accept nanos timestamp types alongside the existing microsecond timestamp types.
- Adds nanos-aware execution/codegen paths that apply the month shift on epoch micros while carrying the `nanosWithinMicro` remainder through unchanged.
- Routes nanos timestamps into `TimestampAddYMInterval` from `BinaryArithmeticWithDatetimeResolver` for both `Add` and `Subtract` (the year-month branches previously matched only `TimestampType | TimestampNTZType`, so nanos timestamps fell through to an unresolved `Add` / `Subtract` and failed analysis).
- Adds a `DateTimeUtils.timestampNanosAddMonths` helper.
- Adds catalyst and SQL test coverage for NTZ/LTZ nanos year-month interval arithmetic parity, and regenerates the impacted nanos SQL golden files.

### Why are the changes needed?
Spark already supports timestamp +/- ANSI year-month interval for the microsecond timestamp families, and SPARK-57501 added the day-time-interval parity for the nanos families, but year-month intervals were still unsupported for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`). This left valid datetime arithmetic unsupported for nanos types.

These changes close that parity gap while preserving nanos precision semantics and existing LTZ/NTZ timezone behavior.

### Does this PR introduce _any_ user-facing change?

Yes.

`TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p in [7, 9]`) now support `+/- ANSI year-month interval` arithmetic.

Examples:
- `TIMESTAMP_NTZ '2020-01-02 03:04:05.123456789' + INTERVAL '1' YEAR`
- `TIMESTAMP_LTZ '2020-01-02 03:04:05.123456789 UTC' - INTERVAL '1-2' YEAR TO MONTH`

### How was this patch tested?
- `build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.DateExpressionsSuite org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite'`
- `SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos" -z "interval"'`
- `build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z "timestamp-ntz-nanos" -z "timestamp-ltz-nanos"'`
- `build/sbt catalyst/scalastyle catalyst/Test/scalastyle`

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

Closes #57636 from stevomitric/stevomitric/spark-57825-nanos.

Authored-by: Stevo Mitric <stevomitric2000@gmail.com>
Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
(cherry picked from commit 88697a3)
Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants