[WIP][SPARK-57814][SQL] Support NTZ and nanosecond-precision timestamps in unix_seconds, unix_millis and unix_micros#57085
Open
stevomitric wants to merge 6 commits into
Conversation
… unix_seconds, unix_millis and unix_micros Widen the input types of the unix_seconds / unix_millis / unix_micros functions (TimestampToLongBase) to accept TIMESTAMP_NTZ and the nanosecond-precision timestamp types TIMESTAMP_LTZ(p) / TIMESTAMP_NTZ(p) (p in [7, 9]) in addition to the existing microsecond TIMESTAMP_LTZ. Previously these functions only accepted TIMESTAMP_LTZ, so callers had to cast NTZ or nanosecond timestamps first. For the nanosecond carrier the child evaluates to a TimestampNanosVal, so nullSafeEval reads its epochMicros field and doGenCode routes the value through a toMicros helper in both the identity (unix_micros) and floorDiv (unix_seconds/millis) branches. Math.floorDiv is preserved so sub-second/sub-microsecond digits truncate toward negative infinity, matching the existing semantics. Adds a DateExpressionsSuite case and end-to-end coverage in TimestampNanosFunctionsSuiteBase, plus golden updates for the timestamp-ltz-nanos / timestamp-ntz-nanos SQL tests. Co-authored-by: Isaac
…ge in unix_* test The DateExpressionsSuite comment said micro TIMESTAMP_NTZ "was rejected before" this change. That mischaracterizes the prior behavior: it was not rejected -- implicit type coercion silently cast it to TIMESTAMP_LTZ, which applies a session-zone shift (Cast NTZ->TIMESTAMP uses convertTz(ts, sessionZone, UTC)). So under a non-UTC session zone the result silently changes for existing micro-NTZ callers; the new zoneless direct read is the more correct behavior. Reword the comment to state the prior behavior accurately, and add a contrasting pair of assertions: the new direct read is session-zone- independent, while the equivalent old coercion (Cast to TIMESTAMP_LTZ) under America/Los_Angeles shifts the same wall clock by +8h to a different value. This makes the behavior change visible and guards against silently reintroducing the old shift. Co-authored-by: Isaac
…ros docs The @ExpressionDescription for the three widened functions only showed a micro TIMESTAMP example, so the docs didn't reflect the new NTZ/nanosecond capability. Add a nanos example to each, mirroring UnixNanos: enable spark.sql.timestampNanosTypes.enabled and evaluate over a TIMESTAMP_NTZ literal with sub-microsecond digits. The example values are the same instant UnixNanos documents (epochMicros 1230219000123456, sub-micro remainder truncated): 1230219000 / 1230219000123 / 1230219000123456. Extend the scalastyle directive to `line.contains.tab` (the SET-output line carries a literal tab, as in the UnixNanos example). Co-authored-by: Isaac
…Z change unix_seconds/unix_millis/unix_micros now accept TIMESTAMP_NTZ (and nanos) directly with no zone shift. A micro TIMESTAMP_NTZ argument previously was not rejected -- it was implicitly cast to TIMESTAMP_LTZ, which applies a session-zone shift -- so under a non-UTC session zone the result silently changes for existing NTZ callers. Document this in the 4.2->4.3 upgrade section, including how to recover the old value (explicit cast to TIMESTAMP_LTZ). Co-authored-by: Isaac
…ide example The migration-guide recovery example cast to bare `TIMESTAMP`, which resolves to TIMESTAMP_NTZ when spark.sql.timestampType = TIMESTAMP_NTZ -- a no-op that would not reproduce the old LTZ-coerced value. Use the explicit `TIMESTAMP_LTZ` cast so the example unambiguously restores the previous behavior, matching the surrounding prose. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Widen the input types of the
unix_seconds/unix_millis/unix_microsfunctions to acceptTIMESTAMP_NTZand the nanosecond-precision timestamp types.For the nanosecond carrier the child evaluates to a
TimestampNanosVal, sonullSafeEvalreads itsepochMicrosfield anddoGenCoderoutes the value through atoMicroshelper.Why are the changes needed?
Previously these functions declared
inputTypes = Seq(TimestampType), so onlyTIMESTAMP_LTZwas accepted directly. ATIMESTAMP_NTZor nanosecond timestamp argument was not rejected, though: implicit type coercion silently cast it toTIMESTAMP_LTZ(which applies a session-zone shift), so callers got a result rather than an error. This change makes the functions accept those types directly and read them without a zone shift, bringing them to parity across the timestamp families.Does this PR introduce any user-facing change?
Yes.
unix_seconds/unix_millis/unix_microsnow acceptTIMESTAMP_NTZand nanosecondtimestamps directly.
Note this is a behavior change, not only a new capability: a micro
TIMESTAMP_NTZargument previously worked via an implicit cast toTIMESTAMP_LTZthat applied a session-zone shift.It is now read directly with no shift, so under a non-UTC session zone existing micro-NTZ callers get a different (zone-correct) result.
TIMESTAMP_NTZis zoneless, so the new value is the correct one.How was this patch tested?
Added a
DateExpressionsSuitecase and end-to-end coverage inTimestampNanosFunctionsSuiteBase.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8