[SPARK-57837][SQL] Support the precision argument in current_timestamp(p) and localtimestamp(p) - #57697
Open
stevomitric wants to merge 6 commits into
Conversation
…p(p) and localtimestamp(p) ### What changes were proposed in this pull request? This PR adds an optional fractional-seconds precision argument `p` to `current_timestamp(p)` / `now(p)` and `localtimestamp(p)`, mirroring the existing `current_time(p)`. It is part of the umbrella SPARK-56822 (timestamps with nanosecond precision). - `p == 6` (and the no-argument forms) keep the historical microsecond `TIMESTAMP` / `TIMESTAMP_NTZ` types unchanged. - `p` in `[7, 9]` returns a nanosecond `TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, gated behind `spark.sql.timestampNanosTypes.enabled`. - Any other precision is rejected with `INVALID_TIMESTAMP_PRECISION`, matching the `TIMESTAMP(p)` type parser (the range check runs before the feature-flag check). The micro leaf expressions `CurrentTimestamp` / `Now` / `LocalTimestamp` are kept as-is to preserve their rendering and pattern matching; two new placeholder expressions `CurrentTimestampNanos` / `LocalTimestampNanos` carry the precision, and `CurrentTimestampExpressionBuilder` / `LocalTimestampExpressionBuilder` dispatch on the argument count. `ComputeCurrentTime` constant-folds the new expressions into query-stable nanosecond literals, just like the microsecond variants. ### Why are the changes needed? ANSI SQL / Foundation defines an optional `<timestamp precision>` on `CURRENT_TIMESTAMP` and `LOCALTIMESTAMP` (Feature F555). This lets users obtain the current time at nanosecond precision when the nanosecond timestamp types are enabled. ### Does this PR introduce any user-facing change? Yes. `current_timestamp(p)` / `now(p)` / `localtimestamp(p)` now accept a precision argument. The no-argument forms are unchanged. ### How was this patch tested? - Catalyst unit tests: `ComputeCurrentTimeSuite` (folding, query-stability, precision flooring, time flow) and `DateExpressionsSuite` (the two expression builders, including the error paths). - End-to-end tests in `TimestampNanosFunctionsSuiteBase` (ANSI on and off): result types, precision flooring, query-stable self-equality, out-of-range precision, and the disabled-flag error. - New golden cases in `timestamp-ltz-nanos.sql` / `timestamp-ntz-nanos.sql`; regenerated `sql-expression-schema.md`. Co-authored-by: Isaac
…nuous processing `LocalTimestampNanos` is a standalone `LeafExpression`, not a subclass of `LocalTimestamp`, so the continuous-processing reject guards in `ContinuousExecution` and `UnsupportedOperationChecker` -- which match `_: CurrentTimestampLike | _: CurrentDate | _: LocalTimestamp` -- let `localtimestamp(p)` slip through, while its LTZ sibling `current_timestamp(p)` (`CurrentTimestampNanos`, which extends `CurrentTimestampLike`) is correctly rejected. Add `_: LocalTimestampNanos` to both guards so the nanosecond NTZ current-time function is rejected consistently with the microsecond and LTZ forms, and add a `ContinuousSuite` test covering current_timestamp(9) / now(9) / localtimestamp(9). Co-authored-by: Isaac
…tamp(p) to the batch timestamp in micro-batch streaming In micro-batch streaming, current_timestamp()/localtimestamp() are rewritten to the replay-stable CurrentBatchTimestamp before the optimizer runs. That rewrite in MicroBatchExecution only matched the microsecond CurrentTimestamp, LocalTimestamp, and CurrentDate, so the new CurrentTimestampNanos / LocalTimestampNanos leaves fell through to ComputeCurrentTime and were folded to a wall-clock-at-planning literal instead of the batch timestamp -- not replay-stable across restarts, and diverging from the microsecond form. Add the two nanosecond leaves to the MicroBatchExecution rewrite and extend CurrentBatchTimestamp.toLiteral to emit TIMESTAMP_LTZ(p) / TIMESTAMP_NTZ(p) literals. The batch timestamp is millisecond resolution, so nanos-within-micro is always 0; TIMESTAMP_LTZ is instant-based and TIMESTAMP_NTZ takes the wall clock in the session zone, mirroring the existing microsecond branches. Add a StreamSuite test asserting the collected nanosecond value is millisecond-aligned, which a wall-clock fold would not satisfy. Co-authored-by: Isaac
Registering `now` via `CurrentTimestampExpressionBuilder` (annotated `since = "1.5.0"`, shared with `current_timestamp`) regressed `now`'s documented version from 1.6.0 to 1.5.0, since `FunctionRegistryBase` falls back to the builder annotation when no explicit `since` is given. Pass the original 1.6.0 explicitly at the registration site, mirroring the `localtime` alias. Co-authored-by: Isaac
…7837-current-timestamp-precision # Conflicts: # sql/core/src/test/resources/sql-tests/analyzer-results/timestamp-ltz-nanos.sql.out # sql/core/src/test/resources/sql-tests/analyzer-results/timestamp-ntz-nanos.sql.out # sql/core/src/test/resources/sql-tests/inputs/timestamp-ltz-nanos.sql # sql/core/src/test/resources/sql-tests/inputs/timestamp-ntz-nanos.sql # sql/core/src/test/resources/sql-tests/results/timestamp-ltz-nanos.sql.out # sql/core/src/test/resources/sql-tests/results/timestamp-ntz-nanos.sql.out
uros-b
reviewed
Aug 2, 2026
uros-b
reviewed
Aug 2, 2026
uros-b
left a comment
uros-b
left a comment
Member
There was a problem hiding this comment.
Left a few comments, but otherwise looks good! Thank you @stevomitric
stevomitric
changed the title
…x import order - Prefix the nanosecond `> SELECT current_timestamp(9)` / `localtimestamp(9)` examples with `> SET spark.sql.timestampNanosTypes.enabled=true;` so the "Documentation generation" job (which runs each example live with the flag off) no longer fails with FEATURE_NOT_ENABLED. Matches the convention used by DayOfYear / NanosToTimestamp / current_time; add line.contains.tab to the scalastyle:off directives for the echo line. - Reorder the org.apache.spark import in DateExpressionsSuite so SPARK_DOC_ROOT comes before SparkUpgradeException, fixing the scalastyle import-order error. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds an optional fractional-seconds precision argument
ptocurrent_timestamp(p)/now(p)andlocaltimestamp(p), mirroring the existingcurrent_time(p). It is part of the umbrella SPARK-56822 (timestamps with nanosecond precision).p == 6(and the no-argument forms) keep the historical microsecondTIMESTAMP/TIMESTAMP_NTZtypes unchanged.pin[7, 9]returns a nanosecondTIMESTAMP_LTZ(p)/TIMESTAMP_NTZ(p), gated behindspark.sql.timestampNanosTypes.enabled.INVALID_TIMESTAMP_PRECISION, matching theTIMESTAMP(p)type parser (the range check runs before the feature-flag check).The micro leaf expressions
CurrentTimestamp/Now/LocalTimestampare kept as-is to preserve their rendering and pattern matching; two new placeholder expressionsCurrentTimestampNanos/LocalTimestampNanoscarry the precision, andCurrentTimestampExpressionBuilder/LocalTimestampExpressionBuilderdispatch on the argument count.ComputeCurrentTimeconstant-folds the new expressions into query-stable nanosecond literals, just like the microsecond variants.Why are the changes needed?
ANSI SQL / Foundation defines an optional
<timestamp precision>onCURRENT_TIMESTAMPandLOCALTIMESTAMP(Feature F555). This lets users obtain the current time at nanosecond precision when the nanosecond timestamp types are enabled.Does this PR introduce any user-facing change?
Yes.
current_timestamp(p)/now(p)/localtimestamp(p)now accept a precision argument. The no-argument forms are unchanged.How was this patch tested?
ComputeCurrentTimeSuite(folding, query-stability, precision flooring, time flow) andDateExpressionsSuite(the two expression builders, including the error paths).TimestampNanosFunctionsSuiteBase(ANSI on and off): result types, precision flooring, query-stable self-equality, out-of-range precision, and the disabled-flag error.timestamp-ltz-nanos.sql/timestamp-ntz-nanos.sql; regeneratedsql-expression-schema.md.Was this patch authored or co-authored using generative AI tooling?
Co-authored: Claude Code (Claude Opus 4.8)