[SPARK-57829][SQL] Support window, session_window and window_time over nanosecond-precision timestamps#56951
[SPARK-57829][SQL] Support window, session_window and window_time over nanosecond-precision timestamps#56951yadavay-amzn wants to merge 8 commits into
Conversation
| import org.apache.spark.sql.catalyst.util.DateTimeConstants.{NANOS_PER_DAY, NANOS_PER_MICROS} | ||
|
|
||
| override def nullIntolerant: Boolean = true | ||
| override def inputTypes: Seq[AbstractDataType] = Seq(CalendarIntervalType) |
There was a problem hiding this comment.
We shouldn't support the legacy type CalendarIntervalType.
There was a problem hiding this comment.
Done - replaced the custom CalendarIntervalToNanos with CalendarIntervalToMicros typed as DayTimeIntervalType, so the nanosecond session-end now goes through the supported, timezone-aware TimestampAddInterval path. No legacy CalendarIntervalType in the nanos computation; window/session tests pass.
|
Following the review discussion on #56944 and #56984 (thanks @uros-b), this PR now carries the consolidated window-struct-over-nanos watermark handling: the CheckAnalysis window-struct guard accepts a nanosecond end field, and the statefulOperators eviction struct-branch extracts endFieldType via watermarkLiteral. That handling depends on the watermark infrastructure from #56944 (SPARK-57830), so this PR is stacked on it - the diff shows #56944's commits until it merges, after which I will rebase onto master. Added e2e tests: window() over a nanosecond event-time column + withWatermark, asserting nanosecond window bounds and state eviction. |
…n timestamp columns
…tesWithinWatermark until SPARK-57843
…iction test (defer window-struct handling to SPARK-57829)
…r nanosecond-precision timestamps
…IntervalType for nanosecond windows
…epted-types error
…ling Widen the CheckAnalysis EventTimeWatermark struct guard to accept nanosecond-precision end fields (AnyTimestampNanoType), and route the struct eviction branch in WatermarkSupport through watermarkLiteral so that the eviction predicate uses the correct TimestampNanosVal literal for nanos window structs. Add two e2e tests: - window() over a nanos event-time column with watermark eviction (assertNumStateRows + assertNumRowsDroppedByWatermark) - window() over nanos preserves nanosecond precision in bounds These tests exercise the struct endFieldType extraction and fail with a ClassCastException (Long vs TimestampNanosVal) without the change.
2 participants
What changes were proposed in this pull request?
Extends
window,session_window, andwindow_timeto accept nanosecond-precision timestamp columns (TimestampNTZNanosType,TimestampLTZNanosType). Window boundaries are computed on the nanosecond scale: durations are scaled x1000 withMath.multiplyExact, timestamp <-> epoch-nanos round-trips go throughPreciseTimestampNanosConversion, andwindow_timesubtracts 1 nanosecond (vs 1 microsecond for micros input). The window start/end preserve the input's nanosecond precision.This also updates the batch
session_windowaggregation iterators (MergingSessionsIterator/UpdatingSessionsIterator) to read and compare session boundaries via the nanosecond-awareTimestampNanosValaccessors through a sharedSessionNanosHelper. These sort-merge iterators are on the shared (non-streaming) batch aggregation path, andgetLongwould misread the variable-lengthTimestampNanosValrepresentation - so the feature cannot function correctly without them.Because this PR is what makes a window struct with a nanosecond
endfield reachable, it also carries the streaming watermark handling for that case (consolidated here from #56944 / #56984 per review, since it was inert until window-over-nanos existed): theCheckAnalysisEventTimeWatermarkguard now accepts a window struct whoseendfield is a nanosecond type, and thestatefulOperatorseviction struct-branch extracts theendFieldTypeand builds the watermark literal viawatermarkLiteral(from SPARK-57830), so state eviction is correct when watermarking a window over a nanosecond event-time column.Why are the changes needed?
Part of nanosecond-precision timestamp support (SPARK-56822), building on the event-time watermark support in SPARK-57830. Previously
window/session_window/window_timerejected nanosecond-precision timestamp columns.Does this PR introduce any user-facing change?
Yes -
window,session_window, andwindow_timenow accept nanosecond-precision timestamp columns, and the resulting window bounds preserve nanosecond precision.How was this patch tested?
New
DataFrameTimeWindowingSuiteandDataFrameSessionWindowingSuitetests for NTZ and LTZ nanosecond inputs, including value-levelcheckAnswerassertions for both (window bounds, session merging, andwindow_timeat nanosecond granularity). New end-to-endEventTimeWatermarkSuitetests exercisewindow()over a nanosecond event-time column withwithWatermarkin Append mode, asserting nanosecond window bounds are preserved and that state is evicted once the watermark advances past the window. All existing window/session tests pass.Was this patch authored or co-authored using generative AI tooling?
Authored with assistance by Claude Opus 4.8.