What is the problem the feature request solves?
This epic tracks support for the full set of Spark date/time expressions in Comet. Each expression below has a checkbox indicating whether it is implemented today. In-flight work is linked to the relevant PR. Versions in parentheses indicate the Spark version that introduced the expression; unmarked entries are present in all supported Spark versions (3.4 / 3.5 / 4.0 / 4.1).
The canonical list is derived from FunctionRegistry.scala in Spark 4.0.1 and Spark 4.1.1 (the // datetime functions section), augmented with the TimeType expressions defined in timeExpressions.scala (4.1+).
Date / Timestamp functions
Interval construction
Timestamp arithmetic (related catalyst expressions)
Time-window expressions
TimeType expressions (Spark 4.1+)
See umbrella issue #4288 for the full TimeType implementation plan.
Related work
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
This epic tracks support for the full set of Spark date/time expressions in Comet. Each expression below has a checkbox indicating whether it is implemented today. In-flight work is linked to the relevant PR. Versions in parentheses indicate the Spark version that introduced the expression; unmarked entries are present in all supported Spark versions (3.4 / 3.5 / 4.0 / 4.1).
The canonical list is derived from
FunctionRegistry.scalain Spark 4.0.1 and Spark 4.1.1 (the// datetime functionssection), augmented with theTimeTypeexpressions defined intimeExpressions.scala(4.1+).Date / Timestamp functions
AddMonths(add_months) - feat: expand date/time expression support using codegen dispatcher #4417CurrentDate(current_date,curdate) - typically constant-folded by Spark before plan rewriting; only relevant when constant folding is disabled (fix: Support scalar inputs for datetime expressions when constant folding is disabled #3448)CurrentTimestamp(current_timestamp) - typically constant-folded; see aboveNow(now) - typically constant-folded; see aboveCurrentTimeZone(current_timezone)LocalTimestamp(localtimestamp)DateDiff(datediff,date_diff)DateAdd(date_add,dateadd)DateFormatClass(date_format)DateSub(date_sub)DayOfMonth(day,dayofmonth)DayOfYear(dayofyear)FromUnixTime(from_unixtime)FromUTCTimestamp(from_utc_timestamp)Hour(hour)LastDay(last_day)Minute(minute)Month(month)MonthsBetween(months_between) - feat: expand date/time expression support using codegen dispatcher #4417NextDay(next_day)Quarter(quarter)Second(second)ParseToTimestamp(to_timestamp) - partial via feat: add support forGetTimestamp,parse_to_dateexpressions #3467ParseToTimestampNTZ(to_timestamp_ntz)ParseToTimestampLTZ(to_timestamp_ltz)ParseToDate(to_date) - feat: add support forGetTimestamp,parse_to_dateexpressions #3467ToUnixTimestamp(to_unix_timestamp) - feat: expand date/time expression support using codegen dispatcher #4417ToUTCTimestamp(to_utc_timestamp)TruncDate(trunc)TruncTimestamp(date_trunc)UnixTimestamp(unix_timestamp)DayOfWeek(dayofweek)WeekDay(weekday)DayName(dayname) (3.5+)WeekOfYear(weekofyear)Year(year)MakeDate(make_date)MakeTimestamp(make_timestamp) - feat: expand date/time expression support using codegen dispatcher #4417MakeTimestampNTZ(make_timestamp_ntz) - [Feature] Support Spark expression: make_timestamp_ntz #3108MakeTimestampLTZ(make_timestamp_ltz)TryMakeTimestamp(try_make_timestamp) - [Feature] Support Spark expression: try_make_timestamp #3117TryMakeTimestampNTZ(try_make_timestamp_ntz)TryMakeTimestampLTZ(try_make_timestamp_ltz)MonthName(monthname) (4.0+)Extract(extract(field FROM source))DatePart(date_part,datepart)DateFromUnixDate(date_from_unix_date)UnixDate(unix_date)SecondsToTimestamp(timestamp_seconds)MillisToTimestamp(timestamp_millis) - feat: expand date/time expression support using codegen dispatcher #4417MicrosToTimestamp(timestamp_micros) - feat: expand date/time expression support using codegen dispatcher #4417UnixSeconds(unix_seconds) - feat: expand date/time expression support using codegen dispatcher #4417UnixMillis(unix_millis) - feat: expand date/time expression support using codegen dispatcher #4417UnixMicros(unix_micros) - feat: expand date/time expression support using codegen dispatcher #4417ConvertTimezone(convert_timezone)Interval construction
MakeInterval(make_interval) - [Feature] Support Spark expression: make_interval #3099MakeDTInterval(make_dt_interval) - feat: Support Spark expression: make_dt_interval #4338MakeYMInterval(make_ym_interval) - [Feature] Support Spark expression: make_ym_interval #3100TryMakeInterval- [Feature] Support Spark expression: try_make_interval #3103Timestamp arithmetic (related catalyst expressions)
TimestampAdd(timestampadd) - [Feature] Support Spark expression: timestamp_add #3113TimestampAddInterval- [Feature] Support Spark expression: timestamp_add_interval #3114TimestampAddYMInterval- [Feature] Support Spark expression: timestamp_add_ym_interval #3115TimestampDiff(timestampdiff) - [Feature] Support Spark expression: timestamp_diff #3116SubtractDates- [Feature] Support Spark expression: subtract_dates #3094SubtractTimestamps- [Feature] Support Spark expression: subtract_timestamps #3112DateTimeSub- [Feature] Support Spark expression: datetime_sub #3134Time-window expressions
TimeWindow(window)SessionWindow(session_window)WindowTime(window_time) - feat: Support Spark expression window_time #3732 (viaPreciseTimestampConversion)TimeType expressions (Spark 4.1+)
See umbrella issue #4288 for the full TimeType implementation plan.
ToTime(to_time) - default format onlyTryToTime(try_to_time) - default format onlyMakeTime(make_time)CurrentTime(current_time) - typically constant-foldedHoursOfTime(extract(HOUR FROM t)) - [Feature] Support Spark expression: hours_of_time #3126, feat: Support Spark expression hours_of_time #3615MinutesOfTime(extract(MINUTE FROM t)) - feat: Support Spark expression minutes_of_time #3447SecondsOfTime(extract(SECOND FROM t)) - [Feature] Support Spark expression: seconds_of_time #3129, feat: Support Spark expression seconds_of_time #3618SecondsOfTimeWithFractionTimeFromSeconds(time_from_seconds)TimeFromMillis(time_from_millis)TimeFromMicros(time_from_micros)TimeToSeconds(time_to_seconds)TimeToMillis(time_to_millis)TimeToMicros(time_to_micros)TimeAddIntervalSubtractTimesTimeDiff(time_diff) - [Feature] Support Spark expression: time_diff #3122TimeTrunc(time_trunc) - [Feature] Support Spark expression: time_trunc #3123Related work
current_date/current_timestamp/nowregression cases)date_truncincorrect results in non-UTC timezonehour/minute/secondincorrect timezone handling forTimestampNTZDescribe the potential solution
No response
Additional context
No response