Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-35716][SQL] Support casting of timestamp without time zone to date type #32869

Closed
wants to merge 3 commits into from
Closed

[SPARK-35716][SQL] Support casting of timestamp without time zone to date type #32869

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
89376af
save for now
gengliangwang Jun 10, 2021
5241e21
cast to date
gengliangwang Jun 10, 2021
dde4cd8
revise
gengliangwang Jun 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 9 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
View file
Open in desktop
Expand Up @@ -72,6 +72,7 @@ object Cast {

case (StringType, DateType) => true
case (TimestampType, DateType) => true
case (TimestampWithoutTZType, DateType) => true

case (StringType, CalendarIntervalType) => true
case (StringType, DayTimeIntervalType) => true
Expand Down Expand Up @@ -534,6 +535,8 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
// throw valid precision more than seconds, according to Hive.
// Timestamp.nanos is in 0 to 999,999,999, no more than a second.
buildCast[Long](_, t => microsToDays(t, zoneId))
case TimestampWithoutTZType =>
buildCast[Long](_, t => microsToDays(t, ZoneOffset.UTC))
}

// IntervalConverter
Expand Down Expand Up @@ -1204,6 +1207,11 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
(c, evPrim, evNull) =>
code"""$evPrim =
org.apache.spark.sql.catalyst.util.DateTimeUtils.microsToDays($c, $zid);"""
case TimestampWithoutTZType =>
(c, evPrim, evNull) =>
// scalastyle:off line.size.limit
code"$evPrim = org.apache.spark.sql.catalyst.util.DateTimeUtils.microsToDays($c, java.time.ZoneOffset.UTC);"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we can add a protected def dtUtilCls = classOf[DateTimeUtils].getClass.getName in CaseBase, to shorten the code here and other places.

// scalastyle:on line.size.limit
case _ =>
(c, evPrim, evNull) => code"$evNull = true;"
}
Expand Down Expand Up @@ -1953,6 +1961,7 @@ object AnsiCast {

case (StringType, DateType) => true
case (TimestampType, DateType) => true
case (TimestampWithoutTZType, DateType) => true

case (_: NumericType, _: NumericType) => true
case (StringType, _: NumericType) => true
Expand Down
11 changes: 9 additions & 2 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
View file
Open in desktop
Expand Up @@ -18,7 +18,7 @@
package org.apache.spark.sql.catalyst.expressions

import java.sql.{Date, Timestamp}
import java.time.{DateTimeException, Duration, LocalDateTime, Period}
import java.time.{DateTimeException, Duration, LocalDate, LocalDateTime, Period}
import java.time.temporal.ChronoUnit
import java.util.{Calendar, TimeZone}

Expand Down Expand Up @@ -1256,10 +1256,17 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {

test("SPARK-35711: cast timestamp without time zone to timestamp with local time zone") {
specialTs.foreach { s =>
val dt = LocalDateTime.parse(s.replace(" ", "T"))
val dt = LocalDateTime.parse(s)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a trivial change for the test case. There is no need to replace the space.

checkEvaluation(cast(dt, TimestampType), DateTimeUtils.localDateTimeToMicros(dt))
}
}

test("SPARK-35716: cast timestamp without time zone to date type") {
specialTs.foreach { s =>
val dt = LocalDateTime.parse(s)
checkEvaluation(cast(dt, DateType), LocalDate.parse(s.split("T")(0)))
}
}
}

/**
Expand Down