[SPARK-8995][SQL] cast date strings like '2015-01-01 12:15:31' to date #7353

tarekbecker · 2015-07-11T02:30:53Z

Jira https://issues.apache.org/jira/browse/SPARK-8995

In PR #6981we noticed that we cannot cast date strings that contains a time, like '2015-03-18 12:39:40' to date. Besides it's not possible to cast a string like '18:03:20' to a timestamp.

If a time is passed without a date, today is inferred as date.

…timestamp

cloud-fan · 2015-07-11T07:42:39Z

This was by intentional as hive doesn't support it. However, our new design about date time says we should support cast string yyyy-mm-dd * to date, which means your case should be supported.
It would be better if you can fix the string to date cast completely, i.e. including all the formats defined in the design doc :)

tarekbecker · 2015-07-11T16:21:53Z

Okay, yes I'm going to extend it. What Hive does support is parsing the minute from 13:20:08. Our minute method requires a TimeStamp. Either we don't support the minute expression on pure time information or we define a cast from a time string to timestamp. But even if we do that, we are still not compatible with Hive. Because we would parse a pure date like 2015-10-20 to 2015-10-20 00:00:00 and we could extract the minute 0, Hive would return null

tarekbecker · 2015-07-11T23:06:20Z

@cloud-fan
The following strings can be parsed now: (String --> Date)
yyyy, yyyy-[m]m, yyyy-[m]m-[d]d, yyyy-[m]m-[d]d, yyyy-[m]m-[d]d *, yyyy-[m]m-[d]dT*

(String -> Timestamp)
yyyy, yyyy-[m]m, yyyy-[m]m-[d]d, yyyy-[m]m-[d]d, yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][ms], yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][ms]Z, yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][ms]-[h]h:[m]m, yyyy-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][ms]+[h]h:[m]m, yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][ms], yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][ms]Z, yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][ms]-[h]h:[m]m, yyyy-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][ms]+[h]h:[m]m

yhuai · 2015-07-13T05:10:01Z

ok to test

SparkQA · 2015-07-13T05:31:04Z

Test build #37124 has finished for PR 7353 at commit 71f89c1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-13T07:15:46Z

Test build #1051 has finished for PR 7353 at commit 71f89c1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-13T09:04:32Z

Test build #37129 has finished for PR 7353 at commit cfbaed7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-07-13T16:55:48Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

-        try DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf(n))
-        catch { case _: java.lang.IllegalArgumentException => null }
+        val parsedDateString = DateTimeUtils.stringToTimestamp(utfs)
+        if (parsedDateString == null) null else DateTimeUtils.fromJavaTimestamp(parsedDateString)


This is a Timestamp object, should we have a better name? parsedTime?

I'm going to adjust this.

davies · 2015-07-13T17:35:17Z

@tarekauel Thanks for working on this, it's in a good shape, just a few comments.

As you mentioned in the description, 18:03:20 could be parsed as timestamp, but it's still not done yet?

tarekbecker · 2015-07-13T18:01:41Z

@davies thanks for all your good comments. I'm going to incorporate your suggestions.

I don't accept 18:03:20 , yet. The design document doesn't allow this. I think we should parse a pure time string to today + time. But I wanted to double check this with you guys.

SparkQA · 2015-07-14T01:02:54Z

Test build #37168 has finished for PR 7353 at commit 71622c0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-14T02:43:42Z

Test build #37171 has finished for PR 7353 at commit 34ec573.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tarekbecker · 2015-07-14T03:03:30Z

@davies How should we deal with this? I don't know the value of 'value, but it seems to be something that the parser can parse can be parsed to 1.1.1970.
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37171/testReport/org.apache.spark.sql.hive.execution/HiveQuerySuite/Cast_Timestamp_to_Timestamp_in_UDF/

Update: My assumption is that value is a numeric value (see links below). Because of that 238 could
be interpreted as long value since 1970. So I don't know why Hives answer is null

https://github.com/apache/spark/blob/master/sql/hive/src/test/resources/data/scripts/q_test_init.sql#L4-L8
https://github.com/apache/spark/blob/master/sql/hive/src/test/resources/data/files/kv1.txt#L1

SparkQA · 2015-07-14T04:08:29Z

Test build #37177 has finished for PR 7353 at commit 01c9ff3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-07-15T17:14:01Z

In master, we can't cast '238' as timestamp,

spark-sql> select DATEDIFF(CAST(238 AS timestamp), CAST('2002-03-21 00:00:00' AS timestamp));
-11767
Time taken: 0.171 seconds, Fetched 1 row(s)
spark-sql> select DATEDIFF(CAST('238' AS timestamp), CAST('2002-03-21 00:00:00' AS timestamp));
NULL
Time taken: 0.1 seconds, Fetched 1 row(s)

I think you should check that the year is larger than 1000, or we do not support casting from YYYY

tarekbecker · 2015-07-15T18:08:25Z

@davies somehow Jenkins wasn't able to fetch from GitHub. Could you trigger Jenkins, again?

SparkQA · 2015-07-15T18:59:42Z

Test build #1075 has finished for PR 7353 at commit ef05753.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-15T23:40:03Z

Test build #37415 has finished for PR 7353 at commit d20b8b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-16T05:05:04Z

Test build #37454 has finished for PR 7353 at commit ca1ae69.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-07-16T06:20:58Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala

+      c.getTimeInMillis * 1000)
+    c.set(2015, 0, 1, 0, 0, 0)
+    c.set(Calendar.MILLISECOND, 0)
+    assert(DateTimeUtils.stringToTimestamp(UTF8String.fromString("2015")).get ==


It's better to use === in tests (for better failure message).

davies · 2015-07-16T06:34:40Z

LGTM, just some minor comments.

SparkQA · 2015-07-16T09:09:18Z

Test build #37476 has finished for PR 7353 at commit 14f333b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-07-16T15:26:19Z

Thanks for the update, merging this into master!

a follow up of #7353 1. we should use `Calendar.HOUR_OF_DAY` instead of `Calendar.HOUR`(this is for AM, PM). 2. we should call `c.set(Calendar.MILLISECOND, 0)` after `Calendar.getInstance` I'm not sure why the tests didn't fail in jenkins, but I ran latest spark master branch locally and `DateTimeUtilsSuite` failed. Author: Wenchen Fan <cloud0fan@outlook.com> Closes #7473 from cloud-fan/datetime and squashes the following commits: 66cdaf2 [Wenchen Fan] fix several bugs in DateTimeUtils.stringToTimestamp

fix 2 bugs introduced in #7353 1. we should use UTC Calendar when cast string to date . Before #7353 , we use `DateTimeUtils.fromJavaDate(Date.valueOf(s.toString))` to cast string to date, and `fromJavaDate` will call `millisToDays` to avoid the time zone issue. Now we use `DateTimeUtils.stringToDate(s)`, we should create a Calendar with UTC in the begging. 2. we should not change the default time zone in test cases. The `threadLocalLocalTimeZone` and `threadLocalTimestampFormat` in `DateTimeUtils` will only be evaluated once for each thread, so we can't set the default time zone back anymore. Author: Wenchen Fan <cloud0fan@outlook.com> Closes #7488 from cloud-fan/datetime and squashes the following commits: 9cd6005 [Wenchen Fan] address comments 21ef293 [Wenchen Fan] fix 2 bugs in datetime

[SPARK-8995] cast date strings like '2015-01-01 12:15:31' to date or …

c1083fb

…timestamp

tarekbecker mentioned this pull request Jul 11, 2015

[SPARK-8199][SPARK-8184][SPARK-8183][SPARK-8182][SPARK-8181][SPARK-8180][SPARK-8179][SPARK-8177][SPARK-8178][SPARK-9115][SQL] date functions #6981

Closed

tarekbecker added 2 commits July 11, 2015 16:03

[SPARK-8995] date and timestamp cast

30e5aec

[SPARK-8995] removed old timestamp parsing

f7452fa

[SPARK-8995] minor style fix

71f89c1

fixed wrong checks

cfbaed7

Hive compatibility

0e30c0a

davies reviewed Jul 13, 2015
View reviewed changes

improved timestamp and date parsing

71622c0

fixed style

34ec573

[SPARK-8995] support for time strings

01c9ff3

[SPARK-8995] added check for year >= 1000

ef05753

[SPARK-8995] bug fix: distinguish between 0 and null

d20b8b4

[SPARK-8995] style fix

ca1ae69

davies reviewed Jul 16, 2015
View reviewed changes

[SPARK-8995] added tests for daylight saving time

14f333b

asfgit closed this in 4ea6480 Jul 16, 2015

cloud-fan mentioned this pull request Jul 17, 2015

[SPARK-9136][SQL] fix several bugs in DateTimeUtils.stringToTimestamp #7473

Closed

cloud-fan mentioned this pull request Jul 18, 2015

[SPARK-9167][SQL] use UTC Calendar in stringToDate #7488

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8995][SQL] cast date strings like '2015-01-01 12:15:31' to date #7353

[SPARK-8995][SQL] cast date strings like '2015-01-01 12:15:31' to date #7353

tarekbecker commented Jul 11, 2015

cloud-fan commented Jul 11, 2015

tarekbecker commented Jul 11, 2015

tarekbecker commented Jul 11, 2015

yhuai commented Jul 13, 2015

SparkQA commented Jul 13, 2015

SparkQA commented Jul 13, 2015

SparkQA commented Jul 13, 2015

davies Jul 13, 2015

tarekbecker Jul 13, 2015

davies commented Jul 13, 2015

tarekbecker commented Jul 13, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 14, 2015

tarekbecker commented Jul 14, 2015

SparkQA commented Jul 14, 2015

davies commented Jul 15, 2015

tarekbecker commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 16, 2015

davies Jul 16, 2015

davies commented Jul 16, 2015

SparkQA commented Jul 16, 2015

davies commented Jul 16, 2015

[SPARK-8995][SQL] cast date strings like '2015-01-01 12:15:31' to date #7353

[SPARK-8995][SQL] cast date strings like '2015-01-01 12:15:31' to date #7353

Conversation

tarekbecker commented Jul 11, 2015

cloud-fan commented Jul 11, 2015

tarekbecker commented Jul 11, 2015

tarekbecker commented Jul 11, 2015

yhuai commented Jul 13, 2015

SparkQA commented Jul 13, 2015

SparkQA commented Jul 13, 2015

SparkQA commented Jul 13, 2015

davies Jul 13, 2015

Choose a reason for hiding this comment

tarekbecker Jul 13, 2015

Choose a reason for hiding this comment

davies commented Jul 13, 2015

tarekbecker commented Jul 13, 2015

SparkQA commented Jul 14, 2015

SparkQA commented Jul 14, 2015

tarekbecker commented Jul 14, 2015

SparkQA commented Jul 14, 2015

davies commented Jul 15, 2015

tarekbecker commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 15, 2015

SparkQA commented Jul 16, 2015

davies Jul 16, 2015

Choose a reason for hiding this comment

davies commented Jul 16, 2015

SparkQA commented Jul 16, 2015

davies commented Jul 16, 2015