[SPARK-28141][SQL] Support special date values #25708

MaxGekk · 2019-09-06T08:46:43Z

What changes were proposed in this pull request?

Supported special string values for DATE type. They are simply notational shorthands that will be converted to ordinary date values when read. The following string values are supported:

epoch [zoneId] - 1970-01-01
today [zoneId] - the current date in the time zone specified by spark.sql.session.timeZone.
yesterday [zoneId] - the current date -1
tomorrow [zoneId] - the current date + 1
now - the date of running the current query. It has the same notion as today.

For example:

spark-sql> SELECT date 'tomorrow' - date 'yesterday';
2

Why are the changes needed?

To maintain feature parity with PostgreSQL, see 8.5.1.4. Special Values

Does this PR introduce any user-facing change?

Previously, the parser fails on the special values with the error:

spark-sql> select date 'today';
Error in query: 
Cannot parse the DATE value: today(line 1, pos 7)

After the changes, the special values are converted to appropriate dates:

spark-sql> select date 'today';
2019-09-06

How was this patch tested?

Added tests to DateFormatterSuite to check parsing special values from regular strings.
Tests in DateTimeUtilsSuite check parsing those values from UTF8String
Uncommented tests in date.sql

SparkQA · 2019-09-06T11:29:43Z

Test build #110230 has finished for PR 25708 at commit 775c7ec.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-06T11:36:44Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala

+
+      assert(toDate("epoch", zoneId).get === 0)
+      val today = localDateToDays(LocalDate.now(zoneId))
+      assert(toDate("yesterday", zoneId).get === today - 1)


Here, there is the risk that today contains previous day if we were unlucky and the test was executed at the end of the day. Maybe, need to introduce some tolerance.

SparkQA · 2019-09-06T14:59:34Z

Test build #110236 has finished for PR 25708 at commit be787f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…ecial-values # Conflicts: # sql/core/src/test/resources/sql-tests/results/pgSQL/date.sql.out

SparkQA · 2019-09-06T20:25:58Z

Test build #110255 has finished for PR 25708 at commit 4fb8834.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-06T23:16:53Z

Test build #110258 has finished for PR 25708 at commit 6da7905.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-07T17:19:28Z

@dongjoon-hyun @maropu @cloud-fan Please, review the PR.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala

SparkQA · 2019-09-07T21:03:28Z

Test build #110285 has finished for PR 25708 at commit d57e11e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-09-08T00:04:11Z

The other DBMS-like systems support these syntaxes? If not, we should turn on/off these features by the compatibility mode flag https://issues.apache.org/jira/browse/SPARK-28934?

maropu · 2019-09-08T00:50:00Z

How about the timestamp case?
You dropped inf/-inf in the supported?

maropu · 2019-09-08T01:15:56Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

+            org.apache.spark.sql.catalyst.util.DateTimeUtils.microsToEpochDays($c, $zid);"""
+      case _ =>
+        (c, evPrim, evNull) => code"$evNull = true;"
+    }
  }


Just put the gen'd code for other reviewers;

/* 051 */ boolean project_isNull_0 = columnartorow_isNull_0; /* 052 */ int project_value_0 = -1; /* 053 */ if (!columnartorow_isNull_0) { /* 054 */ scala.Option<Integer> project_intOpt_0 = /* 055 */ org.apache.spark.sql.catalyst.util.DateTimeUtils.stringToDate(columnartorow_value_0, ((java.time.ZoneId) references[2] /* zoneId */)); /* 056 */ if (project_intOpt_0.isDefined()) { /* 057 */ project_value_0 = ((Integer) project_intOpt_0.get()).intValue(); /* 058 */ } else { /* 059 */ project_isNull_0 = true; /* 060 */ } /* 061 */ }

maropu · 2019-09-08T01:22:26Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

+        val zid = getZoneId()
+        (c, evPrim, evNull) =>
+          code"""
+          scala.Option<Integer> $intOpt =


btw, (this is not related to this pr though), scala.Option<Integer> is ok for java compilers/jvm? @rednaxelafx I remember that jdk compilers cannot compile this statement: #21770 (comment) because it seems the compilers erase the returned type from scala.Option<Integer> to scala.Option<Object>. I'm not sure why janino accepts this though....

SparkQA · 2019-09-10T09:13:41Z

Test build #110409 has finished for PR 25708 at commit 106524b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-09-10T11:39:04Z

retest this please

SparkQA · 2019-09-10T15:19:25Z

Test build #110417 has finished for PR 25708 at commit 106524b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-12T07:15:03Z

@maropu @dongjoon-hyun Here is the same question as in #25716 (comment)

…ecial-values # Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala # sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala # sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala

SparkQA · 2019-09-18T20:43:57Z

Test build #110922 has finished for PR 25708 at commit ff92531.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-19T05:44:23Z

I have rebased this on the master with merged #25716 . @HyukjinKwon Could you take a look at this, please.

MaxGekk · 2019-09-20T18:25:10Z

@dongjoon-hyun @maropu @HyukjinKwon Could take a look at this PR which is similar to already merged changes for the timestamp type.

maropu

Looks nice cc: @HyukjinKwon @dongjoon-hyun

HyukjinKwon · 2019-09-22T08:29:50Z

I am merging this just to unblock and easier to work further but please make sure to have a flag. Seems like that's pointed out by multiple committers such as @maropu and @cloud-fan.

HyukjinKwon · 2019-09-22T08:30:08Z

Otherwise, I will revert this and #25716

Merged to master.

maropu · 2019-09-22T09:44:01Z

We already have a jira ticket for linking this feature to the flag?

MaxGekk · 2019-09-22T17:07:28Z

This PR #25834 hides the feature under the SQL config spark.sql.dialect = "PostgreSQL"

maropu · 2019-09-22T23:04:03Z

Oh, I see. Thanks!

… SQL migration guide ### What changes were proposed in this pull request? Updated the SQL migration guide regarding to recently supported special date and timestamp values, see #25716 and #25708. Closes #25834 ### Why are the changes needed? To let users know about new feature in Spark 3.0. ### Does this PR introduce any user-facing change? No Closes #25948 from MaxGekk/special-values-migration-guide. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…only ### What changes were proposed in this pull request? In the PR, I propose to support special datetime values introduced by #25708 and by #25716 only in typed literals, and don't recognize them in parsing strings to dates/timestamps. The following string values are supported only in typed timestamp literals: - `epoch [zoneId]` - `1970-01-01 00:00:00+00 (Unix system time zero)` - `today [zoneId]` - midnight today. - `yesterday [zoneId]` - midnight yesterday - `tomorrow [zoneId]` - midnight tomorrow - `now` - current query start time. For example: ```sql spark-sql> SELECT timestamp 'tomorrow'; 2019-09-07 00:00:00 ``` Similarly, the following special date values are supported only in typed date literals: - `epoch [zoneId]` - `1970-01-01` - `today [zoneId]` - the current date in the time zone specified by `spark.sql.session.timeZone`. - `yesterday [zoneId]` - the current date -1 - `tomorrow [zoneId]` - the current date + 1 - `now` - the date of running the current query. It has the same notion as `today`. For example: ```sql spark-sql> SELECT date 'tomorrow' - date 'yesterday'; 2 ``` ### Why are the changes needed? In the current implementation, Spark supports the special date/timestamp value in any input strings casted to dates/timestamps that leads to the following problems: - If executors have different system time, the result is inconsistent, and random. Column values depend on where the conversions were performed. - The special values play the role of distributed non-deterministic functions though users might think of the values as constants. ### Does this PR introduce _any_ user-facing change? Yes but the probability should be small. ### How was this patch tested? By running existing test suites: ``` $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z interval.sql" $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z date.sql" $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z timestamp.sql" $ build/sbt "test:testOnly *DateTimeUtilsSuite" ``` Closes #32714 from MaxGekk/remove-datetime-special-values. Lead-authored-by: Max Gekk <max.gekk@gmail.com> Co-authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

MaxGekk added 7 commits September 5, 2019 22:21

Support special date values

c5034a8

Use special values in stringToDate

695a6c6

Tests for stringToDate

ebc5b02

Use map with UTF8String keys

2663542

Bug fix: pass zoneId in generated code

edda1f2

Uncomment tests in date.sql

8d023ec

Change ticket for the special values: infinity and -infinity

775c7ec

MaxGekk commented Sep 6, 2019

View reviewed changes

Bug fix: pass zoneId to Cast

be787f4

MaxGekk mentioned this pull request Sep 6, 2019

[WIP][SQL] Resolve the time zone before inline tables #25711

Closed

MaxGekk added 7 commits September 6, 2019 21:57

Bug fix: use passed zoneId parameter

f356b72

Remove an unused import

864e456

Use outstandingZoneIds in the test

e237b27

Merge remote-tracking branch 'remotes/origin/master' into datetime-sp…

0055a8a

…ecial-values # Conflicts: # sql/core/src/test/resources/sql-tests/results/pgSQL/date.sql.out

Re-get date.sql.out

4fb8834

Create zoneId when it is needed

9a2349b

Bug fix: require time zone in the string to date conversion

6da7905

dongjoon-hyun added the SQL label Sep 6, 2019

Use regex

d57e11e

MaxGekk commented Sep 7, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala Show resolved Hide resolved

maropu reviewed Sep 8, 2019

View reviewed changes

maropu mentioned this pull request Sep 10, 2019

[SPARK-29012][SQL] Support special timestamp values #25716

Closed

MaxGekk added 2 commits September 18, 2019 20:35

Remove an unused import

ff92531

MaxGekk mentioned this pull request Sep 18, 2019

[WIP][SPARK-29155][SQL] Support special date/timestamp values in the PostgreSQL dialect only #25834

Closed

maropu approved these changes Sep 20, 2019

View reviewed changes

HyukjinKwon closed this in 051e691 Sep 22, 2019

MaxGekk mentioned this pull request Sep 27, 2019

[SPARK-29275][SQL][DOC] Describe special date/timestamp values in the SQL migration guide #25948

Closed

MaxGekk deleted the datetime-special-values branch October 5, 2019 19:18

MaxGekk mentioned this pull request May 31, 2021

[SPARK-35581][SQL] Support special datetime values in typed literals only #32714

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-28141][SQL] Support special date values #25708

[SPARK-28141][SQL] Support special date values #25708

MaxGekk commented Sep 6, 2019 •

edited

SparkQA commented Sep 6, 2019

MaxGekk Sep 6, 2019 •

edited

SparkQA commented Sep 6, 2019

SparkQA commented Sep 6, 2019

SparkQA commented Sep 6, 2019

MaxGekk commented Sep 7, 2019

SparkQA commented Sep 7, 2019

maropu commented Sep 8, 2019

maropu commented Sep 8, 2019

maropu Sep 8, 2019

maropu Sep 8, 2019

SparkQA commented Sep 10, 2019

maropu commented Sep 10, 2019

SparkQA commented Sep 10, 2019

MaxGekk commented Sep 12, 2019

SparkQA commented Sep 18, 2019

MaxGekk commented Sep 19, 2019

MaxGekk commented Sep 20, 2019

maropu left a comment

HyukjinKwon commented Sep 22, 2019

HyukjinKwon commented Sep 22, 2019

maropu commented Sep 22, 2019

MaxGekk commented Sep 22, 2019

maropu commented Sep 22, 2019

[SPARK-28141][SQL] Support special date values #25708

[SPARK-28141][SQL] Support special date values #25708

Conversation

MaxGekk commented Sep 6, 2019 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Sep 6, 2019

MaxGekk Sep 6, 2019 • edited

Choose a reason for hiding this comment

SparkQA commented Sep 6, 2019

SparkQA commented Sep 6, 2019

SparkQA commented Sep 6, 2019

MaxGekk commented Sep 7, 2019

SparkQA commented Sep 7, 2019

maropu commented Sep 8, 2019

maropu commented Sep 8, 2019

maropu Sep 8, 2019

Choose a reason for hiding this comment

maropu Sep 8, 2019

Choose a reason for hiding this comment

SparkQA commented Sep 10, 2019

maropu commented Sep 10, 2019

SparkQA commented Sep 10, 2019

MaxGekk commented Sep 12, 2019

SparkQA commented Sep 18, 2019

MaxGekk commented Sep 19, 2019

MaxGekk commented Sep 20, 2019

maropu left a comment

Choose a reason for hiding this comment

HyukjinKwon commented Sep 22, 2019

HyukjinKwon commented Sep 22, 2019

maropu commented Sep 22, 2019

MaxGekk commented Sep 22, 2019

maropu commented Sep 22, 2019

MaxGekk commented Sep 6, 2019 •

edited

MaxGekk Sep 6, 2019 •

edited