[SPARK-29364][SQL] Return an interval from date subtract according to SQL standard #26112

MaxGekk · 2019-10-14T16:05:40Z

What changes were proposed in this pull request?

Proposed new expression SubtractDates which is used in date1 - date2. It has the INTERVAL type, and returns the interval from date1 (inclusive) and date2 (exclusive). For example:

> select date'tomorrow' - date'yesterday';
interval 2 days

Closes #26034

Why are the changes needed?

To conform the SQL standard which states the result type of date operand 1 - date operand 2 must be the interval type. See 4.5.3 Operations involving datetimes and intervals.
Improve Spark SQL UX and allow mixing date and timestamp in subtractions. For example: select timestamp'now' + (date'2019-10-01' - date'2019-09-15')

Does this PR introduce any user-facing change?

Before the query below returns number of days:

spark-sql> select date'2019-10-05' - date'2018-09-01';
399

After it returns an interval:

spark-sql> select date'2019-10-05' - date'2018-09-01';
interval 1 years 1 months 4 days

How was this patch tested?

by new tests in DateExpressionsSuite and TypeCoercionSuite.
by existing tests in date.sql

MaxGekk · 2019-10-14T16:08:09Z

@cloud-fan Could you take a look at this PR, please.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala

docs/sql-migration-guide.md

SparkQA · 2019-10-14T20:19:06Z

Test build #112041 has finished for PR 26112 at commit 4a8173b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-14T21:05:25Z

Test build #112058 has finished for PR 26112 at commit 199adff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-10-14T21:18:19Z

jenkins, retest this, please

SparkQA · 2019-10-15T00:50:29Z

Test build #112067 has finished for PR 26112 at commit 199adff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-15T02:28:03Z

LGTM if tests pass

SparkQA · 2019-10-15T07:05:01Z

Test build #112085 has finished for PR 26112 at commit 8b49c9b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-10-15T07:15:32Z

jenkins, retest this, please

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala

SparkQA · 2019-10-15T10:56:32Z

Test build #112094 has finished for PR 26112 at commit 8b49c9b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-10-15T14:58:23Z

docs/sql-migration-guide.md

@@ -217,6 +217,8 @@ license: |

  - Since Spark 3.0, the `size` function returns `NULL` for the `NULL` input. In Spark version 2.4 and earlier, this function gives `-1` for the same input. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.sizeOfNull` to `true`.

+  - In Spark version 2.4 and earlier, dates subtraction `date1` - `date2` gives the number of days from `date1` to `date2`. Since Spark 3.0, the expression has the `INTERVAL` type and returns an interval between two dates. To get the number of days, you can set `spark.sql.legacy.datesSubtraction.enabled` to `true` or use the `datediff` function.


We added the date1 - date2 operator since SPARK-27898 (Spark 3.0). Not Spark 2.4 or earlier.

good news! then we can remove the config and migration guide :)

This reverts commit 199adff.

This reverts commit 27de6b8.

This reverts commit e82f027.

SparkQA · 2019-10-15T22:29:47Z

Test build #112127 has finished for PR 26112 at commit f232b64.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-10-16T00:47:33Z

sql/core/src/test/resources/sql-tests/results/postgreSQL/date.sql.out

+interval 3 months 2 days
+interval 38 years 3 months 1 weeks
+interval 39 years 3 months 1 weeks 1 days
+interval 40 years 3 months 1 weeks 2 days



These results are inconsistent with PostgreSQL:
https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/expected/date.out#L845-L882

cloud-fan · 2019-10-16T03:21:50Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala

@@ -849,12 +850,13 @@ object TypeCoercion {
      case Add(l @ DateType(), r @ IntegerType()) => DateAdd(l, r)
      case Add(l @ IntegerType(), r @ DateType()) => DateAdd(r, l)
      case Subtract(l @ DateType(), r @ IntegerType()) => DateSub(l, r)
-      case Subtract(l @ DateType(), r @ DateType()) => DateDiff(l, r)
-      case Subtract(l @ TimestampType(), r @ TimestampType()) => TimestampDiff(l, r)
+      case Subtract(l @ DateType(), r @ DateType()) => SubtractDates(l, r)


seems like we should use DateDiff if the dialect is pgsql.

SparkQA · 2019-10-16T07:05:02Z

Test build #112139 has finished for PR 26112 at commit c6ec211.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-10-16T07:10:43Z

retest this please

SparkQA · 2019-10-16T10:52:27Z

Test build #112146 has finished for PR 26112 at commit c6ec211.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

… SQL standard ### What changes were proposed in this pull request? Proposed new expression `SubtractDates` which is used in `date1` - `date2`. It has the `INTERVAL` type, and returns the interval from `date1` (inclusive) and `date2` (exclusive). For example: ```sql > select date'tomorrow' - date'yesterday'; interval 2 days ``` Closes #26034 ### Why are the changes needed? - To conform the SQL standard which states the result type of `date operand 1` - `date operand 2` must be the interval type. See [4.5.3 Operations involving datetimes and intervals](http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt). - Improve Spark SQL UX and allow mixing date and timestamp in subtractions. For example: `select timestamp'now' + (date'2019-10-01' - date'2019-09-15')` ### Does this PR introduce any user-facing change? Before the query below returns number of days: ```sql spark-sql> select date'2019-10-05' - date'2018-09-01'; 399 ``` After it returns an interval: ```sql spark-sql> select date'2019-10-05' - date'2018-09-01'; interval 1 years 1 months 4 days ``` ### How was this patch tested? - by new tests in `DateExpressionsSuite` and `TypeCoercionSuite`. - by existing tests in `date.sql` Closes #26112 from MaxGekk/date-subtract. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Yuming Wang <wgyumg@gmail.com> (cherry picked from commit d11cbf2) Signed-off-by: Yuming Wang <wgyumg@gmail.com>

wangyum · 2019-10-16T13:33:19Z

Merged to master and branch-3.0-preview.

… SQL standard Proposed new expression `SubtractDates` which is used in `date1` - `date2`. It has the `INTERVAL` type, and returns the interval from `date1` (inclusive) and `date2` (exclusive). For example: ```sql > select date'tomorrow' - date'yesterday'; interval 2 days ``` Closes apache#26034 - To conform the SQL standard which states the result type of `date operand 1` - `date operand 2` must be the interval type. See [4.5.3 Operations involving datetimes and intervals](http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt). - Improve Spark SQL UX and allow mixing date and timestamp in subtractions. For example: `select timestamp'now' + (date'2019-10-01' - date'2019-09-15')` Before the query below returns number of days: ```sql spark-sql> select date'2019-10-05' - date'2018-09-01'; 399 ``` After it returns an interval: ```sql spark-sql> select date'2019-10-05' - date'2018-09-01'; interval 1 years 1 months 4 days ``` - by new tests in `DateExpressionsSuite` and `TypeCoercionSuite`. - by existing tests in `date.sql` Closes apache#26112 from MaxGekk/date-subtract. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Yuming Wang <wgyumg@gmail.com>

### What changes were proposed in this pull request? Reprocess all PostgreSQL dialect related PRs, listing in order: - #25158: PostgreSQL integral division support [revert] - #25170: UT changes for the integral division support [revert] - #25458: Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. [revert] - #25697: Combine below 2 feature tags into "spark.sql.dialect" [revert] - #26112: Date substraction support [keep the ANSI-compliant part] - #26444: Rename config "spark.sql.ansi.enabled" to "spark.sql.dialect.spark.ansi.enabled" [revert] - #26463: Cast to boolean support for PostgreSQL dialect [revert] - #26584: Make the behavior of Postgre dialect independent of ansi mode config [keep the ANSI-compliant part] ### Why are the changes needed? As the discussion in http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-PostgreSQL-dialect-td28417.html, we need to remove PostgreSQL dialect form code base for several reasons: 1. The current approach makes the codebase complicated and hard to maintain. 2. Fully migrating PostgreSQL workloads to Spark SQL is not our focus for now. ### Does this PR introduce any user-facing change? Yes, the config `spark.sql.dialect` will be removed. ### How was this patch tested? Existing UT. Closes #26763 from xuanyuanking/SPARK-30125. Lead-authored-by: Yuanjian Li <xyliyuanjian@gmail.com> Co-authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

MaxGekk added 6 commits October 14, 2019 17:29

Add SubtractDates

18bc09b

Add tests for SubtractDates

141d3da

Use SubtractDates for dates subtract

dc37bc7

Update the SQL migration guide

e82f027

Regen date.sql.out

e74f81b

Regen datetime.sql.out

4a8173b

MaxGekk commented Oct 14, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Oct 14, 2019

View reviewed changes

docs/sql-migration-guide.md Outdated Show resolved Hide resolved

dongjoon-hyun added the SQL label Oct 14, 2019

MaxGekk added 2 commits October 14, 2019 22:49

Add config spark.sql.legacy.datesSubtraction.enabled

27de6b8

TimestampDiff -> SubtractTimestamps

fbd2723

Update config description regarding datediff

199adff

Regen datetime.sql.out

8b49c9b

wangyum reviewed Oct 15, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala Outdated Show resolved Hide resolved

wangyum reviewed Oct 15, 2019

View reviewed changes

MaxGekk added 3 commits October 15, 2019 21:27

Revert "Update config description regarding datediff"

85f17f7

This reverts commit 199adff.

Revert "Add config spark.sql.legacy.datesSubtraction.enabled"

228673d

This reverts commit 27de6b8.

Revert "Update the SQL migration guide"

f232b64

This reverts commit e82f027.

wangyum reviewed Oct 16, 2019

View reviewed changes

cloud-fan reviewed Oct 16, 2019

View reviewed changes

Use DateDiff in the PostgreSQL dialect

c6ec211

cloud-fan approved these changes Oct 16, 2019

View reviewed changes

wangyum approved these changes Oct 16, 2019

View reviewed changes

wangyum closed this in d11cbf2 Oct 16, 2019

MaxGekk mentioned this pull request Oct 17, 2019

[SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds #26134

Closed

xuanyuanking mentioned this pull request Dec 4, 2019

[SPARK-30125][SQL] Remove PostgreSQL dialect #26763

Closed

MaxGekk deleted the date-subtract branch June 5, 2020 19:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-29364][SQL] Return an interval from date subtract according to SQL standard #26112

[SPARK-29364][SQL] Return an interval from date subtract according to SQL standard #26112

MaxGekk commented Oct 14, 2019 •

edited

MaxGekk commented Oct 14, 2019

SparkQA commented Oct 14, 2019

SparkQA commented Oct 14, 2019

MaxGekk commented Oct 14, 2019

SparkQA commented Oct 15, 2019

cloud-fan commented Oct 15, 2019

SparkQA commented Oct 15, 2019

MaxGekk commented Oct 15, 2019

SparkQA commented Oct 15, 2019

wangyum Oct 15, 2019

cloud-fan Oct 15, 2019

SparkQA commented Oct 15, 2019

wangyum Oct 16, 2019

cloud-fan Oct 16, 2019

SparkQA commented Oct 16, 2019

wangyum commented Oct 16, 2019

SparkQA commented Oct 16, 2019

wangyum commented Oct 16, 2019

		@@ -217,6 +217,8 @@ license: \|

		- Since Spark 3.0, the `size` function returns `NULL` for the `NULL` input. In Spark version 2.4 and earlier, this function gives `-1` for the same input. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.sizeOfNull` to `true`.

		- In Spark version 2.4 and earlier, dates subtraction `date1` - `date2` gives the number of days from `date1` to `date2`. Since Spark 3.0, the expression has the `INTERVAL` type and returns an interval between two dates. To get the number of days, you can set `spark.sql.legacy.datesSubtraction.enabled` to `true` or use the `datediff` function.

[SPARK-29364][SQL] Return an interval from date subtract according to SQL standard #26112

[SPARK-29364][SQL] Return an interval from date subtract according to SQL standard #26112

Conversation

MaxGekk commented Oct 14, 2019 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

MaxGekk commented Oct 14, 2019

SparkQA commented Oct 14, 2019

SparkQA commented Oct 14, 2019

MaxGekk commented Oct 14, 2019

SparkQA commented Oct 15, 2019

cloud-fan commented Oct 15, 2019

SparkQA commented Oct 15, 2019

MaxGekk commented Oct 15, 2019

SparkQA commented Oct 15, 2019

wangyum Oct 15, 2019

Choose a reason for hiding this comment

cloud-fan Oct 15, 2019

Choose a reason for hiding this comment

SparkQA commented Oct 15, 2019

wangyum Oct 16, 2019

Choose a reason for hiding this comment

cloud-fan Oct 16, 2019

Choose a reason for hiding this comment

SparkQA commented Oct 16, 2019

wangyum commented Oct 16, 2019

SparkQA commented Oct 16, 2019

wangyum commented Oct 16, 2019

MaxGekk commented Oct 14, 2019 •

edited