[SPARK-27898][SQL] Support 4 date operators(date + integer, integer + date, date - integer and date - date)

[wangyum]

What changes were proposed in this pull request?

This pr add support 4 PostgreSQL's date operators(date + integer, integer + date, date - integer and date - date):

Operator	Example	Result
+	date '2001-09-28' + 7	date '2001-10-05'
+	7 + date '2001-09-28'	date '2001-10-05'
-	date '2001-10-01' - 7	date '2001-09-24'
-	date '2001-10-01' - date '2001-09-28'	integer '3' (days)

Most databases support date - date operation, where PostgreSQL, Vertica, Teradata, Oracle and DB2 returns Integer type, Hive and Presto returns Interval type, MySQL returns unexpected value, and SQL Server does not support date - date operation.

PostgreSQL:

postgres=# select substr(version(), 0, 16), date '2001-09-28' + 7, 7 + date '2001-09-28', date '2001-10-01' - 7, date '2001-10-01' - date '2001-09-28';
     substr      |  ?column?  |  ?column?  |  ?column?  | ?column?
-----------------+------------+------------+------------+----------
 PostgreSQL 11.3 | 2001-10-05 | 2001-10-05 | 2001-09-24 |        3
(1 row)

Vertica:

dbadmin=> select version(), date '2001-09-28' + 7, 7 + date '2001-09-28', date '2001-10-01' - 7, date '2001-10-01' - date '2001-09-28';
              version               |  ?column?  |  ?column?  |  ?column?  | ?column?
------------------------------------+------------+------------+------------+----------
 Vertica Analytic Database v9.1.1-0 | 2001-10-05 | 2001-10-05 | 2001-09-24 |        3
(1 row)

Teradata:

Oracle:

DB2:

Hive:

hive> select version(),  date '2001-10-01' - date '2001-09-28';
OK
3.1.1 rf4e0529634b6231a0072295da48af466cf2f10b7	3 00:00:00.000000000
Time taken: 2.038 seconds, Fetched: 1 row(s)

Presto:

presto> select  date '2001-10-01' - date '2001-09-28';
     _col0
----------------
 3 00:00:00.000
(1 row)

MySQL:

mysql> SELECT version(), date '2001-10-01' - date '2001-09-28';
+-----------+---------------------------------------+
| version() | date '2001-10-01' - date '2001-09-28' |
+-----------+---------------------------------------+
| 5.7.26    |                                    73 |
+-----------+---------------------------------------+
1 row in set (0.00 sec)

More details:
https://www.postgresql.org/docs/12/functions-datetime.html

How was this patch tested?

unit tests

[MaxGekk]

According to SQL standard, date '2001-10-01' - date '2001-09-28' must return the Interval type instead of IntegerType as DateDiff does.

[SparkQA]

Test build #106016 has finished for PR 24755 at commit c10533d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

@MaxGekk I prefer to return the Integer type. PostgreSQL, Vertica, Teradata, Oracle, and DB2 all return Integer types, and only Hive and Presto return Interval types.

[gatorsmile]

What is the current behavior of Spark SQL?

[wangyum]

Our DateDiff returns integer:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

Line 1582 in ed280c2

override def dataType: DataType = IntegerType

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #107272 has finished for PR 24755 at commit c10533d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Although I have been preferring Hive/PrestoSQL compatibility over the others, but I'm +1 for this PR since this is a new feature and we are working on PostgreSQL compatibility improvement in 3.0. We can add an additional configuration later if we need Hive/PrestoSQL compatibility, too.

+1, LGTM. Merged to master. Let's move forward.

Thank you so much, @wangyum , @gatorsmile , @MaxGekk .