Please sign in to comment.
[SPARK-27638][SQL] Cast string to date/timestamp in binary comparison…
…s with dates/timestamps ## What changes were proposed in this pull request? The below example works with both Mysql and Hive, however not with spark. ``` mysql> select * from date_test where date_col >= '2000-1-1'; +------------+ | date_col | +------------+ | 2000-01-01 | +------------+ ``` The reason is that Spark casts both sides to String type during date and string comparison for partial date support. Please find more details in https://issues.apache.org/jira/browse/SPARK-8420. Based on some tests, the behavior of Date and String comparison in Hive and Mysql: Hive: Cast to Date, partial date is not supported Mysql: Cast to Date, certain "partial date" is supported by defining certain date string parse rules. Check out str_to_datetime in https://github.com/mysql/mysql-server/blob/5.5/sql-common/my_time.c As below date patterns have been supported, the PR is to cast string to date when comparing string and date: ``` `yyyy` `yyyy-[m]m` `yyyy-[m]m-[d]d` `yyyy-[m]m-[d]d ` `yyyy-[m]m-[d]d *` `yyyy-[m]m-[d]dT* ``` ## How was this patch tested? UT has been added Closes #24567 from pengbo/SPARK-27638. Authored-by: mingbo.pb <firstname.lastname@example.org> Signed-off-by: Wenchen Fan <email@example.com>
- Loading branch information...
Showing with 186 additions and 96 deletions.
- +2 −0 docs/sql-migration-guide-upgrade.md
- +8 −7 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
- +9 −0 sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- +6 −6 sql/core/src/test/resources/sql-tests/results/predicate-functions.sql.out
- +40 −40 sql/core/src/test/resources/sql-tests/results/typeCoercion/native/binaryComparison.sql.out
- +42 −42 sql/core/src/test/resources/sql-tests/results/typeCoercion/native/promoteStrings.sql.out
- +1 −1 sql/core/src/test/resources/tpcds/q32.sql
- +78 −0 sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
Oops, something went wrong.