[SPARK-43357][SQL] Filter date type quote date #41035
Conversation
stijndehaes
force-pushed
the
bugfix/hive-date-partition
branch
from
8f78d3e
to
594f3c8
Compare
stijndehaes
force-pushed
the
bugfix/hive-date-partition
branch
from
594f3c8
to
6a18b18
Compare
|
This change is not backward compatible, I'm not sure if this change is necessary, if necessary, also need parameters to control whether to open. |
|
@stijndehaes Do you have the error message if do not quote date? |
|
There is more info in the issue on JIRA: https://issues.apache.org/jira/browse/SPARK-43357 So basically when integrating we glue we are getting issues: Now this issue might be on the side of glue, but glue is supposed to be hive compatible. So I think this should also work with a hive catalog. However I do not have access to one, and would net to se one up. |
There was a problem hiding this comment.
@cloud-fan Hive MetaStore also support quoted string date. I'm not sure if it should be fixed on Spark side or Glue side： https://github.com/apache/hive/blob/5106bf1c8671740099fca8e1a7d4b37afe97137f/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L776-L786
| @@ -66,17 +66,17 @@ class FiltersSuite extends SparkFunSuite with PlanTest { | |||
|
|
|||
| filterTest("date filter", | |||
| (a("datecol", DateType) === Literal(Date.valueOf("2019-01-01"))) :: Nil, | |||
| "datecol = 2019-01-01") | |||
There was a problem hiding this comment.
I'm surprised hive supports this SQL syntax... datecol = 2019-01-01
There was a problem hiding this comment.
It seems it is only used for partition filtering in metastore:
https://github.com/apache/hive/blob/e8076ef41de842b9adceea4f854a92d9d1d1388b/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g#L48
|
@wangyum does all hive versions support it? |
Hive 0.13.0 and later support it: https://issues.apache.org/jira/browse/HIVE-5679. |
|
thanks, merging to master! |
### What changes were proposed in this pull request? In this PR the date value is quoted before being sent to the hive metastore ### Why are the changes needed? Glue for example expects the dates to be quoted, without quotes we get errors. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Changed the result of unit tests, also tested by running the fix against glue Closes apache#41035 from stijndehaes/bugfix/hive-date-partition. Authored-by: stijndehaes <stijndehaes@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
In this PR the date value is quoted before being sent to the hive metastore
Why are the changes needed?
Glue for example expects the dates to be quoted, without quotes we get errors.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Changed the result of unit tests, also tested by running the fix against glue