New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-15334][SQL] HiveClient facade not compatible with Hive 0.12 #13127
Conversation
Test build #58621 has finished for PR 13127 at commit
|
inputFormat = Some(classOf[org.apache.hadoop.mapred.TextInputFormat].getName), | ||
outputFormat = Some( | ||
classOf[org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat[_, _]].getName), | ||
serde = Some(classOf[org.apache.hadoop.hive.serde2.`lazy`.LazySimpleSerDe].getName()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about importing these Hive classes above instead of using the fully qualified names here?
I have finished my review. Thank you for working on this! |
Test build #58739 has finished for PR 13127 at commit
|
Test build #58747 has finished for PR 13127 at commit
|
Test build #58748 has finished for PR 13127 at commit
|
LGTM, merging to master and branch-2.0. |
## What changes were proposed in this pull request? HiveClient facade is not compatible with Hive 0.12. This PR Fixes the following compatibility issues: 1. `org.apache.spark.sql.hive.client.HiveClientImpl` use `AddPartitionDesc(db, table, ignoreIfExists)` to create partitions, however, Hive 0.12 doesn't have this constructor for `AddPartitionDesc`. 2. `HiveClientImpl` uses `PartitionDropOptions` when dropping partition, however, class `PartitionDropOptions` doesn't exist in Hive 0.12. 3. Hive 0.12 doesn't support adding permanent functions. It is not valid to call `org.apache.hadoop.hive.ql.metadata.Hive.createFunction`, `org.apache.hadoop.hive.ql.metadata.Hive.alterFunction`, and `org.apache.hadoop.hive.ql.metadata.Hive.alterFunction` 4. `org.apache.spark.sql.hive.client.VersionsSuite` doesn't have enough test coverage for different hive versions 0.12, 0.13, 0.14, 1.0.0, 1.1.0, 1.2.0. ## How was this patch tested? Unit test. Author: Sean Zhong <seanzhong@databricks.com> Closes #13127 from clockfly/versionSuite. (cherry picked from commit 6e02aec) Signed-off-by: Cheng Lian <lian@databricks.com>
## What changes were proposed in this pull request? This PR fixes a Scala 2.10 compilation failure introduced in PR #13127. ## How was this patch tested? Jenkins build. Author: Cheng Lian <lian@databricks.com> Closes #13166 from liancheng/hotfix-for-scala-2.10. (cherry picked from commit c4a45fd) Signed-off-by: Cheng Lian <lian@databricks.com>
What changes were proposed in this pull request?
HiveClient facade is not compatible with Hive 0.12.
This PR Fixes the following compatibility issues:
org.apache.spark.sql.hive.client.HiveClientImpl
useAddPartitionDesc(db, table, ignoreIfExists)
to create partitions, however, Hive 0.12 doesn't have this constructor forAddPartitionDesc
.HiveClientImpl
usesPartitionDropOptions
when dropping partition, however, classPartitionDropOptions
doesn't exist in Hive 0.12.org.apache.hadoop.hive.ql.metadata.Hive.createFunction
,org.apache.hadoop.hive.ql.metadata.Hive.alterFunction
, andorg.apache.hadoop.hive.ql.metadata.Hive.alterFunction
org.apache.spark.sql.hive.client.VersionsSuite
doesn't have enough test coverage for different hive versions 0.12, 0.13, 0.14, 1.0.0, 1.1.0, 1.2.0.How was this patch tested?
Unit test.