[SQL] [SPARK-8064] Fixes Parquet dependency issue for Hive 1.2.1 upgrade #7867
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sses into a hive package
…ts pass instead of timing out: undetected regression?
…naryCLIService on port"
…ve dependency of hive-exec
…ependency of hive-exec
…essages to say "w/Hive 1.2.1" in console output
…in the raised exception; this ensures that the stdout/stderr is included in jenkins reports, so it becomes possible to diagnose the cause.
…e rest of the work; can be backported to earlier versions
…TOK_UNIONDISTINCT
…ure that jersey server is on classpath on hadoop versions < 2.6
…till no obvious reason why jersey server code in spark-assembly isn't being picked up -it hasn't been shaded
…nd/or error string patterns seen in log.
…ight to map of serde options
…nings. It's all autogenerated code, so still ugly.
-things that aren't needed are excluded -sql/hive built with ivy is in sync with the maven reconciliation policy, rather than latest-first
…ed references in the hive- packages; this keeps mvn and ivy resolution compatible, as the reconciliation policy is "by hand"
… that automatically.
… This one has the protobuf-shaded hive-exec jar
…inal (?) hive-exec spark
|
Test build #39426 has finished for PR 7867 at commit
|
|
Closing this since it's merged into PR #7191. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a fork of PR #7191 authored by @steveloughran. This PR fixes the Parquet dependency issue, which causes exception while creating metastore Parquet tables due to missing Hive Parquet SerDe classes.
The missing classes originally come from
com.twitter:parquet-hadoop-bundle:1.6.0, which should be a transitive dependency oforg.apache.hive:hive-exec:1.2.1.spark, but somehow gets evicted in SBT build. This PR adds it back as a dependency ofspark-hive.Please refer to [this failed Jenkins build] for details of related test failures.