[SPARK-37069][SQL] Properly fallback when Hive.getWithoutRegisterFns is not available #34360
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Kubernetes integration test starting |
|
Test build #144516 has finished for PR 34360 at commit
|
|
Kubernetes integration test status failure |
|
retest this please |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #144518 has finished for PR 34360 at commit
|
HyukjinKwon
approved these changes
Oct 22, 2021
HyukjinKwon
reviewed
Oct 22, 2021
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
Outdated
Show resolved
Hide resolved
wangyum
approved these changes
Oct 22, 2021
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #144523 has finished for PR 34360 at commit
|
|
Merged to master and branch-3.2. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Oct 22, 2021
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes #34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
3 tasks
|
+1, LGTM. Thank you, @sunchao and all. |
sunchao
added a commit
to sunchao/spark
that referenced
this pull request
Dec 8, 2021
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes apache#34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
catalinii
pushed a commit
to lyft/spark
that referenced
this pull request
Feb 22, 2022
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes apache#34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
catalinii
pushed a commit
to lyft/spark
that referenced
this pull request
Mar 4, 2022
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes apache#34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Properly fallback to
Hive.getwhenHive.getWithoutRegisterFnsis unavailable to a Hive version.Does this PR introduce any user-facing change?
In SPARK-35321 we switched to use the new method
Hive.getWithoutRegisterFnsintroduced by HIVE-21563. The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to howHiveVersionis initialized inIsolatedClientLoader, if users setspark.sql.hive.metastore.versionto2.3.8, Spark will still convert it to "2.3.9" and thus will subsequently fail withNoSuchMethodError.This fixes it by always fallback on
NoSuchMethodError. By doing this we are also able to support other Hive versions withHive.getWithoutRegisterFnsimplemented.How was this patch tested?
I manually tested via launching a Spark session with custom Hive version:
And then tried this command:
The command is failing before this PR, but working afterwards.