-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-37069][SQL] Properly fallback when Hive.getWithoutRegisterFns is not available #34360
Conversation
Kubernetes integration test starting |
Test build #144516 has finished for PR 34360 at commit
|
Kubernetes integration test status failure |
retest this please |
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #144518 has finished for PR 34360 at commit
|
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
Outdated
Show resolved
Hide resolved
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #144523 has finished for PR 34360 at commit
|
Merged to master and branch-3.2. |
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes #34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
+1, LGTM. Thank you, @sunchao and all. |
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes apache#34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes apache#34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…is not available Properly fallback to `Hive.get` when `Hive.getWithoutRegisterFns` is unavailable to a Hive version. In SPARK-35321 we switched to use the new method `Hive.getWithoutRegisterFns` introduced by [HIVE-21563](https://issues.apache.org/jira/browse/HIVE-21563). The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to how `HiveVersion` is initialized in `IsolatedClientLoader`, if users set `spark.sql.hive.metastore.version` to `2.3.8`, Spark will still convert it to "2.3.9" and thus will subsequently fail with `NoSuchMethodError`. This fixes it by always fallback on `NoSuchMethodError`. By doing this we are also able to support other Hive versions with `Hive.getWithoutRegisterFns` implemented. I manually tested via launching a Spark session with custom Hive version: ``` $SPARK_HOME/bin/spark-shell --conf spark.sql.hive.metastore.version=2.3.8 --conf spark.sql.hive.metastore.jars="/tmp/apache-hive-2.3.8-bin/lib/* ``` And then tried this command: ``` import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .master("local[*]") .enableHiveSupport() .config("spark.sql.hive.metastore.version", "2.3.8") .config("spark.sql.hive.metastore.jars", "/tmp/apache-hive-2.3.8-bin/lib/*") .getOrCreate() spark.sql("show tables").show ``` The command is failing before this PR, but working afterwards. Closes apache#34360 from sunchao/SPARK-37069. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 39a0c22) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
What changes were proposed in this pull request?
Properly fallback to
Hive.get
whenHive.getWithoutRegisterFns
is unavailable to a Hive version.Does this PR introduce any user-facing change?
In SPARK-35321 we switched to use the new method
Hive.getWithoutRegisterFns
introduced by HIVE-21563. The code path is supposed to only active for Hive versions that are >= 2.3.9. However, due to howHiveVersion
is initialized inIsolatedClientLoader
, if users setspark.sql.hive.metastore.version
to2.3.8
, Spark will still convert it to "2.3.9" and thus will subsequently fail withNoSuchMethodError
.This fixes it by always fallback on
NoSuchMethodError
. By doing this we are also able to support other Hive versions withHive.getWithoutRegisterFns
implemented.How was this patch tested?
I manually tested via launching a Spark session with custom Hive version:
And then tried this command:
The command is failing before this PR, but working afterwards.