[SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 #25775

AngersZhuuuu · 2019-09-12T15:56:53Z

What changes were proposed in this pull request?

Hive 2.3 will set a new UDFClassLoader to hiveConf.classLoader when initializing SessionState since HIVE-11878, and

ADDJarCommand will add jars to clientLoader.classLoader.
--jar passed jar will be added to clientLoader.classLoader
jar passed by hive conf hive.aux.jars SPARK-28954 SPARK-28840 will be added to clientLoader.classLoader too

For these reason we cannot load the jars added by ADDJarCommand because of class loader got changed. We reset it to clientLoader.ClassLoader here.

Why are the changes needed?

support for jdk11

Does this PR introduce any user-facing change?

NO

How was this patch tested?

UT

export JAVA_HOME=/usr/lib/jdk-11.0.3
export PATH=$JAVA_HOME/bin:$PATH

build/sbt -Phive-thriftserver -Phadoop-3.2

hive/test-only *HiveSparkSubmitSuite -- -z "SPARK-8368: includes jars passed in through --jars"
hive-thriftserver/test-only *HiveThriftBinaryServerSuite -- -z "test add jar"

This reverts commit c3de557.

dongjoon-hyun · 2019-09-12T17:09:31Z

ok to test.

SparkQA · 2019-09-12T17:53:20Z

Test build #110523 has finished for PR 25775 at commit edaf402.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit edaf402.

dongjoon-hyun · 2019-09-12T18:21:19Z

Retest this please.

AngersZhuuuu · 2019-09-12T18:34:22Z

Retest this please.

ThriftServer MetaData Operation and CLI have some rely on SessionState. Some code rely on hive code too much.

SparkQA · 2019-09-12T19:35:24Z

Test build #110529 has finished for PR 25775 at commit 098a432.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-12T19:39:30Z

Test build #110532 has finished for PR 25775 at commit 098a432.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-13T00:55:04Z

Test build #110548 has finished for PR 25775 at commit 07dd32e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-13T02:25:02Z

Test build #110551 has finished for PR 25775 at commit 35f0a01.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-13T05:32:09Z

Test build #110555 has finished for PR 25775 at commit fb39a53.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2019-09-13T05:34:58Z

@dongjoon-hyun
Hi , I have change it to jdk8, please trigger building again.
Then I will change title to hadoop2.7.

dongjoon-hyun · 2019-09-13T06:17:13Z

Retest this please.

SparkQA · 2019-09-13T07:05:01Z

Test build #110561 has finished for PR 25775 at commit fb39a53.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-14T07:28:02Z

Test build #110587 has finished for PR 25775 at commit a510a23.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-14T09:32:55Z

Test build #110588 has finished for PR 25775 at commit bc73663.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala

SparkQA · 2019-09-16T20:01:06Z

Test build #110640 has finished for PR 25775 at commit 35c961f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

wangyum · 2019-09-17T07:54:48Z

This PR tested by HiveThriftBinaryServerSuite#test add jar?

SparkQA · 2019-09-17T12:13:12Z

Test build #110771 has finished for PR 25775 at commit f30d001.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-17T21:44:00Z

Test build #110805 has finished for PR 25775 at commit 4db3b54.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-25T05:56:52Z

@AngersZhuuuu Could you add these to How was this patch tested?:

export JAVA_HOME=/usr/lib/jdk-11.0.3
export PATH=$JAVA_HOME/bin:$PATH

build/sbt -Phive-thriftserver -Phadoop-3.2

hive/test-only *HiveSparkSubmitSuite -- -z "SPARK-8368: includes jars passed in through --jars"
hive-thriftserver/test-only *HiveThriftBinaryServerSuite -- -z "test add jar"

srowen · 2019-09-25T09:37:56Z

@wangyum per your comment at #25690 let's merge this first. It looks like it has passed a few times, but I see a number of failures too; since you have been following this more closely, they were due to previous versions of the change? the comment and change sound plausible.

wangyum · 2019-09-25T15:02:07Z

I see. Thank you @srowen

SparkQA · 2019-09-25T17:54:09Z

Test build #111360 has finished for PR 25775 at commit f37335c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-09-25T22:11:10Z

Could we revise the PR title more specifically, @AngersZhuuuu and @wangyum ?

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

wangyum · 2019-09-26T05:52:21Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

+    // got changed. We reset it to clientLoader.ClassLoader here.
+    if (HiveUtils.isHive23) {
+      state.getConf.setClassLoader(clientLoader.classLoader)
+    }


@gatorsmile @srowen @HyukjinKwon @dongjoon-hyun @juliuszsompolski Do you have more comments?

No opinion from me, I know too little context of how these classloaders should be used to review.

srowen · 2019-09-27T15:24:04Z

Merged to master

yaooqinn · 2021-01-28T09:57:35Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

+    // since HIVE-11878, and ADDJarCommand will add jars to clientLoader.classLoader.
+    // For this reason we cannot load the jars added by ADDJarCommand because of class loader
+    // got changed. We reset it to clientLoader.ClassLoader here.
+    if (HiveUtils.isHive23) {


Why not set the classLoader blindly to clientLoader.classLoader? HIVE-11878 got merged into hive 1.3.0 and 2.0.0? This issue can still exist in the production environment when we try to use non-builtin hive libraries.

cc @ulysses-you @cloud-fan @wangyum @AngersZhuuuu

AngersZhuuuu added 5 commits September 10, 2019 17:44

save change

c3de557

Revert "save change"

2cf3153

This reverts commit c3de557.

Merge remote-tracking branch 'upstream/master'

e5cd06c

Merge remote-tracking branch 'upstream/master'

569ab8a

fix bug of jdk11

edaf402

AngersZhuuuu added 2 commits September 13, 2019 02:10

Revert "fix bug of jdk11"

7ec01e4

This reverts commit edaf402.

fix only this point

098a432

dongjoon-hyun added the SQL label Sep 12, 2019

still detach SessionState

07dd32e

remove unused import

35f0a01

fix in HiveTableScanExec

fb39a53

AngersZhuuuu changed the title ~~[WIP][SPARK-29015][SQL][test-hadoop3.2][test-java11] fix bug for jdk11, can't use 'add jar' jar's class as create table serde~~ [WIP][SPARK-29015][SQL][test-hadoop3.2] fix bug for jdk11, can't use 'add jar' jar's class as create table serde Sep 13, 2019

AngersZhuuuu changed the title ~~[WIP][SPARK-29015][SQL][test-hadoop3.2] fix bug for jdk11, can't use 'add jar' jar's class as create table serde~~ [WIP][SPARK-29015][SQL] fix bug for jdk11, can't use 'add jar' jar's class as create table serde Sep 13, 2019

AngersZhuuuu changed the title ~~[WIP][SPARK-29015][SQL] fix bug for jdk11, can't use 'add jar' jar's class as create table serde~~ [WIP][SPARK-29015][SQL][test-hadoop3.2] fix bug for jdk11, can't use 'add jar' jar's class as create table serde Sep 13, 2019

reset origin classloader

a510a23

fix scala style

bc73663

wangyum reviewed Sep 16, 2019

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala Outdated Show resolved Hide resolved

solve this when construct HiveClientImpl's SessionState.

35c961f

wangyum mentioned this pull request Sep 16, 2019

[SPARK-29022][SQL]Fix SparkSQLCLI can not add jars by AddJarCommand #25729

Closed

wangyum reviewed Sep 17, 2019

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala Outdated Show resolved Hide resolved

fix and add comment

f30d001

use clientLoader.classLoader

4db3b54

AngersZhuuuu changed the title ~~[WIP][SPARK-29015][SQL][test-hadoop3.2] fix bug for jdk11, can't use 'add jar' jar's class as create table serde~~ [SPARK-29015][SQL][test-hadoop3.2] fix bug for jdk11, can't use 'add jar' jar's class as create table serde Sep 20, 2019

wangyum mentioned this pull request Sep 25, 2019

[SPARK-27831][FOLLOW-UP][SQL][TEST] Should not use maven to add Hive test jars #25690

Closed

fix hive version wrong in comment

8e17346

change comment and add Hive Version condition

f37335c

AngersZhuuuu changed the title ~~[SPARK-29015][SQL][test-hadoop3.2] fix bug for jdk11, can't use 'add jar' jar's class as create table serde~~ [SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 Sep 26, 2019

wangyum reviewed Sep 26, 2019

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala Show resolved Hide resolved

wangyum reviewed Sep 26, 2019

View reviewed changes

srowen approved these changes Sep 27, 2019

View reviewed changes

srowen closed this in cc852d4 Sep 27, 2019

07ARB mentioned this pull request Oct 23, 2019

[SPARK-29453][WEBUI] Improve tooltips information for SQL tab. #26216

Closed

yaooqinn reviewed Jan 28, 2021

View reviewed changes

[SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 #25775

[SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 #25775

Uh oh!

Conversation

AngersZhuuuu commented Sep 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

dongjoon-hyun commented Sep 12, 2019

Uh oh!

AngersZhuuuu commented Sep 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

AngersZhuuuu commented Sep 13, 2019

Uh oh!

dongjoon-hyun commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 13, 2019

Uh oh!

SparkQA commented Sep 14, 2019

Uh oh!

SparkQA commented Sep 14, 2019

Uh oh!

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!

Uh oh!

wangyum commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

wangyum commented Sep 25, 2019

Uh oh!

srowen commented Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangyum commented Sep 25, 2019

Uh oh!

SparkQA commented Sep 25, 2019

Uh oh!

dongjoon-hyun commented Sep 25, 2019

Uh oh!

Uh oh!

wangyum Sep 26, 2019

Choose a reason for hiding this comment

Uh oh!

juliuszsompolski Sep 26, 2019

Choose a reason for hiding this comment

Uh oh!

srowen commented Sep 27, 2019

Uh oh!

yaooqinn Jan 28, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

AngersZhuuuu commented Sep 12, 2019 •

edited

Loading

AngersZhuuuu commented Sep 12, 2019 •

edited

Loading

srowen commented Sep 25, 2019 •

edited

Loading