[SPARK-27831][FOLLOW-UP][SQL][TEST] Should not use maven to add Hive test jars #25690

wangyum · 2019-09-05T07:09:02Z

What changes were proposed in this pull request?

This PR moves Hive test jars(hive-contrib-*.jar and hive-hcatalog-core-*.jar) from maven dependency to local file.

Why are the changes needed?

--jars can't be tested since hive-contrib-*.jar and hive-hcatalog-core-*.jar are already in classpath.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

manual test

…ncy" This reverts commit d53b61c

wangyum · 2019-09-05T07:10:01Z

@felixcheung @dongjoon-hyun

SparkQA · 2019-09-05T09:38:13Z

Test build #110166 has finished for PR 25690 at commit b02cc4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TestHiveVersion(hiveClient: HiveClient)
class TestHiveContext(
case class TestTable(name: String, commands: (() => Unit)*)
protected[hive] implicit class SqlCmd(sql: String)

wangyum · 2019-09-05T10:53:59Z

retest this please

SparkQA · 2019-09-05T13:45:56Z

Test build #110178 has finished for PR 25690 at commit b02cc4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TestHiveVersion(hiveClient: HiveClient)
class TestHiveContext(
case class TestTable(name: String, commands: (() => Unit)*)
protected[hive] implicit class SqlCmd(sql: String)

srowen · 2019-09-05T13:46:05Z

sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala


  test("Commands using SerDe provided in --jars") {
-    val jarFile = HiveTestUtils.getHiveHcatalogCoreJar.getCanonicalPath
+    val jarFile = "../hive/src/test/resources/" +


Adding these JARs to the source tree has some LICENSE and NOTICE implications. The Hive NOTICE text from these JARs would have to go in NOTICE (and probably NOTICE-binary as we published test JARs). They'd have to be listed in LICENSE and LICENSE-binary too.

This is possible, but is it equally possible to just download these like we do with other Hive jars?

dongjoon-hyun

Technically, this doesn't look like a complete revert of that commit. In this case, please be clear on that.
For example, this PR didn't revert the following from hive-thriftserver/pom.xml.

d53b61c#diff-e5643ab5905e6ce5d6447ad57d3a3067R50-R56

dongjoon-hyun · 2019-09-05T17:32:08Z

IMO, if you still depends on a some code in the previous commit, this is just a clean-up follow-up.

If you use Revert, some downstreams tries to skip both commits (this and the original one). Then, it will fails.

wangyum · 2019-09-06T03:26:04Z

thank you @srowen @dongjoon-hyun I see. Will update it later.

HyukjinKwon · 2019-09-06T03:49:15Z

(Let's update and elaborate the PR description as well)

SparkQA · 2019-09-06T03:53:49Z

Test build #110216 has finished for PR 25690 at commit ca57402.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-06T07:05:02Z

Test build #110223 has finished for PR 25690 at commit 1a90f38.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-06T07:22:13Z

retest this please

SparkQA · 2019-09-06T13:20:40Z

Test build #110228 has finished for PR 25690 at commit 1a90f38.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-09-06T14:17:26Z

I think the approach is valid, although I know there's a strong preference to avoid putting JAR files in the source tree. This is possibly a valid case for doing so, as it's test code and kind of requires the JAR file to exist locally. That said, I wonder if we can just reuse the code that is used elsewhere to download Hive JARs here? it might need a little refactoring, but it already exists, to use the ASF mirrors, etc.

SparkQA · 2019-09-07T09:44:52Z

Test build #110276 has finished for PR 25690 at commit e18cf28.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TestHiveVersion(hiveClient: HiveClient)
class TestHiveContext(
case class TestTable(name: String, commands: (() => Unit)*)
protected[hive] implicit class SqlCmd(sql: String)

SparkQA · 2019-09-07T10:11:35Z

Test build #110277 has finished for PR 25690 at commit 64c614a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-07T15:54:14Z

Test build #110279 has finished for PR 25690 at commit d5d0b7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-24T15:27:50Z

retest this please

SparkQA · 2019-09-24T15:32:11Z

Test build #111294 has started for PR 25690 at commit e4ed806.

wangyum · 2019-09-24T17:07:35Z

retest this please

SparkQA · 2019-09-24T22:06:27Z

Test build #111301 has finished for PR 25690 at commit e4ed806.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-24T23:28:54Z

retest this please

SparkQA · 2019-09-25T04:36:18Z

Test build #111314 has finished for PR 25690 at commit e4ed806.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-25T05:59:12Z

@srowen @HyukjinKwon @dongjoon-hyun The test failure fixed by #25775. Which one should be merged first?

- SPARK-8368: includes jars passed in through --jars *** FAILED ***
...
  2019-09-24 20:55:40.8 - stderr> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:123)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf$lzycompute(HiveTableScanExec.scala:101)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf(HiveTableScanExec.scala:98)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader$lzycompute(HiveTableScanExec.scala:110)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader(HiveTableScanExec.scala:105)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.hive.execution.HiveTableScanExec.$anonfun$doExecute$1(HiveTableScanExec.scala:188)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.util.Utils$.withDummyCallSite(Utils.scala:2488)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.hive.execution.HiveTableScanExec.doExecute(HiveTableScanExec.scala:188)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:189)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:227)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:224)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:185)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD$lzycompute(ShuffleExchangeExec.scala:64)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.inputRDD(ShuffleExchangeExec.scala:64)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency$lzycompute(ShuffleExchangeExec.scala:74)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.shuffleDependency(ShuffleExchangeExec.scala:72)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.createShuffledRDD(ShuffleExchangeExec.scala:82)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.$anonfun$doExecute$1(ShuffleExchangeExec.scala:93)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
  2019-09-24 20:55:40.8 - stderr> 	... 67 more
  2019-09-24 20:55:40.8 - stderr> Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
  2019-09-24 20:55:40.8 - stderr> 	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
  2019-09-24 20:55:40.8 - stderr> 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
  2019-09-24 20:55:40.8 - stderr> 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
  2019-09-24 20:55:40.8 - stderr> 	at java.base/java.lang.Class.forName0(Native Method)
  2019-09-24 20:55:40.8 - stderr> 	at java.base/java.lang.Class.forName(Class.java:398)
  2019-09-24 20:55:40.8 - stderr> 	at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:76)

srowen · 2019-09-27T15:24:25Z

@wangyum #25775 is merged

SparkQA · 2019-09-27T15:25:08Z

Test build #4886 has started for PR 25690 at commit e4ed806.

wangyum · 2019-09-27T15:41:28Z

Thank you @srowen

wangyum · 2019-09-27T15:52:32Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala

 }
+
+private[hive] object HiveTestJars {
+  private val repository = SQLConf.ADDITIONAL_REMOTE_REPOSITORIES.defaultValueString


We can also verify that the default value is valid.

SparkQA · 2019-09-27T18:50:00Z

Test build #111486 has finished for PR 25690 at commit 882ae45.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-27T23:14:53Z

retest this please

SparkQA · 2019-09-28T05:19:36Z

Test build #111502 has finished for PR 25690 at commit 882ae45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-28T07:11:42Z

retest this please

SparkQA · 2019-09-28T10:00:41Z

Test build #111527 has finished for PR 25690 at commit 882ae45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-09-28T16:41:52Z

@srowen @dongjoon-hyun @HyukjinKwon Do you have any comment?

wangyum · 2019-09-28T23:56:32Z

Thank you all.

wangyum · 2019-09-28T23:56:43Z

Merged to master.

Revert "[SPARK-27831][SQL][TEST] Move Hive test jars to maven depende…

b02cc4d

…ncy" This reverts commit d53b61c

wangyum changed the title ~~Revert "[SPARK-27831][SQL][TEST] Move Hive test jars to maven dependency"~~ [test-hadoop3.2] Revert "[SPARK-27831][SQL][TEST] Move Hive test jars to maven dependency" Sep 5, 2019

srowen reviewed Sep 5, 2019

View reviewed changes

dongjoon-hyun added the TESTS label Sep 5, 2019

dongjoon-hyun previously requested changes Sep 5, 2019

View reviewed changes

wangyum changed the title ~~[test-hadoop3.2] Revert "[SPARK-27831][SQL][TEST] Move Hive test jars to maven dependency"~~ [SPARK-27831][FOLLOW-UP][SQL][TEST][test-maven] Move Hive test jars to maven dependency Sep 6, 2019

HiveTestJars

ca57402

Fix build error

1a90f38

wangyum changed the title ~~[SPARK-27831][FOLLOW-UP][SQL][TEST][test-maven] Move Hive test jars to maven dependency~~ [WIP][SPARK-27831][FOLLOW-UP][SQL][TEST][test-maven] Move Hive test jars to local file Sep 6, 2019

wangyum changed the title ~~[WIP][SPARK-27831][FOLLOW-UP][SQL][TEST][test-maven] Move Hive test jars to local file~~ [WIP][SPARK-27831][FOLLOW-UP][SQL][TEST] Move Hive test jars to local file Sep 7, 2019

wangyum added 2 commits September 7, 2019 15:59

Download Hive test jars

e18cf28

Fix test error

64c614a

wangyum added 2 commits September 7, 2019 21:32

try fix test error

d5d0b7f

Try fix test error 2

469d533

wangyum changed the title ~~[WIP][SPARK-27831][FOLLOW-UP][SQL][TEST] Move Hive test jars to local file~~ [WIP][SPARK-27831][FOLLOW-UP][SQL][TEST][test-hadoop3.2] Move Hive test jars to local file Sep 7, 2019

srowen mentioned this pull request Sep 25, 2019

[SPARK-29015][SQL][test-hadoop3.2]Reset class loader after initializing SessionState for built-in Hive 2.3 #25775

Closed

Merge remote-tracking branch 'upstream/master' into SPARK-27831-revert

cd94ac1

SQLConf.ADDITIONAL_REMOTE_REPOSITORIES.defaultValueString

882ae45

wangyum commented Sep 27, 2019

View reviewed changes

wangyum changed the title ~~[SPARK-27831][FOLLOW-UP][SQL][TEST][test-maven][test-hadoop3.2][test-java11] Move Hive test jars to local file~~ [SPARK-27831][FOLLOW-UP][SQL][TEST] Move Hive test jars to local file Sep 28, 2019

wangyum changed the title ~~[SPARK-27831][FOLLOW-UP][SQL][TEST] Move Hive test jars to local file~~ [SPARK-27831][FOLLOW-UP][SQL][TEST] Should not use maven to add Hive test jars Sep 28, 2019

srowen approved these changes Sep 28, 2019

View reviewed changes

wangyum closed this in 8167714 Sep 28, 2019

wangyum deleted the SPARK-27831-revert branch September 28, 2019 23:56

Uh oh!

[SPARK-27831][FOLLOW-UP][SQL][TEST] Should not use maven to add Hive test jars #25690

[SPARK-27831][FOLLOW-UP][SQL][TEST] Should not use maven to add Hive test jars #25690

Uh oh!

Conversation

wangyum commented Sep 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

wangyum commented Sep 5, 2019

Uh oh!

SparkQA commented Sep 5, 2019

Uh oh!

wangyum commented Sep 5, 2019

Uh oh!

SparkQA commented Sep 5, 2019

Uh oh!

srowen Sep 5, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Sep 5, 2019

Uh oh!

wangyum commented Sep 6, 2019

Uh oh!

HyukjinKwon commented Sep 6, 2019

Uh oh!

SparkQA commented Sep 6, 2019

Uh oh!

SparkQA commented Sep 6, 2019

Uh oh!

wangyum commented Sep 6, 2019

Uh oh!

SparkQA commented Sep 6, 2019

Uh oh!

srowen commented Sep 6, 2019

Uh oh!

SparkQA commented Sep 7, 2019

Uh oh!

SparkQA commented Sep 7, 2019

Uh oh!

SparkQA commented Sep 7, 2019

Uh oh!

wangyum commented Sep 24, 2019

Uh oh!

SparkQA commented Sep 24, 2019

Uh oh!

wangyum commented Sep 24, 2019

Uh oh!

SparkQA commented Sep 24, 2019

Uh oh!

wangyum commented Sep 24, 2019

Uh oh!

SparkQA commented Sep 25, 2019

Uh oh!

wangyum commented Sep 25, 2019

Uh oh!

srowen commented Sep 27, 2019

Uh oh!

SparkQA commented Sep 27, 2019

Uh oh!

wangyum commented Sep 27, 2019

Uh oh!

wangyum Sep 27, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 27, 2019

Uh oh!

wangyum commented Sep 27, 2019

Uh oh!

SparkQA commented Sep 28, 2019

Uh oh!

wangyum commented Sep 28, 2019

Uh oh!

SparkQA commented Sep 28, 2019

wangyum commented Sep 5, 2019 •

edited

Loading