[SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency

[wangyum]

What changes were proposed in this pull request?

This pr moves Hive test jars(hive-contrib-0.13.1.jar, hive-hcatalog-core-0.13.1.jar, hive-contrib-2.3.5.jar and hive-hcatalog-core-2.3.5.jar) to maven dependency.

How was this patch tested?

Existing test

[wangyum]

cc @felixcheung @dongjoon-hyun @HyukjinKwon

[SparkQA]

Test build #105756 has finished for PR 24695 at commit 0fa9766.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

retest this please

[SparkQA]

Test build #105759 has finished for PR 24695 at commit 0fa9766.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1. This looks much better to me. Merged to master.

Thank you for guide, @felixcheung . Thank you for the patch, @wangyum !

[srowen]

@wangyum I'm seeing failures from the master build from about the time this was committed, like:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/6393

===== FINISHED o.a.s.sql.hive.thriftserver.HiveThriftBinaryServerSuite: 'test add jar' =====

*** RUN ABORTED ***
  java.lang.NoClassDefFoundError: org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax
  at org.apache.spark.sql.hive.test.HiveTestUtils$.<init>(HiveTestUtils.scala:28)
  at org.apache.spark.sql.hive.test.HiveTestUtils$.<clinit>(HiveTestUtils.scala)

Do you think it could be related?

[dongjoon-hyun]

Thank you for heading up. Maven seems to work differently. Let me revert this.

[srowen]

I'm not sure if it's the problem, just looked potentially related and happened right before the failures started happening. We might investigate a little bit to see if there's an easy fix-forward, or whether it's the issue

[dongjoon-hyun]

We had better recover the Jenkins first. Then, we can have enough time to get a correct patch. Sorry for the troubles in maven build~

[wangyum]

Add hive-hcatalog-core to sql/hive-thriftserver/pom.xml can fix this issue:

    <!-- Explicit listing hive-hcatalog-core. Otherwise the maven build fails. -->
    <dependency>
      <groupId>${hive.group}.hcatalog</groupId>
      <artifactId>hive-hcatalog-core</artifactId>
    </dependency>

It can find hive-hcatalog-core:jar after adding hive-hcatalog-core to sql/hive-thriftserver/pom.xml:

[INFO] +- org.spark-project.hive.hcatalog:hive-hcatalog-core:jar:1.2.1.spark2:test

[srowen]

@wangyum if that fixes forward, let's try that instead of a revert.

[dongjoon-hyun]

Thank you, @wangyum. Yes, please make a PR with that approach, @wangyum . At this time, let's test with all combinations. Last time, we tested SBT(hadoop-2.7/hadoop-3.2) only.

@srowen . Unfortunately, this patch is already reverted from the master, and it's verified in another PR (here, #24745 (review)) to pass the Maven.

• https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105971/

The last Jenkins passed the Hive UT but failed at some YARN UTs. I guess the next run will pass.