Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-27737][SQL] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile #24620

Closed
wants to merge 4 commits into from

Conversation

Projects
None yet
6 participants
@wangyum
Copy link
Contributor

commented May 15, 2019

What changes were proposed in this pull request?

This PR aims to upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile.

Release Notes - Hive - Version 2.3.5

  • [HIVE-21536] - Backport HIVE-17764 to branch-2.3
  • [HIVE-21585] - Upgrade branch-2.3 to ORC 1.3.4
  • [HIVE-21639] - Spark test failed since HIVE-10632
  • [HIVE-21680] - Backport HIVE-17644 to branch-2 and branch-2.3

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345394&styleName=Text&projectId=12310843

How was this patch tested?

This PR is tested in two ways.

  • Pass the Jenkins with the default configuration for Hive Metastore Client testing.
  • Pass the Jenkins with test-hadoop3.2 configuration for Hadoop 3.2 testing.
@SparkQA

This comment has been minimized.

Copy link

commented May 15, 2019

Test build #105431 has finished for PR 24620 at commit 34e4817.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@HyukjinKwon

This comment has been minimized.

Copy link
Member

commented May 16, 2019

retest this please

@HyukjinKwon
Copy link
Member

left a comment

yes, looks good if the tests pass.

@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

The Hive 2.3.5 jars is not in the maven repository. May be need to wait for a while?

@SparkQA

This comment has been minimized.

Copy link

commented May 16, 2019

Test build #105451 has started for PR 24620 at commit 34e4817.

@HyukjinKwon

This comment has been minimized.

Copy link
Member

commented May 16, 2019

okie.

@dongjoon-hyun
Copy link
Member

left a comment

@wangyum . Please merge this and #24621 because we had better update the dependency file consistently across all profiles from Hive 2.3.5.

cc @gatorsmile and @srowen .

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 16, 2019

BTW, @wangyum . To get reviews, you had better make a PR after the maven repository is ready next time. :)

@HyukjinKwon

This comment has been minimized.

Copy link
Member

commented May 16, 2019

Yea agree.

@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

retest this please

@SparkQA

This comment has been minimized.

Copy link

commented May 17, 2019

Test build #105478 has finished for PR 24620 at commit 34e4817.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@srowen

srowen approved these changes May 17, 2019

@SparkQA

This comment has been minimized.

Copy link

commented May 20, 2019

Test build #105556 has finished for PR 24620 at commit 4fd1f83.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.
@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 20, 2019

retest this please

@SparkQA

This comment has been minimized.

Copy link

commented May 20, 2019

Test build #105558 has finished for PR 24620 at commit 4fd1f83.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 20, 2019

Thank you for updating, @wangyum . After #24639 is merged, I'll retrigger this with [test-hadoop3.2][test-maven].

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-27737][SQL] Upgrade to 2.3.5 for Hive Metastore Client 2.3 [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to 2.3.5 for Hive Metastore Client 2.3 May 20, 2019

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 20, 2019

Retest this please.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to 2.3.5 for Hive Metastore Client 2.3 [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 May 20, 2019

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 20, 2019

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 20, 2019

I updated the PR title and description. The last Jenkins run is running with the following.

[info] Building Spark using Maven with these arguments: 
-Phadoop-3.2 -Pkubernetes -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl
-Phive -Pmesos clean package -DskipTests
@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 20, 2019

Hive 2.3.5 should support test with sbt.

@wangyum wangyum changed the title [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 20, 2019

@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 20, 2019

retest this please

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 20, 2019

Thank you for additional one, @wangyum . To sum up,

- [info] Building Spark using SBT with these arguments:  -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly

@wangyum . Do you want to trigger maven on hadoop-2.7, too?

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 21, 2019

Yep. So, if you want, you can trigger it to verify this PR completely.

Do you think we need test maven on hadoop-2.7?

@wangyum wangyum changed the title [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile [SPARK-27737][SQL][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 21, 2019

@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

retest this please

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 21, 2019

Ur, @wangyum . During merging, it seems that we missed to upgrade hadoop-3.2 profile properly.

      <id>hadoop-3.2</id>
        ...
        <hive.version.short>2.3.4</hive.version.short>

Please update hadoop-3.2 profile and update the manifests.

@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

How about change hive.version.short to 2.3.0?

@SparkQA

This comment has been minimized.

Copy link

commented May 21, 2019

Test build #105585 has finished for PR 24620 at commit 4fd1f83.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 21, 2019

Is there a reason to use that?

How about change hive.version.short to 2.3.0?

@SparkQA

This comment has been minimized.

Copy link

commented May 21, 2019

Test build #105587 has finished for PR 24620 at commit 4fd1f83.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

2 reasons:

  1. Avoid changing these directories: sql/core/v2.3.4/src, sql/hive-thriftserver/v2.3.4 and hive.version.short if we upgrade Hive 2.3 to 2.3.6, 2.3.7 ...
  2. 2.3.0 seems more natural:
    image
@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 21, 2019

Does something fail with 2.3.5? Before Spark 3.0.0 release, Hive 2.3.5 seems to be the last and the most stable Hive release. So, shall we catch up at this time? It would be great if you do the following.

  • dev/test-dependencies.sh --replace-manifest
  • sql/core/v2.3.4/src -> sql/core/v2.3.5/src

For shortVersion, we use 1.2.1 previously. So, 2.3.5 looks fine to me.

@wangyum wangyum changed the title [SPARK-27737][SQL][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 21, 2019

@SparkQA

This comment has been minimized.

Copy link

commented May 21, 2019

Test build #105605 has finished for PR 24620 at commit 0f2eb07.

  • This patch fails to build.
  • This patch does not merge cleanly.
  • This patch adds no public classes.
@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

@HyukjinKwon It add -Phive-thriftserver after merge upstream:

========================================================================
Building Spark
========================================================================
[info] Building Spark using SBT with these arguments:  -Phadoop-3.2 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly
@SparkQA

This comment has been minimized.

Copy link

commented May 21, 2019

Test build #105607 has finished for PR 24620 at commit 32fc1ca.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@HyukjinKwon

This comment has been minimized.

Copy link
Member

commented May 21, 2019

========================================================================
Building Spark
========================================================================
[info] Building Spark using SBT with these arguments:  -Phadoop-3.2 -Pkubernetes -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly

Seems not. which build are you referring?

@HyukjinKwon

This comment has been minimized.

Copy link
Member

commented May 21, 2019

@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

Retest this please

@HyukjinKwon

This comment has been minimized.

Copy link
Member

commented May 21, 2019

retest this please

@SparkQA

This comment has been minimized.

Copy link

commented May 21, 2019

Test build #105626 has finished for PR 24620 at commit 32fc1ca.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
@wangyum

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

retest this please

@SparkQA

This comment has been minimized.

Copy link

commented May 21, 2019

Test build #105631 has finished for PR 24620 at commit 32fc1ca.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 21, 2019

@dongjoon-hyun

This comment has been minimized.

Copy link
Member

commented May 21, 2019

Retest this please.

@SparkQA

This comment has been minimized.

Copy link

commented May 22, 2019

Test build #105636 has finished for PR 24620 at commit 32fc1ca.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon changed the title [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile [SPARK-27737][SQL] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 22, 2019

@HyukjinKwon

This comment has been minimized.

Copy link
Member

commented May 22, 2019

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.