[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

pan3793 · 2024-03-04T11:54:27Z

What changes were proposed in this pull request?

This PR aims to do a pre-test before the Apache Hive 2.3.10 official release, it includes

it should unblock Spark to upgrade to Guava 32+
it should unblock Spark to upgrade to Thrift 0.16+
it should allow Spark to drop Jackson 1.x while Hive UDF still works well
it should allow Spark to drop javax.servlet(HIVE-28104. (2.3) Move HTTP related methods from Utils to HttpUtils in shims hive#5114)
it should allow Spark to get rid of the low version of jodd which is affected by CVEs (HIVE-25054: (2.3) Drop vulnerable jodd-core dependency hive#5151)
it should allow Spark to drop commons-lang:2.6

Why are the changes needed?

Integration test of Hive 2.3.10 RC1 https://lists.apache.org/thread/3ton9oxlkkb1v7tbqqw1rhjpz9y8ymv5

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No

pan3793 · 2024-03-04T17:51:08Z

cc @sunchao, so far, the Hive branch-2.3 looks good on the Spark side.

sunchao · 2024-03-04T17:55:10Z

Thanks, that's good to know. I was informed that I should consider the following JIRAs too. Plan to check soon.

dongjoon-hyun · 2024-03-04T18:44:27Z

Thank you for making progress, @pan3793 and @sunchao .

LuciferYang · 2024-03-05T05:13:04Z

happy to see this hive upgrade, thanks to @pan3793 and @sunchao

LuciferYang · 2024-03-05T05:18:56Z

pom.xml

@@ -199,14 +197,14 @@
    <!-- org.apache.commons/commons-pool2/-->
    <commons-pool2.version>2.12.0</commons-pool2.version>
    <datanucleus-core.version>4.1.17</datanucleus-core.version>
-    <guava.version>14.0.1</guava.version>
+    <guava.version>33.0.0-jre</guava.version>


@pan3793 If we upgrade the version of Guava, and it is also packaged into the newwork-shuffle and core module through the shaded+relocation method, then we need to package com.google.guava:failureaccess at the same time.

We actually cannot avoid exposing vanilla Guava to classpath, except to Spark internal usage, 3-rd party libs still require Guava to work, e.g. Curator, Hive. Then, should we continue to do shading after upgrading?

pan3793 · 2024-03-21T16:10:01Z

@sunchao I checked the JIRA tickets you listed, all are related to LICENSE, I created those PRs except for the last one (obviously it has conflicts with previous patches), should be easy to review, please take a look when you have time.

pan3793 · 2024-04-21T09:27:05Z

@dongjoon-hyun @sunchao @LuciferYang the integration test with Hive 2.3.10 RC0 passed, do you have other concerns or extra cases that need to be verified?

pan3793 · 2024-04-21T09:28:20Z

also cc @wangyum @HyukjinKwon @yaooqinn

LuciferYang · 2024-04-22T02:41:53Z

dev/deps/spark-deps-hadoop-3-hive-2.3

@@ -33,6 +33,7 @@ breeze-macros_2.13/2.1.0//breeze-macros_2.13-2.1.0.jar
 breeze_2.13/2.1.0//breeze_2.13-2.1.0.jar
 bundle/2.24.6//bundle-2.24.6.jar
 cats-kernel_2.13/2.8.0//cats-kernel_2.13-2.8.0.jar
+checker-qual/3.42.0//checker-qual-3.42.0.jar


Is this a cascading dependency brought by the new version of Guava?

LuciferYang · 2024-04-22T02:47:35Z

project/SparkBuild.scala

@@ -952,7 +955,7 @@ object Unsafe {
 object DockerIntegrationTests {
  // This serves to override the override specified in DependencyOverrides:
  lazy val settings = Seq(
-    dependencyOverrides += "com.google.guava" % "guava" % "33.0.0-jre"
+    dependencyOverrides += "com.google.guava" % "guava" % "33.1.0-jre"


Will similar configurations still be needed after this pr?

This PR is mainly of PoC, so I just make minimal changes to verify it works. I think we can unify the Guava version in all places after upgrading Hive 2.3.10, in an independent PR.

Let me unblock this like

[SPARK-47025][BUILD][TESTS] Upgrade Guava dependency in docker-integration-tests test module #45088

sql/hive/src/test/java/org/apache/spark/sql/hive/test/Complex.java

dongjoon-hyun

Thank you for making a progress on this issue. Since this is Draft PR, I believe it's okay to use the existing JIRA ID.

BTW, as mentioned by Yang Jie. There are several irrelevant updates in this PR. I guess we can minimize them before the official Hive release.

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

dongjoon-hyun · 2024-04-22T03:58:52Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala

@@ -1626,6 +1625,7 @@ class HiveQuerySuite extends HiveComparisonTest with SQLTestUtils with BeforeAnd

  test("SPARK-33084: Add jar support Ivy URI in SQL") {
    val testData = TestHive.getHiveFile("data/files/sample.json").toURI
+    val hiveVersion = "2.3.9" // TODO remove after Hive 2.3.10 jar available in maven central


Ditto. Do you know why DEFAULT_ARTIFACT_REPOSITORY change isn't enough here?

I remember it did not work on my first try, let me try again

I searched DEFAULT_ARTIFACT_REPOSITORY, it looks like affects multiple places, I addressed the ADD JAR in another approach ec9848f

dongjoon-hyun · 2024-04-22T17:37:18Z

For Docker Integration test dependency, I made the following PR with @pan3793 's authorship.

[SPARK-47940][BUILD][TESTS] Upgrade guava dependency to 33.1.0-jre in Docker IT #46167

dongjoon-hyun · 2024-04-22T17:39:06Z

dev/test-dependencies.sh

@@ -49,7 +49,7 @@ OLD_VERSION=$($MVN -q \
    --non-recursive \
    org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -E '[0-9]+\.[0-9]+\.[0-9]+')
 # dependency:get for guava and jetty-io are workaround for SPARK-37302.
-GUAVA_VERSION=$(build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout | grep -E "^[0-9.]+$")
+GUAVA_VERSION=$(build/mvn help:evaluate -Dexpression=guava.version -q -DforceStdout | grep -E "^[0-9\.]+")


Just a question. Why do we need this change?

The recent Guava version has the suffix -jre or -android, the change is required to match the new version

…` in Docker IT ### What changes were proposed in this pull request? This PR aims to upgrade `guava` dependency to `33.1.0-jre` in Docker Integration tests. ### Why are the changes needed? This is a preparation of the following PR. - #45372 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46167 from dongjoon-hyun/SPARK-47940. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

dongjoon-hyun · 2024-05-06T18:26:39Z

Could you update the PR description with RC1 link, @pan3793 ?

https://lists.apache.org/thread/3ton9oxlkkb1v7tbqqw1rhjpz9y8ymv5

pan3793 · 2024-05-06T18:34:29Z

@dongjoon-hyun thanks for checking, updated

…` in Docker IT ### What changes were proposed in this pull request? This PR aims to upgrade `guava` dependency to `33.1.0-jre` in Docker Integration tests. ### Why are the changes needed? This is a preparation of the following PR. - apache#45372 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46167 from dongjoon-hyun/SPARK-47940. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to bump Spark's built-in Hive from 2.3.9 to Hive 2.3.10, with two additional changes: - due to API breaking changes of Thrift, `libthrift` is upgraded from `0.12` to `0.16`. - remove version management of `commons-lang:2.6`, it comes from Hive transitive deps, Hive 2.3.10 drops it in apache/hive#4892 This is the first part of apache#45372 ### Why are the changes needed? Bump Hive to the latest version of 2.3, prepare for upgrading Guava, and dropping vulnerable dependencies like Jackson 1.x / Jodd ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. (wait for sunchao to complete the 2.3.10 release to make jars visible on Maven Central) ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45372 Closes apache#46468 from pan3793/SPARK-47018. Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

github-actions bot added SQL BUILD CORE CONNECT labels Mar 4, 2024

LuciferYang reviewed Mar 5, 2024

View reviewed changes

pan3793 force-pushed the hive-2310 branch from 8591dbd to 9880a55 Compare March 29, 2024 05:48

dongjoon-hyun marked this pull request as draft March 29, 2024 20:38

pan3793 force-pushed the hive-2310 branch from b575573 to 7194f8d Compare April 21, 2024 04:34

github-actions bot added the DSTREAM label Apr 21, 2024

pan3793 changed the title ~~[DO-NOT-MERGE] Test Hive pre-2.3.10~~ [DO-NOT-MERGE] Test Apache Hive 2.3.10 RC0 Apr 21, 2024

pan3793 force-pushed the hive-2310 branch from a4902c6 to 2f220f6 Compare April 21, 2024 06:04

LuciferYang reviewed Apr 22, 2024

View reviewed changes

sql/hive/src/test/java/org/apache/spark/sql/hive/test/Complex.java Outdated Show resolved Hide resolved

dongjoon-hyun changed the title ~~[DO-NOT-MERGE] Test Apache Hive 2.3.10 RC0~~ [SPARK-47018][BUILD] Upgrade built-in Hive to 2.3.10 Apr 22, 2024

dongjoon-hyun reviewed Apr 22, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 22, 2024

View reviewed changes

This was referenced Apr 22, 2024

[SPARK-47928][SQL][TEST] Speed up test "Add jar support Ivy URI in SQL" #46150

Closed

[SPARK-47940][BUILD][TESTS] Upgrade guava dependency to 33.1.0-jre in Docker IT #46167

Closed

dongjoon-hyun reviewed Apr 22, 2024

View reviewed changes

pan3793 force-pushed the hive-2310 branch from 04e3613 to f0eb2d9 Compare April 23, 2024 01:55

pan3793 added 14 commits May 6, 2024 14:03

github-repo

d457f85

Hive 2.3.101

647b0a4

jackson

f504753

Bump Hive 2.3.102 and exclude javax.servlet

2ecdfd8

Hive 2.3.10-3

59323ce

remove jodd

c8ca897

Apache Hive 2.3.10 RC0

eb7cc46

guava

e191b99

Remove codehaus.jackson.version

05c511e

Remove commons-lang definition

d7619bb

aggdesigner-algorithm

32d481f

guava

613eb85

ADD JAR

71e8f93

Apache Hive 2.3.10 RC1

a31b060

pan3793 force-pushed the hive-2310 branch from f0eb2d9 to a31b060 Compare May 6, 2024 06:07

fix

0a14994

pan3793 mentioned this pull request May 8, 2024

[SPARK-47018][BUILD][SQL] Bump built-in Hive to 2.3.10 #46468

Closed

pan3793 changed the title ~~[SPARK-47018][BUILD] Upgrade built-in Hive to 2.3.10~~ [DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 May 10, 2024

dongjoon-hyun closed this in 2d609bf May 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

pan3793 commented Mar 4, 2024 •

edited

Loading

pan3793 commented Mar 4, 2024

sunchao commented Mar 4, 2024 •

edited

Loading

dongjoon-hyun commented Mar 4, 2024

LuciferYang commented Mar 5, 2024

LuciferYang Mar 5, 2024 •

edited

Loading

pan3793 Mar 21, 2024

pan3793 commented Mar 21, 2024

pan3793 commented Apr 21, 2024 •

edited

Loading

pan3793 commented Apr 21, 2024

LuciferYang Apr 22, 2024

pan3793 Apr 22, 2024

LuciferYang Apr 22, 2024

pan3793 Apr 22, 2024

LuciferYang Apr 22, 2024

dongjoon-hyun Apr 22, 2024

dongjoon-hyun left a comment

dongjoon-hyun Apr 22, 2024

pan3793 Apr 22, 2024

pan3793 Apr 22, 2024

dongjoon-hyun commented Apr 22, 2024

dongjoon-hyun Apr 22, 2024

pan3793 Apr 23, 2024

dongjoon-hyun commented May 6, 2024

pan3793 commented May 6, 2024

[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

Conversation

pan3793 commented Mar 4, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

pan3793 commented Mar 4, 2024

sunchao commented Mar 4, 2024 • edited Loading

dongjoon-hyun commented Mar 4, 2024

LuciferYang commented Mar 5, 2024

LuciferYang Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pan3793 commented Mar 21, 2024

pan3793 commented Apr 21, 2024 • edited Loading

pan3793 commented Apr 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Apr 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented May 6, 2024

pan3793 commented May 6, 2024

pan3793 commented Mar 4, 2024 •

edited

Loading

sunchao commented Mar 4, 2024 •

edited

Loading

LuciferYang Mar 5, 2024 •

edited

Loading

pan3793 commented Apr 21, 2024 •

edited

Loading