[SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2 #34715

AngersZhuuuu · 2021-11-26T02:49:34Z

What changes were proposed in this pull request?

Rename the maven hadoop profile to hadoop-3 and hadoop-2

Why are the changes needed?

Upgrade hadoop profile

Does this PR introduce any user-facing change?

n this pr, we change hadoop profile form hadoop-3 and hadoop-2, and hadoop-3 is default hadoop profile.
User can build other lower version by -Dhadoop.version=3.2.1 such as

./build/mvn -Pyarn -Dhadoop.version=3.2.1 -DskipTests clean package

If user want to built hadoop 2 related version. you can run

./build/mvn -Pyarn -Phadoop-2 -Dhadoop.version=2.8.3 -DskipTests clean package

How was this patch tested?

Not need

…pport hadoop-3.3 as default now

SparkQA · 2021-11-26T03:54:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50113/

AngersZhuuuu · 2021-11-26T03:55:22Z

Should we need to clarify this change in https://spark.apache.org/docs/latest/api/python/getting_started/install.html#manually-downloading ?

cc @HyukjinKwon

SparkQA · 2021-11-26T04:11:24Z

Test build #145648 has finished for PR 34715 at commit 259dfc7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-26T04:38:13Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50113/

SparkQA · 2021-11-26T05:00:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50119/

SparkQA · 2021-11-26T05:10:15Z

Test build #145653 has finished for PR 34715 at commit 7f9b1f0.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-29T10:53:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50175/

SparkQA · 2021-11-29T11:47:34Z

Test build #145705 has finished for PR 34715 at commit 758b267.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sunchao

Looks pretty good @AngersZhuuuu ! will take another pass later.

python/pyspark/install.py

HyukjinKwon · 2021-12-01T09:08:33Z

From a cursory look, seems fine to me too but I think it's best to defer to @sunchao

mridulm · 2021-12-01T18:58:28Z

+CC @xkrogen

python/pyspark/install.py

resource-managers/kubernetes/integration-tests/README.md

resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

dev/run-tests-jenkins.py

sunchao · 2021-12-02T03:32:34Z

dev/test-dependencies.sh

@@ -34,8 +34,8 @@ HADOOP_MODULE_PROFILES="-Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive \
    -Pspark-ganglia-lgpl -Pkinesis-asl -Phadoop-cloud"
 MVN="build/mvn"
 HADOOP_HIVE_PROFILES=(
-    hadoop-2.7-hive-2.3
-    hadoop-3.2-hive-2.3
+    hadoop-2-hive-2.3


unrelated but the hive-2.3 seems redundant, since we don't support alternative Hive profile right now

docs/building-spark.md

pom.xml

python/docs/source/getting_started/install.rst

python/pyspark/install.py

SparkQA · 2021-12-05T02:26:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50401/

SparkQA · 2021-12-05T03:24:46Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50401/

SparkQA · 2021-12-05T04:22:11Z

Test build #145925 has finished for PR 34715 at commit c222d0e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-07T03:32:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50444/

SparkQA · 2021-12-07T04:14:26Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50444/

SparkQA · 2021-12-07T05:29:09Z

Test build #145969 has finished for PR 34715 at commit 07621be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-07T05:46:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50450/

SparkQA · 2021-12-07T06:47:08Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50450/

SparkQA · 2021-12-07T07:55:46Z

Test build #145975 has finished for PR 34715 at commit 7b39401.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sunchao

LGTM, thanks @AngersZhuuuu !

HyukjinKwon

Looks pretty good. After this gets merged, let's also send an email to dev mailing list about the renamed profiles to prevent other external build trains to be updated too in other forked repos.

HyukjinKwon · 2021-12-08T00:13:45Z

python/pyspark/install.py

+    return spark_version, convert_old_hadoop_version(spark_version, hadoop_version), hive_version
+
+
+def convert_old_hadoop_version(spark_version, hadoop_version):


Thanks for fixing this together guys.

AngersZhuuuu · 2021-12-08T02:22:14Z

@HyukjinKwon This GA failed caused by the same SQLMetricsSuite mentioned by you in my PR.

SPARK-34399: Add job commit duration metrics for DataWritingCommand

@sunchao I think we can merge this if there are no more question.
And I will review

SPARK-34399: Add job commit duration metrics for DataWritingCommand

and try to make it stable.

sunchao · 2021-12-08T02:28:58Z

Thanks, merged!

HyukjinKwon · 2021-12-08T02:46:51Z

Awesome. @AngersZhuuuu, mind sending an email to dev mailing list articulating what's the diff and impact? We would have to list up e.g.) profile name, released tar file name, pip installation option, etc.

HyukjinKwon · 2021-12-08T02:48:59Z

e.g.) if people are using hadoop-2.7 in their build, now their build will use hadoop 3 by default with warning that hadoop 2.7 profile does not exist.

AngersZhuuuu · 2021-12-08T02:50:58Z

Awesome. @AngersZhuuuu, mind sending an email to dev mailing list articulating what's the diff and impact? We would have to list up e.g.) profile name, released tar file name, pip installation option, etc.

Yea. Will organize an email later

…p3 for the scheduled GA job for branch-3.2 ### What changes were proposed in this pull request? This PR fixes an issue that the scheduled GA job for `branch-3.2` fails. SPARK-37445 (#34715), renamed the profile name `hadoop3.2` to `hadoop3` but It should be `hadoop3.2` for the scheduled build. https://github.com/apache/spark/runs/4453894964?check_suite_focus=true ### Why are the changes needed? To recover the job. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The scheduled job itself. Closes #34835 from sarutak/followup-SPARK-37445. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

AngersZhuuuu added 7 commits November 23, 2021 14:56

[SPARK-37445][BUILD] Upgrade hadoop profile to hadoop-3.3 since we su…

6a446eb

…pport hadoop-3.3 as default now

update

68eeb49

update

7d8e8b4

Update install.py

3e132f3

Merge branch 'master' into SPARK-37445

d3ad6e7

update hadoop-2

a5e1edd

update

d8d4092

github-actions bot added BUILD DOCS INFRA KUBERNETES SQL YARN labels Nov 26, 2021

AngersZhuuuu added 5 commits November 26, 2021 10:50

Merge branch 'master' into SPARK-37445

452f254

update

4c1d58b

update

c762f9e

update

b2b40d5

update

e2455e7

github-actions bot added CORE PYTHON labels Nov 26, 2021

Update install.rst

259dfc7

Update install.py

c441939

AngersZhuuuu added 2 commits November 26, 2021 13:01

fix ut

7f9b1f0

update

08cf8af

sunchao reviewed Nov 30, 2021

View reviewed changes

python/pyspark/install.py Show resolved Hide resolved

sunchao reviewed Dec 2, 2021

View reviewed changes

AngersZhuuuu added 2 commits December 5, 2021 09:21

Merge branch 'master' into SPARK-37445

7ed35af

follow comment

c222d0e

AngersZhuuuu added 2 commits December 7, 2021 10:23

Merge branch 'master' into SPARK-37445

e083cde

Follow comment

07621be

Update HiveExternalCatalogVersionsSuite.scala

7b39401

sunchao approved these changes Dec 7, 2021

View reviewed changes

HyukjinKwon reviewed Dec 8, 2021

View reviewed changes

sunchao closed this in fdc276b Dec 8, 2021

sarutak mentioned this pull request Dec 8, 2021

[SPARK-37445][INFRA][FOLLOWUP] Use hadoop3.2 profile instead of hadoop3 for the scheduled GA job for branch-3.2 #34835

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2 #34715

[SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2 #34715

AngersZhuuuu commented Nov 26, 2021 •

edited

SparkQA commented Nov 26, 2021

AngersZhuuuu commented Nov 26, 2021 •

edited

SparkQA commented Nov 26, 2021

SparkQA commented Nov 26, 2021

SparkQA commented Nov 26, 2021

SparkQA commented Nov 26, 2021

SparkQA commented Nov 29, 2021

SparkQA commented Nov 29, 2021

sunchao left a comment

HyukjinKwon commented Dec 1, 2021

mridulm commented Dec 1, 2021

sunchao Dec 2, 2021

SparkQA commented Dec 5, 2021

SparkQA commented Dec 5, 2021

SparkQA commented Dec 5, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

sunchao left a comment

HyukjinKwon left a comment •

edited

HyukjinKwon Dec 8, 2021

AngersZhuuuu commented Dec 8, 2021

sunchao commented Dec 8, 2021

HyukjinKwon commented Dec 8, 2021

HyukjinKwon commented Dec 8, 2021

AngersZhuuuu commented Dec 8, 2021

		return spark_version, convert_old_hadoop_version(spark_version, hadoop_version), hive_version


		def convert_old_hadoop_version(spark_version, hadoop_version):

[SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2 #34715

[SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2 #34715

Conversation

AngersZhuuuu commented Nov 26, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Nov 26, 2021

AngersZhuuuu commented Nov 26, 2021 • edited

SparkQA commented Nov 26, 2021

SparkQA commented Nov 26, 2021

SparkQA commented Nov 26, 2021

SparkQA commented Nov 26, 2021

SparkQA commented Nov 29, 2021

SparkQA commented Nov 29, 2021

sunchao left a comment

Choose a reason for hiding this comment

HyukjinKwon commented Dec 1, 2021

mridulm commented Dec 1, 2021

sunchao Dec 2, 2021

Choose a reason for hiding this comment

SparkQA commented Dec 5, 2021

SparkQA commented Dec 5, 2021

SparkQA commented Dec 5, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

SparkQA commented Dec 7, 2021

sunchao left a comment

Choose a reason for hiding this comment

HyukjinKwon left a comment • edited

Choose a reason for hiding this comment

HyukjinKwon Dec 8, 2021

Choose a reason for hiding this comment

AngersZhuuuu commented Dec 8, 2021

sunchao commented Dec 8, 2021

HyukjinKwon commented Dec 8, 2021

HyukjinKwon commented Dec 8, 2021

AngersZhuuuu commented Dec 8, 2021

AngersZhuuuu commented Nov 26, 2021 •

edited

AngersZhuuuu commented Nov 26, 2021 •

edited

HyukjinKwon left a comment •

edited