[SPARK-33104][BUILD] Exclude 'org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests' by HyukjinKwon · Pull Request #30133 · apache/spark

HyukjinKwon · 2020-10-22T15:04:18Z

What changes were proposed in this pull request?

This PR proposes to exclude org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests from hadoop-yarn-server-tests when we use Hadoop 2 profile.

For some reasons, after SBT 1.3 upgrade at SPARK-21708, SBT starts to pull the dependencies of 'hadoop-yarn-server-tests' with 'tests' classifier:

org/apache/hadoop/hadoop-common/2.7.4/hadoop-common-2.7.4-tests.jar
org/apache/hadoop/hadoop-yarn-common/2.7.4/hadoop-yarn-common-2.7.4-tests.jar
org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.4/hadoop-yarn-server-resourcemanager-2.7.4-tests.jar

these were not pulled before the upgrade.

This specific hadoop-yarn-server-resourcemanager-2.7.4-tests.jar causes the problem (SPARK-33104)

When the test case creates the Hadoop configuration here,

spark/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

Line 122 in cc06266

val hadoopConf = SparkHadoopUtil.newConfiguration(conf)
Such jars above have higher precedence in the class path, instead of the specified custom core-site.xml in the test:

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

Line 1375 in e93b8f0

buildPath(Environment.PWD.$$(), LOCALIZED_CONF_DIR, LOCALIZED_HADOOP_CONF_DIR), env)

Later, core-site.xml in the jar is picked instead in Hadoop's Configuration:

Before this fix:

jar:file:/.../https/maven-central.storage-download.googleapis.com/maven2/org/apache/hadoop/
hadoop-yarn-server-resourcemanager/2.7.4/hadoop-yarn-server-resourcemanager-2.7.4-tests.jar!/core-site.xml

After this fix:

file:/.../spark/resource-managers/yarn/target/org.apache.spark.deploy.yarn.YarnClusterSuite/
org.apache.spark.deploy.yarn.YarnClusterSuite-localDir-nm-0_0/
usercache/.../filecache/10/__spark_conf__.zip/__hadoop_conf__/core-site.xml

the core-site.xml in the jar of course does not contain:

spark/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala

Lines 133 to 141 in 2cfd215

    
               val coreSite = """<?xml version="1.0" encoding="UTF-8"?> 
        
                 |<configuration> 
        
                 |  <property> 
        
                 |    <name>spark.test.key</name> 
        
                 |    <value>testvalue</value> 
        
                 |  </property> 
        
                 |</configuration> 
        
                 |""".stripMargin 
        
               Files.write(coreSite, new File(customConf, "core-site.xml"), StandardCharsets.UTF_8)

and the specific test fails.

This PR uses some kind of hacky approach. It was excluded from 'hadoop-yarn-server-tests' with 'tests' classifier, and then added back as a proper dependency (when Hadoop 2 profile is used). In this way, SBT does not pull hadoop-yarn-server-resourcemanager with tests classifier anymore.

Why are the changes needed?

To make the build pass. This is a blocker.

Does this PR introduce any user-facing change?

No, test-only.

How was this patch tested?

Manually tested and debugged:

build/sbt clean "yarn/testOnly *.YarnClusterSuite -- -z SparkHadoopUtil" -Pyarn -Phadoop-2.7 -Phive -Phive-2.3

HyukjinKwon · 2020-10-22T15:04:41Z

cc @srowen, @dongjoon-hyun and @gemelen

HyukjinKwon · 2020-10-22T15:08:09Z

retest this please

HyukjinKwon · 2020-10-22T15:37:29Z

Thanks @srowen. I'll take a look if there are related test failures tomorrow although logically I think there'd not be because these were not pulled in before.

SparkQA · 2020-10-22T15:47:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34771/

dongjoon-hyun · 2020-10-22T15:50:14Z

Thank you so much, @HyukjinKwon !

SparkQA · 2020-10-22T15:55:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34772/

dongjoon-hyun

+1, LGTM (Pending Jenkins).
I verified that this fixes SPARK-33104, too.

$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SparkHadoopUtil" -Pyarn -Phadoop-2.7 -Phive -Phive-2.3
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) (13 seconds, 216 milliseconds)
[info] ScalaTest
[info] Run completed in 31 seconds, 103 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 131 s (02:11), completed Oct 22, 2020 8:55:10 AM

SparkQA · 2020-10-22T16:08:31Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34771/

SparkQA · 2020-10-22T16:24:28Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34772/

gemelen · 2020-10-22T16:40:10Z

This PR proposes to exclude org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests from hadoop-yarn-server-tests.

For some reasons, after SBT 1.3 upgrade at SPARK-21708, SBT starts to pull the dependencies of 'hadoop-yarn-server-tests' with 'tests' classifier:
org/apache/hadoop/hadoop-common/2.7.4/hadoop-common-2.7.4-tests.jar
org/apache/hadoop/hadoop-yarn-common/2.7.4/hadoop-yarn-common-2.7.4-tests.jar
org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.4/hadoop-yarn-server-resourcemanager-2.7.4-tests.jar
these were not pulled before the upgrade.

This sounds exactly as a new behaviour of 'sbt-pom-reader' plugin of latest version. It could be corner case of scope/classifier clash of maven dependency that we are pulling via this plugin.

SparkQA · 2020-10-22T17:11:08Z

Test build #130164 has finished for PR 30133 at commit 4748abd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-10-22T17:14:26Z

Retest this please

SparkQA · 2020-10-22T17:33:15Z

Test build #130165 has finished for PR 30133 at commit 4748abd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-22T18:03:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34776/

SparkQA · 2020-10-22T18:25:54Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34776/

SparkQA · 2020-10-22T19:26:32Z

Test build #130169 has finished for PR 30133 at commit 4748abd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-10-22T20:28:56Z

Retest this please

SparkQA · 2020-10-22T21:22:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34782/

SparkQA · 2020-10-22T21:49:52Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34782/

SparkQA · 2020-10-22T22:40:48Z

Test build #130176 has finished for PR 30133 at commit 4748abd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sunchao · 2020-10-23T00:16:47Z

It seems the test failure in Yarn is related, although I don't know why yet. Also you may need to update dependency files.

HyukjinKwon · 2020-10-23T00:30:34Z

Will take another look for the test failures.

…parkHadoopUtil`

HyukjinKwon · 2020-10-23T05:51:41Z

The GitHub Actions build looked legitimate. I only excluded this explicitly with Hadoop 2 which should be more correct.
Jenkins build looks just flaky. I already have seen all these features regardless of the current fix.

dongjoon-hyun · 2020-10-23T06:01:54Z

Ya. The additional commit looks better. Let's see the result.

SparkQA · 2020-10-23T06:33:36Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34792/

SparkQA · 2020-10-23T07:02:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34792/

viirya · 2020-10-23T07:06:36Z

retest this please

SparkQA · 2020-10-23T07:49:10Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34794/

SparkQA · 2020-10-23T08:17:14Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34794/

SparkQA · 2020-10-23T09:35:21Z

Test build #130193 has finished for PR 30133 at commit 00c8335.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-10-23T10:18:23Z

Looks like all tests passed properly. I am going to merge. Thanks all for taking care of this PR.

Merged to master.

dongjoon-hyun · 2020-10-23T15:36:47Z

Thank you!

HyukjinKwon · 2020-10-24T00:42:55Z

Okay .. it finally starts to pass: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1454/ 👍

dongjoon-hyun · 2020-11-26T00:14:09Z

Sorry, guys.

The original PR seems to exposes Apache Spark to HADOOP-16080 . We may need to revert this together.

[SPARK-33212][BUILD] Move to shaded clients for Hadoop 3.x profile #29843 (comment) (The regression)

We are searching all options.

The Spark-side workaround
New Hadoop release.
Reverting to the old dependency ([SPARK-33618][CORE] Use hadoop-client instead of hadoop-client-api to make hadoop-aws work #30508)

HyukjinKwon · 2020-11-26T00:16:42Z

Sure, thats fine. Thanks for taking care of this.

HyukjinKwon changed the title ~~[SPARK-33104][BUILD] Exclude 'org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests'~~ [SPARK-33104][BUILD][test-hadoop2.7] Exclude 'org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests' Oct 22, 2020

srowen approved these changes Oct 22, 2020

View reviewed changes

dongjoon-hyun approved these changes Oct 22, 2020

View reviewed changes

HyukjinKwon marked this pull request as draft October 23, 2020 00:30

HyukjinKwon added 2 commits October 23, 2020 14:36

Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in S…

3838dfe

…parkHadoopUtil`

Narrow down only to Hadoop 2 profile

00c8335

HyukjinKwon force-pushed the SPARK-33104 branch from 4748abd to 00c8335 Compare October 23, 2020 05:49

HyukjinKwon marked this pull request as ready for review October 23, 2020 05:51

This comment has been minimized.

Sign in to view

HyukjinKwon changed the title ~~[SPARK-33104][BUILD][test-hadoop2.7] Exclude 'org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests'~~ [SPARK-33104][BUILD] Exclude 'org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests' Oct 23, 2020

HyukjinKwon closed this in 10bd42c Oct 23, 2020

HyukjinKwon deleted the SPARK-33104 branch December 7, 2020 02:05

	val coreSite = """<?xml version="1.0" encoding="UTF-8"?>
	\|<configuration>
	\| <property>
	\| <name>spark.test.key</name>
	\| <value>testvalue</value>
	\| </property>
	\|</configuration>
	\|""".stripMargin
	Files.write(coreSite, new File(customConf, "core-site.xml"), StandardCharsets.UTF_8)

Conversation

HyukjinKwon commented Oct 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon commented Oct 22, 2020

Uh oh!

HyukjinKwon commented Oct 22, 2020

Uh oh!

HyukjinKwon commented Oct 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

dongjoon-hyun commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

gemelen commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

dongjoon-hyun commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

dongjoon-hyun commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

SparkQA commented Oct 22, 2020

Uh oh!

sunchao commented Oct 23, 2020

Uh oh!

HyukjinKwon commented Oct 23, 2020

Uh oh!

HyukjinKwon commented Oct 23, 2020

Uh oh!

dongjoon-hyun commented Oct 23, 2020

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

This comment has been minimized.

viirya commented Oct 23, 2020

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

SparkQA commented Oct 23, 2020

Uh oh!

HyukjinKwon commented Oct 23, 2020

Uh oh!

dongjoon-hyun commented Oct 23, 2020

Uh oh!

HyukjinKwon commented Oct 24, 2020

Uh oh!

dongjoon-hyun commented Nov 26, 2020

Uh oh!

HyukjinKwon commented Nov 26, 2020

HyukjinKwon commented Oct 22, 2020 •

edited

Loading

HyukjinKwon commented Oct 22, 2020 •

edited

Loading