[SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile

[sunchao]

What changes were proposed in this pull request?

This adds hadoop-yarn-server-web-proxy as dependency for Yarn and Hadoop 3.x profile (it is already a dependency for 2.x). Also excludes some dependencies from the module which are already covered by other Hadoop jars used by Spark.

Why are the changes needed?

The class org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter is used by ApplicationMaster:

  private def addAmIpFilter(driver: Option[RpcEndpointRef], proxyBase: String) = {
    val amFilter = "org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter"
    val params = client.getAmIpFilterParams(yarnConf, proxyBase)
    driver match {
      case Some(d) =>
        d.send(AddWebUIFilter(amFilter, params, proxyBase))
   ...

and will be loaded at runtime. Therefore, without the above jar Spark Yarn app will fail with ClassNotFoundError.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing unit tests. Also tested manually and it worked with the fix, while was failing previously.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40039/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40039/

[SparkQA]

Test build #135459 has finished for PR 31642 at commit 17a63bd.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xkrogen]

This seems like a decent workaround, but is it the right long-term fix? It seems like depending on the non-shaded JAR breaks the benefits of leveraging only the shaded hadoop-client-api and hadoop-client-runtime JARs.

Should AmIpFilter be a public class, i.e., is it something AMs are expected to leverage? If yes -- can we get it moved to hadoop-client-runtime? If no -- should we stop using it here and re-implement some of the required logic ourselves?

[sunchao]

This is a small jar though and we've striped all its transitive dependencies. Agree that in the long term we should either: 1) consider to re-implement the logic in Spark which allows us to get away from server-side dependency in Hadoop. cc @tgravescs for more insights on this. 2) move the class to hadoop-client-runtime as you suggested.

I need to take a look at the test failures - seems WebAppProxyServlet was pulled from hadoop-yarn-server-web-proxy jar instead of hadoop-client-minicluster.

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

Thank you for the follow-up, @sunchao !

[sunchao]

This artifact is only needed at runtime so included here in distribution. Ideally we should declare it as a runtime dependency. However, the WebAppProxyServlet class in the jar conflicts with the same one from hadoop-cluster-client-minicluster during tests.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40063/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40063/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40065/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40065/

[dongjoon-hyun]

@sunchao . The PR title looks misleading. Which hadoop profile are you testing now?

[dongjoon-hyun]

I checked the Jenkins log. It's testing Hadoop 3.2 correctly. I'll fix the PR title.

[info] Building Spark using Maven with these arguments:  -Phadoop-3.2 -Phive-2.3 -Pyarn -Phadoop-cloud -Pkinesis-asl -Pkubernetes -Phive-thriftserver -Phive -Pspark-ganglia-lgpl -Pmesos clean package -DskipTests

[dongjoon-hyun]

+1, LGTM (Pending Jenkins)

[sunchao]

Sorry I wanted to test 2.7 and 3.2 at the same time. Is that possible?

[SparkQA]

Test build #135483 has finished for PR 31642 at commit 17a63bd.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

It's not possible for the single title because one Jenkins run only will pick-up a single profile.

What you need is to run two Jenkins runs. So, you need to trigger twice like the following.

1. Set the title with test-hadoop3.2 and trigger.
2. Wait 5 or 10 minutes until Jenkins posts a comment. Then, check the log.
3. Then, rename the title with test-hadoop3.2 and trigger.
4. Wait another 5 or 10 minutes until Jenkins posts a comment. Then, check the log.

Then, you will have effectively concurrent two runs (for both profiles)

[SparkQA]

Test build #135485 has finished for PR 31642 at commit e2d32aa.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sunchao]

Thanks @dongjoon-hyun . Let me trigger another run for Hadoop 2.7 just to be sure.

Jenkins CI says there're 2 failures in SQL module but I'm guessing they're not related (this PR doesn't touch anything in SQL), although I can't seem to find which two tests are these.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40100/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40100/

[SparkQA]

Test build #135519 has finished for PR 31642 at commit 1de8590.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #135525 has finished for PR 31642 at commit e2d32aa.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40113/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40113/

[SparkQA]

Test build #135532 has finished for PR 31642 at commit c31e06d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Could you confirm which links are the two successful runs (Hadoop 3.2/2.7)?

[sunchao]

@dongjoon-hyun Github CI all succeeded (except the flaky tests).

For the Jenkins CI, here are the results:

• 3.2
• 2.7

The tests all passed except two in SQL:

Caused by: java.lang.IllegalArgumentException: Unknown key pii
	at org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider.getCurrentKeyVersion(HadoopShimsPre2_3.java:71)
	at org.apache.orc.impl.WriterImpl.getKey(WriterImpl.java:871)
	at org.apache.orc.impl.WriterImpl.visitTypeTree(WriterImpl.java:902)
	at org.apache.orc.impl.WriterImpl.visitTypeTree(WriterImpl.java:915)
	at org.apache.orc.impl.WriterImpl.setupEncryption(WriterImpl.java:938)
	at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:162)
	at org.apache.orc.OrcFile.createWriter(OrcFile.java:1004)
	at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:47)
	at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:120)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:111)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:270)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:210)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
- Write and read an encrypted file *** FAILED ***
  org.apache.spark.SparkException: Job aborted.
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:232)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)

which doesn't look like related as this PR only touches dependencies for YARN.

Also the above succeeded in Jenkins CI if I remove [test-maven].

[dongjoon-hyun]

Thank you for the summary. For those test cases, I'll take a look in this week, @sunchao .

[dongjoon-hyun]

+1, LGTM. The two failures are not related to this PR.
I'll merge this.

[tgravescs]

yes AmIpFilter is expected to be used by ApplicationMasters so ideally Hadoop would move it to somewhere easily accessible.