[SPARK-12807] [YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

[steveloughran]

Patch to

1. Shade jackson 2.x in spark-yarn-shuffle JAR: core, databind, annotation
2. Use maven antrun to verify the JAR has the renamed classes

Being Maven-based, I don't know if the verification phase kicks in on an SBT/jenkins build. It will on a mvn install

[steveloughran]

Updated patch which has shading of com.fasterxml.jackson -> org.spark-project.com.fasterxml.jackson

There's a test for this working; it uses the ant plugin to probe the JAR for .class files. It works, and I've verified that it stops working if you don't have the classes in. This validation runs in the verify phase, immediately before the install action: you can't install this package without the shading working. Accordingly: its a regression test on the feature.

This test does not (obviously) verify that this fixes the conflict between YARN NM and Shuffle. This is going to need external tests

[SparkQA]

Test build #49508 has finished for PR 10780 at commit 94d68a5.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49510 has finished for PR 10780 at commit c365551.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

Jenkins, retest this please

[SparkQA]

Test build #49522 has finished for PR 10780 at commit c365551.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

The latest patch

1. relocates everything except leveldb. Leveldb uses JNI stuff, which doesn't relocate, and on JVMs with multiple classloaders, can still only be loaded once. Jackson, netty and javax.annotation all go under org.spark-project.
2. Added a leveldb-provided profile which excludes leveldb from both the shuffle and hadoop-hdfs dependencies. (The latter could be excluded in the root spark POM; it's server-side only).

When built with leveldb-provided, the relocation stops version conflict in simple java classes; leveldb will be needed to be added to the NM classpath —Hadoop 2.6+(?) certainly does that.

Maybe instead of leveldb-provided, the profile could just become hadoop-2.6, which is already in the top-level

[SparkQA]

Test build #49698 has finished for PR 10780 at commit 1813955.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49736 has finished for PR 10780 at commit 98f18aa.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50675 has finished for PR 10780 at commit 0ae0955.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

@steveloughran given the discussion on the bug, can you remove shading for anything but Jackson?

In particular, shading the javax annotations is wrong because that would probably break tools that rely on them to do checks.

[steveloughran]

Can do

[steveloughran]

I've just pushed out a new version which leaves only jackson and guava as the relocations: the ones that are most trouble. I could turn off the guava relocate, though that introduces a risk of the shuffle failing if the hadoop bundled guava 11.x gets picked up first. The hadoop codebase is designed to work with later guavas, so can cope with the spark one being picked up.

leveldb can be omitted with a Pleveldb-provided profile; this is disabled by default (i.e. leveldb is bundled)

[SparkQA]

Test build #50815 has finished for PR 10780 at commit 43d2b34.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Why not use the shade property you created here too?

[steveloughran]

one is {{org/spark-project}}, the other {{org.spark-project}}; you'd have to do s/r renaming to automatically derive one from the other. With hindsight, it's a common enough use case we should have added it to Ant 15 years ago, but its too late now.

[SparkQA]

Test build #50918 has finished for PR 10780 at commit 947a798.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

super nit: I wonder if this is needed; we have it for Jetty but not for Guava, and both end up being relocated the same way.

[steveloughran]

maven shading is a dark mystery to me. I don't understand what it does; I don't trust it. What you see there is cargo-cult cut and paste backed by a bit of ant to verify the final outcome of the C&P matches what I want.

[vanzin]

LGTM.

[vanzin]

Merging this.

[steveloughran]

thanks!