Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-12807] [YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 #10780

Closed
wants to merge 15 commits into from
Closed

[SPARK-12807] [YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 #10780

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
4265afb
SPARK-12807 WiP shading patch + integration tests (after shading) to …
steveloughran Jan 15, 2016
2a876fe
SPARK-12807 add log4j.properties test resource
steveloughran Jan 15, 2016
4221e73
SPARK-12807 IT test to use / paths. Moot, as tests classpaths aren't …
steveloughran Jan 16, 2016
35d478d
SPARK-12807 Switch to using Ant to scan internals of zipfile. Guarant…
steveloughran Jan 16, 2016
a57853b
SPARK-12807 class check taking place in verify phase; works on clean …
steveloughran Jan 16, 2016
25144ec
SPARK-12807 add probe for scala module
steveloughran Jan 16, 2016
586fc79
SPARK-12807 cut explicit references for jackson
steveloughran Jan 16, 2016
32980ea
SPARK-12807 tightened build: reduced diff to master, leave out jackso…
steveloughran Jan 16, 2016
2e694aa
SPARK-12807 macrodefing the build process, for scalability of checks
steveloughran Jan 19, 2016
f16cf13
SPARK-12807 <shaded> extends <exists>, which can be used for probing …
steveloughran Jan 19, 2016
7489f89
SPARK-12807 add leveldb-provided profile, which excludes leveldb from…
steveloughran Jan 19, 2016
f39275d
SPARK-12807 cut leveldb check
steveloughran Jan 19, 2016
81f7304
SPARK-12807 cull (broken) test methods obsoleted by build file checks
steveloughran Jan 20, 2016
e8d4b3a
[SPARK-12807] remove netty & javax.annotation from relocation
steveloughran Feb 5, 2016
947a798
[SPARK-12807] remove all leveldb related work; rely on hadoop and shu…
steveloughran Feb 8, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
49 changes: 48 additions & 1 deletion network/yarn/pom.xml
View file
Open in desktop
Expand Up @@ -35,6 +35,8 @@
<sbt.project.name>network-yarn</sbt.project.name>
<!-- Make sure all Hadoop dependencies are provided to avoid repackaging. -->
<hadoop.deps.scope>provided</hadoop.deps.scope>
<shuffle.jar>${project.build.directory}/scala-${scala.binary.version}/spark-${project.version}-yarn-shuffle.jar</shuffle.jar>
<shade>org/spark-project/</shade>
</properties>

<dependencies>
Expand Down Expand Up @@ -70,7 +72,7 @@
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<outputFile>${project.build.directory}/scala-${scala.binary.version}/spark-${project.version}-yarn-shuffle.jar</outputFile>
<outputFile>${shuffle.jar}</outputFile>
<artifactSet>
<includes>
<include>*:*</include>
Expand All @@ -86,6 +88,15 @@
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>org.spark-project.com.fasterxml.jackson</shadedPattern>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the shade property you created here too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one is {{org/spark-project}}, the other {{org.spark-project}}; you'd have to do s/r renaming to automatically derive one from the other. With hindsight, it's a common enough use case we should have added it to Ant 15 years ago, but its too late now.

<includes>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: I wonder if this is needed; we have it for Jetty but not for Guava, and both end up being relocated the same way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maven shading is a dark mystery to me. I don't understand what it does; I don't trust it. What you see there is cargo-cult cut and paste backed by a bit of ant to verify the final outcome of the C&P matches what I want.

<include>com.fasterxml.jackson.**</include>
</includes>
</relocation>
</relocations>
</configuration>
<executions>
<execution>
Expand All @@ -96,6 +107,42 @@
</execution>
</executions>
</plugin>

<!-- probes to validate that those dependencies which must be shaded are -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<macrodef name="shaded">
<attribute name="resource"/>
<sequential>
<fail message="Not found ${shade}@{resource}">
<condition>
<not>
<resourceexists>
<zipentry zipfile="${shuffle.jar}" name="${shade}@{resource}"/>
</resourceexists>
</not>
</condition>
</fail>
</sequential>
</macrodef>
<echo>Verifying dependency shading</echo>
<shaded resource="com/fasterxml/jackson/core/JsonParser.class" />
<shaded resource="com/fasterxml/jackson/annotation/JacksonAnnotation.class" />
<shaded resource="com/fasterxml/jackson/databind/JsonSerializer.class" />
</target>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>