Skip to content

[BEAM-1092] Shade commonly used libraries (e.g. Guava) to avoid class conflicts #2096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
aviemzur wants to merge 3 commits into from
Closed

[BEAM-1092] Shade commonly used libraries (e.g. Guava) to avoid class conflicts #2096

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e1fb6f8
[BEAM-1092] Shade commonly used libraries (e.g. Guava) to avoid class…
aviemzur Feb 24, 2017
45edd30
[BEAM-1092] Replace hyphens in artifactId with dots in shading reloca…
aviemzur Mar 14, 2017
7eb6434
[BEAM-1092] Add ServicesResourceTransformer to common shading configu…
aviemzur Mar 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 82 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1310,7 +1310,52 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<version>2.4.3</version>
<executions>
<execution>
<id>bundle-and-repackage</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<includes>
<include>com.google.guava:guava</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<!--suppress MavenModelInspection -->
<shadedPattern>
org.apache.${renderedArtifactId}.repackaged.com.google.common
</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<!--suppress MavenModelInspection -->
<shadedPattern>
org.apache.${renderedArtifactId}.repackaged.com.google.thirdparty
</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
Expand Down Expand Up @@ -1403,6 +1448,32 @@
</filesets>
</configuration>
</plugin>

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>render-artifact-id</id>
<goals>
<goal>regex-properties</goal>
</goals>
<phase>prepare-package</phase>
<configuration>
<regexPropertySettings>
<regexPropertySetting>
<name>renderedArtifactId</name>
<regex>[^A-Za-z0-9]</regex>
<replacement>.</replacement>
<value>${project.artifactId}</value>
<failIfNoMatch>false</failIfNoMatch>
</regexPropertySetting>
</regexPropertySettings>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>

Expand Down Expand Up @@ -1448,7 +1519,7 @@
<version>[1.7,)</version>
</requireJavaVersion>
<requireMavenVersion>
<!-- Keep aligned with prerequisite section below. -->
<!-- Keep aligned with preqrequisite section below. -->
<version>[3.2,)</version>
</requireMavenVersion>
</rules>
Expand All @@ -1463,6 +1534,14 @@
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Copy link
Contributor

@dhalperi dhalperi Feb 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we ensure this only executes in the modules that use guava?

What happens in the modules that need to shade more than guava?

Copy link
Member Author

@aviemzur aviemzur Feb 26, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will execute in every module, to ensure forward compatability, when new modules are introduced, they will not need to know they should shade guava and add this shading to the pom, this will happen automatically. (If they don't depend on guava, no extra classes will be shaded into the resulting jar).

For modules that need to shade more than guava we should ask the following:
Should this artifact be shaded in all modules that will use it?
If so: add it to the root pom
If not: override shading for that specific module.

<artifactId>maven-shade-plugin</artifactId>
</plugin>
</plugins>
</build>

Expand All @@ -1483,6 +1562,7 @@
</plugin>
</plugins>
</reporting>

<prerequisites>
<!-- Keep aligned with requireMavenVersion section above. -->
<maven>3.2</maven>
Expand Down
52 changes: 0 additions & 52 deletions runners/core-construction-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,58 +50,6 @@
</systemPropertyVariables>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>bundle-and-repackage</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadeTestJar>true</shadeTestJar>
<artifactSet>
<includes>
<include>com.google.guava:guava</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<relocations>
<!-- TODO: Once ready, change the following pattern to 'com'
only, exclude 'org.apache.beam.**', and remove
the second relocation. -->
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>
org.apache.beam.runners.core.construction.repackaged.com.google.common
</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>
org.apache.beam.runners.core.construction.repackaged.com.google.thirdparty
</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Expand Down
48 changes: 0 additions & 48 deletions runners/core-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,54 +57,6 @@
<artifactId>maven-jar-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>bundle-and-repackage</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadeTestJar>true</shadeTestJar>
<artifactSet>
<includes>
<include>com.google.guava:guava</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<relocations>
<!-- TODO: Once ready, change the following pattern to 'com'
only, exclude 'org.apache.beam.**', and remove
the second relocation. -->
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>org.apache.beam.runners.core.repackaged.com.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>org.apache.beam.runners.core.repackaged.com.google.thirdparty</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

<!-- Coverage analysis for unit tests. -->
<plugin>
<groupId>org.jacoco</groupId>
Expand Down
52 changes: 0 additions & 52 deletions runners/direct-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,58 +86,6 @@
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>bundle-and-repackage</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadeTestJar>true</shadeTestJar>
<artifactSet>
<includes>
<include>com.google.guava:guava</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<relocations>
<!-- TODO: Once ready, change the following pattern to 'com'
only, exclude 'org.apache.beam.**', and remove
the second relocation. -->
<relocation>
<pattern>com.google.common</pattern>
<excludes>
<!-- com.google.common is too generic, need to exclude guava-testlib -->
<exclude>com.google.common.**.testing.*</exclude>
</excludes>
<shadedPattern>org.apache.beam.runners.direct.repackaged.com.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>org.apache.beam.runners.direct.repackaged.com.google.thirdparty</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

<!-- Coverage analysis for unit tests. -->
<plugin>
<groupId>org.jacoco</groupId>
Expand Down
38 changes: 0 additions & 38 deletions runners/spark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,40 +393,6 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<includes>
<include>com.google.guava:guava</include>
</includes>
</artifactSet>
<relocations>
<!-- relocate Guava used by Beam (v18) since it conflicts with
version used by Hadoop (v11) -->
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>org.apache.beam.spark.repackaged.com.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.thirdparty</pattern>
<shadedPattern>org.apache.beam.spark.repackaged.com.google.thirdparty</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
<plugins>
Expand All @@ -438,10 +404,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
</plugin>
</plugins>
</build>

Expand Down
Loading