Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-13748][S3][build] Fix jaxb relocation for S3. #9695

Closed
wants to merge 1 commit into from
Closed

[FLINK-13748][S3][build] Fix jaxb relocation for S3. #9695

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
62 changes: 57 additions & 5 deletions flink-filesystems/flink-s3-fs-hadoop/pom.xml
View file
Open in desktop
Expand Up @@ -106,11 +106,6 @@ under the License.
<pattern>com.amazon</pattern>
<shadedPattern>org.apache.flink.fs.s3base.shaded.com.amazon</shadedPattern>
</relocation>
<!-- relocated S3 hadoop dependencies -->
<relocation>
<pattern>javax.xml.bind</pattern>
<shadedPattern>org.apache.flink.fs.s3hadoop.shaded.javax.xml.bind</shadedPattern>
</relocation>
<!-- shade Flink's Hadoop FS utility classes -->
<relocation>
<pattern>org.apache.flink.runtime.util</pattern>
Expand Down Expand Up @@ -146,6 +141,63 @@ under the License.
<version>2.3.0</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>shade-flink</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadeTestJar>false</shadeTestJar>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<relocations>
<!-- relocate the references to Hadoop to match the shaded Hadoop config -->
<relocation>
<pattern>org.apache.hadoop</pattern>
<shadedPattern>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop</shadedPattern>
</relocation>
<!-- relocate the AWS dependencies -->
<relocation>
<pattern>com.amazon</pattern>
<shadedPattern>org.apache.flink.fs.s3base.shaded.com.amazon</shadedPattern>
</relocation>

<!-- relocated S3 hadoop dependencies -->
<relocation>
<pattern>javax.xml.bind</pattern>
<shadedPattern>org.apache.flink.fs.s3hadoop.shaded.javax.xml.bind</shadedPattern>
</relocation>
<!-- shade Flink's Hadoop FS utility classes -->
<relocation>
<pattern>org.apache.flink.runtime.util</pattern>
<shadedPattern>org.apache.flink.fs.s3hadoop.common</shadedPattern>
</relocation>
</relocations>
<filters>
<filter>
<artifact>*</artifact>
<excludes>
<exclude>META-INF/maven/org.apache.flink/force-shading/**</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding this rather complicated shade snippet, why don't we always add the jaxb-api dependency via

<dependency>
    <groupId>javax.xml.bind</groupId>
	<artifactId>jaxb-api</artifactId>
	<version>2.3.0</version>
</dependency>

</profile>
</profiles>

Expand Down
128 changes: 124 additions & 4 deletions flink-filesystems/flink-s3-fs-presto/pom.xml
View file
Open in desktop
Expand Up @@ -270,10 +270,6 @@ under the License.
<pattern>io.airlift</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.io.airlift</shadedPattern>
</relocation>
<relocation>
<pattern>javax.xml.bind</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.javax.xml.bind</shadedPattern>
</relocation>
<relocation>
<pattern>org.HdrHistogram</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.org.HdrHistogram</shadedPattern>
Expand Down Expand Up @@ -342,6 +338,130 @@ under the License.
<version>2.3.0</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<executions>
<execution>
<id>ban-openjdk.jol</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<bannedDependencies>
<excludes>
<!-- Incompatible license -->
<exclude>org.openjdk.jol:*</exclude>
</excludes>
</bannedDependencies>
</rules>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>shade-flink</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadeTestJar>false</shadeTestJar>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<relocations>
<!-- relocate the references to Hadoop to match the pre-shaded hadoop artifact -->
<relocation>
<pattern>org.apache.hadoop</pattern>
<shadedPattern>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop</shadedPattern>
</relocation>
<!-- relocate the AWS dependencies -->
<relocation>
<pattern>com.amazon</pattern>
<shadedPattern>org.apache.flink.fs.s3base.shaded.com.amazon</shadedPattern>
</relocation>
<!-- relocate S3 presto and dependencies -->
<relocation>
<pattern>com.facebook</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.com.facebook</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.com.fasterxml</shadedPattern>
</relocation>
<relocation>
<pattern>io.airlift</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.io.airlift</shadedPattern>
</relocation>
<relocation>
<pattern>javax.xml.bind</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.javax.xml.bind</shadedPattern>
</relocation>
<relocation>
<pattern>org.HdrHistogram</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.org.HdrHistogram</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.org.joda</shadedPattern>
</relocation>
<relocation>
<pattern>org.weakref</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.org.weakref</shadedPattern>
</relocation>
<relocation>
<pattern>org.openjdk</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.org.openjdk</shadedPattern>
</relocation>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.shaded.com.google</shadedPattern>
</relocation>

<!-- shade Flink's Hadoop FS utility classes -->
<relocation>
<pattern>org.apache.flink.runtime.util</pattern>
<shadedPattern>org.apache.flink.fs.s3presto.common</shadedPattern>
</relocation>
</relocations>
<filters>
<filter>
<artifact>*</artifact>
<excludes>
<exclude>META-INF/maven/org.weakref/**</exclude>
<exclude>META-INF/maven/org.hdrhistogram/**</exclude>
<exclude>META-INF/maven/joda-time/**</exclude>
<exclude>META-INF/maven/io.airlift/**</exclude>
<exclude>META-INF/maven/com*/**</exclude>
<exclude>META-INF/maven/org.apache.flink/force-shading/**</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
</excludes>
</filter>
<filter>
<artifact>com.facebook.presto.hadoop:hadoop-apache2</artifact>
<includes>
<include>com/facebook/presto/hadoop/**</include>
</includes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. I think maintaining this shading logic can be quite a pain in the ass.

</profile>
</profiles>

Expand Down