Skip to content

[FLINK-22555][build][python] Exclude leftover jboss files #15827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zentol merged 4 commits into from
May 7, 2021
Merged

[FLINK-22555][build][python] Exclude leftover jboss files #15827

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ee87f7c
[FLINK-22560][build] Move generic filters/transformers into general s…
zentol May 5, 2021
c161969
[FLINK-22560][build] Filter maven metadata directory
zentol May 4, 2021
f90acde
[FLINK-22560][build] Add dedicated name to flink-dist shade-plugin ex…
zentol May 5, 2021
2632d0b
[FLINK-22555][build][python] Exclude leftover jboss files
zentol May 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions flink-connectors/flink-connector-cassandra/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,9 @@ under the License.
<goals>
<goal>shade</goal>
</goals>
<configuration combine.self="override">
Copy link
Contributor Author

@zentol zentol May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this removal is safe because it the override was plain unnecessary, and we get rid of an odd special case.

<dependencyReducedPomLocation>${project.basedir}/target/dependency-reduced-pom.xml</dependencyReducedPomLocation>
<configuration>
<!-- This is necessary because we bundle a subset of our dependencies,
and transitive dependencies of that subset should still be pulled into the user jar.-->
<promoteTransitiveDependencies>true</promoteTransitiveDependencies>
<artifactSet>
<includes>
Expand Down
13 changes: 0 additions & 13 deletions flink-connectors/flink-connector-elasticsearch5/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,19 +245,6 @@ under the License.
<exclude>log4j.properties</exclude>
<exclude>config/favicon.ico</exclude>
<exclude>mozilla/**</exclude>
<exclude>META-INF/maven/com*/**</exclude>
<exclude>META-INF/maven/io*/**</exclude>
<exclude>META-INF/maven/joda*/**</exclude>
<exclude>META-INF/maven/net*/**</exclude>
<exclude>META-INF/maven/org.an*/**</exclude>
<exclude>META-INF/maven/org.apache.h*/**</exclude>
<exclude>META-INF/maven/org.apache.commons/**</exclude>
<exclude>META-INF/maven/org.apache.flink/force-shading/**</exclude>
<exclude>META-INF/maven/org.apache.logging*/**</exclude>
<exclude>META-INF/maven/org.e*/**</exclude>
<exclude>META-INF/maven/org.h*/**</exclude>
<exclude>META-INF/maven/org.j*/**</exclude>
<exclude>META-INF/maven/org.y*/**</exclude>
</excludes>
</filter>
<filter>
Expand Down
4 changes: 4 additions & 0 deletions flink-connectors/flink-connector-kafka/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,10 @@ under the License.
<goal>test-jar-no-fork</goal>
</goals>
<configuration>
<archive>
<!-- Globally exclude maven metadata, because it may accidentally bundle files we don't intend to -->
<addMavenDescriptor>false</addMavenDescriptor>
Copy link
Contributor Author

@zentol zentol May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally we move this into the pluginManagement section in the root pom, but I already spent too much time on this ticket (and actually pin the plugin version...)

</archive>
<includes>
<include>**/KafkaTestEnvironmentImpl*</include>
<include>META-INF/LICENSE</include>
Expand Down
34 changes: 3 additions & 31 deletions flink-dist/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -675,11 +675,12 @@ under the License.
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>shade-dist</id>
Copy link
Contributor Author

@zentol zentol May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially just a safeguard against accidentally inheriting something from somewhere.

<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration combine.self="override">
Copy link
Contributor Author

@zentol zentol May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this removal is safe because it is a distinct execution, and hence never had any effect.

<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<shadedArtifactAttached>false</shadedArtifactAttached>
<finalName>${project.artifactId}-${project.version}</finalName>
Expand All @@ -688,11 +689,6 @@ under the License.
<filter>
<artifact>*</artifact>
<excludes>
<exclude>log4j.properties</exclude>
<exclude>log4j-test.properties</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>org/apache/flink/runtime/util/bash/BashJavaUtils.class</exclude>
</excludes>
</filter>
Expand All @@ -709,12 +705,6 @@ under the License.
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<!-- The service transformer is needed to merge META-INF/services files -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
<projectName>Apache Flink</projectName>
<encoding>UTF-8</encoding>
</transformer>
</transformers>
</configuration>
</execution>
Expand All @@ -734,22 +724,11 @@ under the License.
<goals>
<goal>shade</goal>
</goals>
<configuration combine.self="override">
Copy link
Contributor Author

@zentol zentol May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this removal is safe because it is a distinct execution, and hence never had any effect.

<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<shadedArtifactAttached>false</shadedArtifactAttached>
<finalName>bash-java-utils</finalName>
<filters>
<!-- Globally exclude log4j.properties from our JAR files. -->
<filter>
<artifact>*</artifact>
<excludes>
<exclude>log4j.properties</exclude>
<exclude>log4j-test.properties</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
<!-- Include only the BashJavaUtils, other required classes should come from the flink-dist-->
<filter>
<artifact>org.apache.flink:*</artifact>
Expand All @@ -770,13 +749,6 @@ under the License.
<resource>log4j2.properties</resource>
<file>src/main/resources/log4j-bash-utils.properties</file>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
<projectName>Apache Flink</projectName>
<encoding>UTF-8</encoding>
</transformer>
</transformers>
</configuration>
</execution>
Expand Down
1 change: 0 additions & 1 deletion flink-filesystems/flink-azure-fs-hadoop/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,6 @@ under the License.
<exclude>properties.dtd</exclude>
<exclude>PropertyList-1.0.dtd</exclude>
<exclude>mozilla/**</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
Expand Down
1 change: 0 additions & 1 deletion flink-filesystems/flink-fs-hadoop-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,6 @@ under the License.
<excludes>
<exclude>properties.dtd</exclude>
<exclude>PropertyList-1.0.dtd</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>META-INF/services/javax.xml.stream.*</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
</excludes>
Expand Down
1 change: 0 additions & 1 deletion flink-filesystems/flink-oss-fs-hadoop/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,6 @@ under the License.
<exclude>.gitkeep</exclude>
<exclude>mime.types</exclude>
<exclude>mozilla/**</exclude>
<exclude>META-INF/maven/**</exclude>
</excludes>
</filter>
</filters>
Expand Down
7 changes: 0 additions & 7 deletions flink-filesystems/flink-s3-fs-hadoop/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,12 +203,6 @@ under the License.
</relocation>
</relocations>
<filters>
<filter>
<artifact>*</artifact>
<excludes>
<exclude>META-INF/maven/org.apache.flink/force-shading/**</exclude>
</excludes>
</filter>
<filter>
<artifact>*</artifact>
<excludes>
Expand All @@ -224,7 +218,6 @@ under the License.
<excludes>
<exclude>properties.dtd</exclude>
<exclude>PropertyList-1.0.dtd</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>META-INF/services/javax.xml.stream.*</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
</excludes>
Expand Down
7 changes: 0 additions & 7 deletions flink-filesystems/flink-s3-fs-presto/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -401,19 +401,12 @@ under the License.
<exclude>.gitkeep</exclude>
<exclude>mime.types</exclude>
<exclude>mozilla/**</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
</excludes>
</filter>
<filter>
<artifact>*</artifact>
<excludes>
<exclude>META-INF/maven/org.weakref/**</exclude>
<exclude>META-INF/maven/org.hdrhistogram/**</exclude>
<exclude>META-INF/maven/joda-time/**</exclude>
<exclude>META-INF/maven/io.airlift/**</exclude>
<exclude>META-INF/maven/com*/**</exclude>
<exclude>META-INF/maven/org.apache.flink/force-shading/**</exclude>
<exclude>META-INF/LICENSE.txt</exclude>
</excludes>
</filter>
Expand Down
3 changes: 1 addition & 2 deletions flink-kubernetes/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ under the License.
<goals>
<goal>shade</goal>
</goals>
<configuration combine.children="append">
Copy link
Contributor Author

@zentol zentol May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

afaik this doesn't make sense and just results in any parent configurations being ignored. Similar to cassandra it is unnecessary because we aren't doing anything special, and we get rid of an odd special case.

<configuration>
<artifactSet>
<includes combine.children="append">
<include>io.fabric8:kubernetes-client</include>
Expand All @@ -156,7 +156,6 @@ under the License.
<artifact>*:*</artifact>
<excludes>
<exclude>*.aut</exclude>
<exclude>META-INF/maven/**</exclude>
<exclude>META-INF/services/*com.fasterxml*</exclude>
<exclude>META-INF/proguard/**</exclude>
<exclude>OSGI-INF/**</exclude>
Expand Down
4 changes: 4 additions & 0 deletions flink-libraries/flink-gelly-scala/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,10 @@ under the License.
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<!-- Globally exclude maven metadata, because it may accidentally bundle files we don't intend to -->
<addMavenDescriptor>false</addMavenDescriptor>
</archive>
</configuration>
<executions>
<execution>
Expand Down
2 changes: 1 addition & 1 deletion flink-python/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,6 @@ under the License.
<artifact>org.apache.beam:beam-sdks-java-core</artifact>
<excludes>
<exclude>org/apache/beam/repackaged/core/org/antlr/**</exclude>
<exclude>META-INF/maven/org.antlr/**</exclude>
<exclude>org/apache/beam/repackaged/core/org/apache/commons/compress/**</exclude>
<exclude>org/apache/beam/repackaged/core/org/apache/commons/lang3/**</exclude>
</excludes>
Expand All @@ -357,6 +356,7 @@ under the License.
<artifact>org.apache.beam:beam-vendor-grpc-1_26_0</artifact>
<excludes>
<exclude>org/apache/beam/vendor/grpc/v1p26p0/org/jboss/**</exclude>
<exclude>schema/**</exclude>
<exclude>org/apache/beam/vendor/grpc/v1p26p0/org/eclipse/jetty/**</exclude>
</excludes>
</filter>
Expand Down
6 changes: 0 additions & 6 deletions flink-runtime/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -572,12 +572,6 @@ under the License.
copied into this modules's NOTICE file. -->
<exclude>META-INF/NOTICE.txt</exclude>
</excludes>
</filter>
<filter>
<artifact>*</artifact>
<excludes>
<exclude>META-INF/maven/io.netty/**</exclude>
</excludes>
</filter>
</filters>
</configuration>
Expand Down
1 change: 0 additions & 1 deletion flink-test-utils-parent/flink-test-utils/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,6 @@ under the License.
<filter>
<artifact>io.netty:netty</artifact>
<excludes>
<exclude>META-INF/maven/io.netty/**</exclude>
<!-- Only some of these licenses actually apply to the JAR and have been manually
placed in this module's resources directory. -->
<exclude>META-INF/license/**</exclude>
Expand Down
65 changes: 46 additions & 19 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1366,6 +1366,8 @@ under the License.
<version>2.4</version><!--$NO-MVN-MAN-VER$-->
<configuration>
<archive>
<!-- Globally exclude maven metadata, because it may accidentally bundle files we don't intend to -->
<addMavenDescriptor>false</addMavenDescriptor>
<manifest>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
Expand Down Expand Up @@ -1726,6 +1728,50 @@ under the License.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<!-- This section contains the core configuration that is applied to every jar that we create.-->
<filters combine.children="append">
<filter>
<artifact>*</artifact>
<excludes>
<!-- Globally exclude log4j.properties from our JAR files. -->
<exclude>log4j.properties</exclude>
<exclude>log4j2.properties</exclude>
<exclude>log4j-test.properties</exclude>
<exclude>log4j2-test.properties</exclude>
<!-- Do not copy the signatures in the META-INF folder.
Otherwise, this might cause SecurityExceptions when using the JAR. -->
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<!-- META-INF/maven can contain 2 things:
- For archetypes, it contains an archetype-metadata.xml.
- For other jars, it contains the pom for all dependencies under the respective <groupId>/<artifactId>/ directory.

We want to exclude the poms because they may be under an incompatible license,
however the archetype metadata is required and we need to keep that around.

This pattern excludes directories under META-INF/maven.
('?*/**' does not work because '**' also matches zero directories;
everything that matches '?*' also matches '?*/**')

The initial '**' allows the pattern to also work for multi-release jars that may contain such entries under
'META-INF/versions/11/META-INF/maven/'.
-->
<exclude>**/META-INF/maven/?*/?*/**</exclude>
Copy link
Contributor

@tillrohrmann tillrohrmann May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it work to explicitly add the archetype-metadata.xml in the quickstart as an inclusion?

Copy link
Contributor Author

@zentol zentol May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the same idea, but don't think it would work because exclusions are used to narrow down inclusions.
The default inclusion is *, matching all files and we exclude some of those..
If set an inclusion for archetype-metadata.xml, then all other files are implicitly excluded. We'd need include for everything we truly want in the jar, but that seems like more overhead than the current approach.

Copy link
Contributor

@tillrohrmann tillrohrmann May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok. Then let's keep the current solution.

</excludes>
</filter>
</filters>
<transformers combine.children="append">
<!-- The service transformer is needed to merge META-INF/services files -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<!-- The ApacheNoticeResourceTransformer collects and aggregates NOTICE files -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
<projectName>Apache Flink</projectName>
<encoding>UTF-8</encoding>
</transformer>
</transformers>
</configuration>
<executions>
<execution>
<id>shade-flink</id>
Expand All @@ -1740,16 +1786,6 @@ under the License.
<dependencyReducedPomLocation>${project.basedir}/target/dependency-reduced-pom.xml</dependencyReducedPomLocation>
<!-- Filters MUST be appended; merging filters does not work properly, see MSHADE-305 -->
<filters combine.children="append">
<!-- Globally exclude log4j.properties from our JAR files. -->
<filter>
<artifact>*</artifact>
<excludes>
<exclude>log4j.properties</exclude>
<exclude>log4j2.properties</exclude>
<exclude>log4j-test.properties</exclude>
<exclude>log4j2-test.properties</exclude>
</excludes>
</filter>
<!-- drop entries into META-INF and NOTICE files for the dummy artifact -->
<filter>
<artifact>org.apache.flink:force-shading</artifact>
Expand All @@ -1776,15 +1812,6 @@ under the License.
<include>org.apache.flink:force-shading</include>
</includes>
</artifactSet>
<transformers combine.children="append">
<!-- The service transformer is needed to merge META-INF/services files -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<!-- The ApacheNoticeResourceTransformer collects and aggregates NOTICE files -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
<projectName>Apache Flink</projectName>
<encoding>UTF-8</encoding>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
Expand Down