[FLINK-7973] fix shading and relocating Hadoop for the S3 filesystems #4961
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- do not shade everything, especially not JDK classes! -> instead define include patterns explicitly - do not shade core Flink classes (only those imported from flink-hadoop-fs) - hack around Hadoop loading (unshaded/non-relocated) classes based on names in the core-default.xml by overwriting the Configuration class (we may need to extend this for the mapred-default.xml and hdfs-defaults.xml): -> provide a core-default-shaded.xml file with shaded class names and copy and adapt the Configuration class of the respective Hadoop version to load this file instead of core-default.xml.
NicoK
changed the title
zentol
reviewed
Nov 7, 2017
| @@ -33,6 +33,7 @@ under the License. | |||
| <packaging>jar</packaging> | |||
|
|
|||
| <properties> | |||
| <!-- Do not change this without updating the copied Configuration class! --> | |||
There was a problem hiding this comment.
should we add a similar comment to the presto hadoop dependency?
There was a problem hiding this comment.
done (also the other suggestions)
| @@ -277,6 +336,7 @@ under the License. | |||
| <exclude>META-INF/maven/org.apache.commons/**</exclude> | |||
| <exclude>META-INF/maven/org.apache.flink/flink-hadoop-fs/**</exclude> | |||
| <exclude>META-INF/maven/org.apache.flink/force-shading/**</exclude> | |||
| <exclude>core-default.xml</exclude> | |||
There was a problem hiding this comment.
add comment that we're using our own shaded core-default.xml
| @@ -284,6 +322,7 @@ under the License. | |||
| <exclude>META-INF/maven/org.apache.h*/**</exclude> | |||
| <exclude>META-INF/maven/org.apache.flink/flink-hadoop-fs/**</exclude> | |||
| <exclude>META-INF/maven/org.apache.flink/force-shading/**</exclude> | |||
| <exclude>core-default.xml</exclude> | |||
- also fix the (integration) tests not working because they tried to load the relocated classes which are apparently not available there
zentol
approved these changes
Nov 8, 2017
| - `src/test/resources/core-site.xml` (as is) | ||
| 3. verify the shaded jar: | ||
| - does not contain any unshaded classes except for `org.apache.flink.fs.s3presto.S3FileSystemFactory` | ||
| - every other classes should be under `org.apache.flink.fs.s3presto.shaded` |
There was a problem hiding this comment.
"every other class" or "all other classes"
| - `src/test/resources/core-site.xml` (as is) | ||
| 3. verify the shaded jar: | ||
| - does not contain any unshaded classes except for `org.apache.flink.fs.s3hadoop.S3FileSystemFactory` | ||
| - every other classes should be under `org.apache.flink.fs.s3hadoop.shaded` |
There was a problem hiding this comment.
"every other class" or "all other classes"
aljoscha
reviewed
Nov 8, 2017
| setup. For this to work, however, we needed to adapt Hadoop's `Configuration` | ||
| class to load a (shaded) `core-default-shaded.xml` configuration with the | ||
| relocated class names of classes loaded via reflection | ||
| (in the fute, we may need to extend this to `mapred-default.xml` and `hdfs-defaults.xml` and their respective configuration classes). |
aljoscha
reviewed
Nov 8, 2017
| - `src/test/resources/core-site.xml` (as is) | ||
| 3. verify the shaded jar: | ||
| - does not contain any unshaded classes except for `org.apache.flink.fs.s3hadoop.S3FileSystemFactory` | ||
| - every other classes should be under `org.apache.flink.fs.s3hadoop.shaded` |
aljoscha
reviewed
Nov 8, 2017
| setup. For this to work, however, we needed to adapt Hadoop's `Configuration` | ||
| class to load a (shaded) `core-default-shaded.xml` configuration with the | ||
| relocated class names of classes loaded via reflection | ||
| (in the fute, we may need to extend this to `mapred-default.xml` and `hdfs-defaults.xml` and their respective configuration classes). |
aljoscha
reviewed
Nov 8, 2017
| - `src/test/resources/core-site.xml` (as is) | ||
| 3. verify the shaded jar: | ||
| - does not contain any unshaded classes except for `org.apache.flink.fs.s3presto.S3FileSystemFactory` | ||
| - every other classes should be under `org.apache.flink.fs.s3presto.shaded` |
|
I think this looks good now that the comments are addressed. @zentol what do you think? |
|
|
|
I added two end-to-end tests and it seems for presto this currently fails with: This is my branch with the end-to-end tests: https://github.com/aljoscha/flink/tree/finish-pr-4961-s3-shading-fixing |
This was causing "java.lang.ClassNotFoundException: org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.impl.LogFactoryImpl" since these classes are not statically imported and thus removed when minimizing.
|
unfortunately, it seems we cannot minimize the shaded |
This uses traps to ensure that we properly do cleanups, remove config values and shutdown things.
|
I did include your end-to-end tests (with some fixes) and the fixes for the errors they found. Should be fine now, let's see what travis says... |
Also improve some search patterns in general.
|
This look even better now! Waiting for Travis and then I'll merge. Thanks! 😃 |
|
Could you please close the PR if it doesn't auto-close? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
The current shading of the
flink-s3-fs-hadoopandflink-s3-fs-prestoprojects also relocates Flink core classes and even some from the JDK itself. Additionally, the relocation of Hadoop does not work as expected since Hadoop loads classes based on class names in itscore-default.xmlwhich are unshaded and thus use the original namespace.Brief change log
pom.xmlof bothflink-s3-fs-hadoopandflink-s3-fs-presto:core-default.xmlby overwriting theConfigurationclass (we may need to also extend this for themapred-default.xmlandhdfs-defaults.xmland their respective configuration classes in the future):core-default-shaded.xmlfile with shaded class names andConfigurationclass of the respective Hadoop version to load this file instead ofcore-default.xml(Thanks @zentol and @StephanEwen for helping to find and fix this.)
Verifying this change
This change can (and was) manually tested as follows:
jarfile does not contain non-relocated classesConfigurationclasses reside in the shaded namespace where the original HadoopConfigurationclasses would go into, e.g.org.apache.flink.fs.s3hadoop.shaded.org.hadoop.conf(look forcore-default-shaded.xmlstring in theConfiguration.classfile)META-INF/servicesfiles are still correct (name + content)Does this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation