Skip to content
Permalink
Browse files
[FLINK-27308][Filesystem][S3] Update the Hadoop implementation for fi…
…lesystems to 3.3.2. This closes #19514

This PR includes making sure that commons-compress is included so that class org.apache.commons.compress.archivers.tar.TarArchiveInputStream is available. This class is required in order to read a TAR archive as an InputStream.

The NOTICE files (where applicable) also include a reference to the Hadoop Thirdparty Shaded dependencies, like we also do for flink-python.

Co-authored-by: Chinmay Sumant <csumant@apple.com>
  • Loading branch information
MartijnVisser and Chinmay Sumant committed Apr 26, 2022
1 parent fe09836 commit c35133c6ec19d027566d1fdda9f6ce7f8c11a2a7
Show file tree
Hide file tree
Showing 9 changed files with 107 additions and 50 deletions.
@@ -7,25 +7,26 @@ The Apache Software Foundation (http://www.apache.org/).
This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.fasterxml.jackson.core:jackson-core:2.13.2
- com.google.errorprone:error_prone_annotations:2.2.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- com.google.guava:guava:20.0
- commons-codec:commons-codec:1.15
- commons-logging:commons-logging:1.1.3
- org.apache.hadoop:hadoop-azure:3.2.2
- org.apache.hadoop:hadoop-azure:3.3.2
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- org.codehaus.jackson:jackson-mapper-asl:1.9.13
- org.codehaus.jackson:jackson-core-asl:1.9.13
- org.eclipse.jetty:jetty-util:9.3.24.v20180605
- org.eclipse.jetty:jetty-util-ajax:9.3.24.v20180605
- org.wildfly.openssl:wildfly-openssl:1.0.7.Final
- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1

This project bundles the following dependencies under the MIT (https://opensource.org/licenses/MIT)

- com.microsoft.azure:azure-keyvault-core:1.0.0
- com.microsoft.azure:azure-storage:7.0.0
- org.checkerframework:checker-qual:2.5.2
- org.codehaus.mojo:animal-sniffer-annotations:1.17
- com.microsoft.azure:azure-storage:7.0.1

The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
org.apache.hadoop.thirdparty:hadoop-shaded-guava dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.google.guava:guava:30.1.1-jre
@@ -153,10 +153,6 @@ under the License.
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
@@ -6,26 +6,32 @@ The Apache Software Foundation (http://www.apache.org/).

This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- org.apache.hadoop:hadoop-annotations:3.2.2
- org.apache.hadoop:hadoop-auth:3.2.2
- org.apache.hadoop:hadoop-common:3.2.2
- org.apache.htrace:htrace-core4:4.1.0-incubating
- org.apache.hadoop:hadoop-annotations:3.3.2
- org.apache.hadoop:hadoop-auth:3.3.2
- org.apache.hadoop:hadoop-common:3.3.2
- org.apache.commons:commons-configuration2:2.1.1
- org.apache.commons:commons-lang3:3.3.2
- org.apache.commons:commons-text:1.4
- org.apache.commons:commons-compress:1.21
- commons-collections:commons-collections:3.2.2
- commons-io:commons-io:2.11.0
- commons-logging:commons-logging:1.1.3
- commons-beanutils:commons-beanutils:1.9.4
- com.google.errorprone:error_prone_annotations:2.2.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- com.fasterxml.jackson.core:jackson-annotations:2.13.2
- com.fasterxml.jackson.core:jackson-core:2.13.2
- com.fasterxml.jackson.core:jackson-databind:2.13.2.2
- com.fasterxml.woodstox:woodstox-core:5.0.3
- com.fasterxml.woodstox:woodstox-core:5.3.0
- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
- org.apache.kerby:kerb-core:1.0.1
- org.apache.kerby:kerby-pkix:1.0.1
- org.apache.kerby:kerby-asn1:1.0.1
- org.apache.kerby:kerby-util:1.0.1
- org.xerial.snappy:snappy-java:1.1.8.3

This project bundles the following dependencies under the MIT (https://opensource.org/licenses/MIT)

@@ -45,14 +51,21 @@ See bundled license files for details.
This project bundles the following dependencies under BSD License (https://opensource.org/licenses/bsd-license.php).
See bundled license files for details.

- org.codehaus.woodstox:stax2-api:3.1.4 (https://github.com/FasterXML/stax2-api/tree/stax2-api-3.1.4)
- org.codehaus.woodstox:stax2-api:4.2.1 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.2.1)

This project bundles the following dependencies under the CDDL 1.1 license.
See bundled license files for details.
The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
org.apache.hadoop.thirdparty:hadoop-shaded-guava dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.google.guava:guava:30.1.1-jre

The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7 dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- javax.activation:javax.activation-api:1.2.0
- com.google.protobuf:protobuf-java:3.7.1

This project bundles org.apache.hadoop:*:3.2.2 from which it inherits the following notices:
This project bundles org.apache.hadoop:*:3.3.2 from which it inherits the following notices:

The Apache Hadoop project contains subcomponents with separate copyright
notices and license terms. Your use of the source code for the these
@@ -13,7 +13,7 @@ This project bundles the following dependencies under the Apache Software Licens
- com.aliyun:aliyun-java-sdk-sts:3.0.0
- commons-codec:commons-codec:1.15
- commons-logging:commons-logging:1.1.3
- org.apache.hadoop:hadoop-aliyun:3.2.2
- org.apache.hadoop:hadoop-aliyun:3.3.2
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- org.codehaus.jettison:jettison:1.1
@@ -155,10 +155,6 @@ under the License.
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
@@ -34,6 +34,9 @@
import org.apache.hadoop.fs.s3a.S3AFileSystem;
import org.apache.hadoop.fs.s3a.S3AUtils;
import org.apache.hadoop.fs.s3a.WriteOperationHelper;
import org.apache.hadoop.fs.s3a.statistics.S3AStatisticsContext;
import org.apache.hadoop.fs.store.audit.AuditSpan;
import org.apache.hadoop.fs.store.audit.AuditSpanSource;

import java.io.File;
import java.io.FileOutputStream;
@@ -52,8 +55,14 @@ public class HadoopS3AccessHelper implements S3AccessHelper {
private final InternalWriteOperationHelper s3accessHelper;

public HadoopS3AccessHelper(S3AFileSystem s3a, Configuration conf) {
checkNotNull(s3a);
this.s3accessHelper =
new InternalWriteOperationHelper(checkNotNull(s3a), checkNotNull(conf));
new InternalWriteOperationHelper(
s3a,
checkNotNull(conf),
s3a.createStoreContext().getInstrumentation(),
s3a.getAuditSpanSource(),
s3a.getActiveAuditSpan());
this.s3a = s3a;
}

@@ -144,8 +153,13 @@ public ObjectMetadata getObjectMetadata(String key) throws IOException {
*/
private static final class InternalWriteOperationHelper extends WriteOperationHelper {

InternalWriteOperationHelper(S3AFileSystem owner, Configuration conf) {
super(owner, conf);
InternalWriteOperationHelper(
S3AFileSystem owner,
Configuration conf,
S3AStatisticsContext statisticsContext,
AuditSpanSource auditSpanSource,
AuditSpan auditSpan) {
super(owner, conf, statisticsContext, auditSpanSource, auditSpan);
}
}
}
@@ -13,8 +13,7 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-core:2.13.2
- com.fasterxml.jackson.core:jackson-databind:2.13.2.2
- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.13.2
- com.fasterxml.woodstox:woodstox-core:5.0.3
- com.google.errorprone:error_prone_annotations:2.2.0
- com.fasterxml.woodstox:woodstox-core:5.3.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
@@ -28,14 +27,22 @@ This project bundles the following dependencies under the Apache Software Licens
- org.apache.commons:commons-configuration2:2.1.1
- org.apache.commons:commons-lang3:3.3.2
- org.apache.commons:commons-text:1.4
- org.apache.hadoop:hadoop-auth:3.2.2
- org.apache.hadoop:hadoop-annotations:3.2.2
- org.apache.hadoop:hadoop-aws:3.2.2
- org.apache.hadoop:hadoop-common:3.2.2
- org.apache.htrace:htrace-core4:4.1.0-incubating
- org.apache.commons:commons-compress:1.21
- org.apache.hadoop:hadoop-auth:3.3.2
- org.apache.hadoop:hadoop-annotations:3.3.2
- org.apache.hadoop:hadoop-aws:3.3.2
- org.apache.hadoop:hadoop-common:3.3.2
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- software.amazon.ion:ion-java:1.0.2
- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
- org.apache.kerby:kerb-core:1.0.1
- org.apache.kerby:kerby-pkix:1.0.1
- org.apache.kerby:kerby-asn1:1.0.1
- org.apache.kerby:kerby-util:1.0.1
- org.xerial.snappy:snappy-java:1.1.8.3
- org.wildfly.openssl:wildfly-openssl:1.0.7.Final

This project bundles the following dependencies under BSD-2 License (https://opensource.org/licenses/BSD-2-Clause).
See bundled license files for details.
@@ -50,7 +57,6 @@ This project bundles the following dependencies under the MIT (https://opensourc
This project bundles the following dependencies under the CDDL 1.1 license.
See bundled license files for details.

- javax.activation:javax.activation-api:1.2.0
- javax.xml.bind:jaxb-api:2.3.1

This project bundles the following dependencies under the Go License (https://golang.org/LICENSE).
@@ -61,4 +67,16 @@ See bundled license files for details.
This project bundles the following dependencies under BSD License (https://opensource.org/licenses/bsd-license.php).
See bundled license files for details.

- org.codehaus.woodstox:stax2-api:3.1.4 (https://github.com/FasterXML/stax2-api/tree/stax2-api-3.1.4)
- org.codehaus.woodstox:stax2-api:4.2.1 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.2.1)

The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
org.apache.hadoop.thirdparty:hadoop-shaded-guava dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.google.guava:guava:30.1.1-jre

The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7 dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.google.protobuf:protobuf-java:3.7.1
@@ -26,7 +26,7 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-core:2.13.2
- com.fasterxml.jackson.core:jackson-databind:2.13.2.2
- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.13.2
- com.fasterxml.woodstox:woodstox-core:5.0.3
- com.fasterxml.woodstox:woodstox-core:5.3.0
- com.google.guava:guava:26.0-jre
- com.google.inject:guice:4.2.2
- com.facebook.airlift:configuration:0.201
@@ -39,16 +39,24 @@ This project bundles the following dependencies under the Apache Software Licens
- org.apache.commons:commons-configuration2:2.1.1
- org.apache.commons:commons-lang3:3.3.2
- org.apache.commons:commons-text:1.4
- org.apache.hadoop:hadoop-annotations:3.2.2
- org.apache.hadoop:hadoop-aws:3.2.2
- org.apache.hadoop:hadoop-auth:3.2.2
- org.apache.hadoop:hadoop-common:3.2.2
- org.apache.htrace:htrace-core4:4.1.0-incubating
- org.apache.commons:commons-compress:1.21
- org.apache.hadoop:hadoop-annotations:3.3.2
- org.apache.hadoop:hadoop-aws:3.3.2
- org.apache.hadoop:hadoop-auth:3.3.2
- org.apache.hadoop:hadoop-common:3.3.2
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- org.apache.hudi:hudi-presto-bundle:0.10.1
- org.weakref:jmxutils:1.19
- software.amazon.ion:ion-java:1.0.2
- org.xerial.snappy:snappy-java:1.1.8.3
- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
- org.apache.kerby:kerb-core:1.0.1
- org.apache.kerby:kerby-pkix:1.0.1
- org.apache.kerby:kerby-asn1:1.0.1
- org.apache.kerby:kerby-util:1.0.1
- org.wildfly.openssl:wildfly-openssl:1.0.7.Final

This project bundles the following dependencies under BSD-2 License (https://opensource.org/licenses/BSD-2-Clause).
See bundled license files for details.
@@ -63,7 +71,6 @@ See bundled license files for details.
This project bundles the following dependencies under the CDDL 1.1 license.
See bundled license files for details.

- javax.activation:javax.activation-api:1.2.0
- javax.xml.bind:jaxb-api:2.3.1

This project bundles the following dependencies under the Go License (https://golang.org/LICENSE).
@@ -74,9 +81,21 @@ See bundled license files for details.
This project bundles the following dependencies under BSD License (https://opensource.org/licenses/bsd-license.php).
See bundled license files for details.

- org.codehaus.woodstox:stax2-api:3.1.4 (https://github.com/FasterXML/stax2-api/tree/stax2-api-3.1.4)
- org.codehaus.woodstox:stax2-api:4.2.1 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.2.1)

This project bundles the following dependencies under the Public Domain.
See bundled license files for details.

- aopalliance:aopalliance:1.0

The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
org.apache.hadoop.thirdparty:hadoop-shaded-guava dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.google.guava:guava:30.1.1-jre

The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7 dependency bundles the following dependencies under
the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.google.protobuf:protobuf-java:3.7.1
@@ -34,7 +34,7 @@ under the License.
<packaging>pom</packaging>

<properties>
<fs.hadoopshaded.version>3.2.2</fs.hadoopshaded.version>
<fs.hadoopshaded.version>3.3.2</fs.hadoopshaded.version>
</properties>

<modules>

0 comments on commit c35133c

Please sign in to comment.