[fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 #2674
[fs] bump Hadoop to 3.4 and migrate to AWS SDK Java v2 #2674xx789633 wants to merge 2 commits intoapache:mainfrom
Conversation
xx789633
force-pushed
the
aws_upgrade
branch
from
c87d05c to
c7955c2
Compare
| - commons-io:commons-io:2.16.1 | ||
| - commons-logging:commons-logging:1.2 | ||
| - io.dropwizard.metrics:metrics-core:3.2.4 | ||
| - io.netty:netty-buffer:4.1.100.Final | ||
| - io.netty:netty-codec:4.1.100.Final | ||
| - io.netty:netty-common:4.1.100.Final | ||
| - io.netty:netty-handler:4.1.100.Final | ||
| - io.netty:netty-resolver:4.1.100.Final | ||
| - io.netty:netty-transport:4.1.100.Final | ||
| - io.netty:netty-transport-classes-epoll:4.1.100.Final | ||
| - io.netty:netty-transport-native-epoll:4.1.100.Final | ||
| - io.netty:netty-transport-native-unix-common:4.1.100.Final | ||
| - org.apache.commons:commons-compress:1.26.1 | ||
| - org.apache.commons:commons-configuration2:2.10.1 | ||
| - org.apache.commons:commons-lang3:3.18.0 | ||
| - org.apache.commons:commons-text:1.4 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 | ||
| - org.apache.hadoop:hadoop-annotations:3.3.4 | ||
| - org.apache.hadoop:hadoop-auth:3.3.4 | ||
| - org.apache.hadoop:hadoop-common:3.3.4 | ||
| - org.apache.kerby:kerb-core:1.0.1 | ||
| - org.apache.kerby:kerby-asn1:1.0.1 | ||
| - org.apache.kerby:kerby-pkix:1.0.1 | ||
| - org.apache.kerby:kerby-util:1.0.1 | ||
| - org.apache.commons:commons-text:1.10.0 | ||
| - org.apache.hadoop:hadoop-annotations:3.4.1 | ||
| - org.apache.hadoop:hadoop-auth:3.4.1 | ||
| - org.apache.hadoop:hadoop-common:3.4.1 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 | ||
| - org.apache.kerby:kerb-core:2.0.3 | ||
| - org.apache.kerby:kerb-crypto:2.0.3 | ||
| - org.apache.kerby:kerb-util:2.0.3 | ||
| - org.apache.kerby:kerby-asn1:2.0.3 | ||
| - org.apache.kerby:kerby-config:2.0.3 | ||
| - org.apache.kerby:kerby-pkix:2.0.3 | ||
| - org.apache.kerby:kerby-util:2.0.3 | ||
| - org.codehaus.jettison:jettison:1.5.4 |
There was a problem hiding this comment.
Are the new dependencies really needed? Especially the netty and kerby dependecies. Netty is a big dependency.
There was a problem hiding this comment.
it seems those dependencies are always included somewhere (like in https://github.com/apache/fluss/blob/main/fluss-filesystems/fluss-fs-azure/src/main/resources/META-INF/NOTICE)
There was a problem hiding this comment.
@xx789633 these new ones are introduced by Hadoop 3.4 right?
If we set them as exclusions, would everything fall apart?
There was a problem hiding this comment.
Hi @wuchong @affo, I have excluded netty dependency.
netty dependency is introduced by Hadoop-common 3.4. I have checked the source code of hadoop-common 3.4 and found it does not use netty at all. The Hadoop community likely declared these dependencies in hadoop-common to unify Netty version management across all sub-modules (such as hadoop-hdfs, hadoop-yarn, and other server-side components), but hadoop-common itself does not actually use them. By the way, netty in Hadoop is mainly used for IPC/RPC, while Fluss only calls standard filesystem calls.
For kerby, those dependencies were there before this pull request. We just upgrade their versions.
| .withCredentials( | ||
| new AWSStaticCredentialsProvider( | ||
| new BasicAWSCredentials(accessKey, secretKey))) | ||
| StsClient stsClient = |
There was a problem hiding this comment.
Seems this will also fix #2661 ? Right?
Could you please help verify it?
Also check minio can be supported or not.
There was a problem hiding this comment.
I don't think so. According to the rustfs discussion at https://github.com/orgs/rustfs/discussions/1746, rustfs currently only support STS AssumeRole action and does not yet support GetSessionToken, which is precisely what our code attempts to invoke (https://github.com/apache/fluss/blob/main/fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/token/S3DelegationTokenProvider.java#L80)
| - commons-io:commons-io:2.16.1 | ||
| - commons-logging:commons-logging:1.2 | ||
| - io.dropwizard.metrics:metrics-core:3.2.4 | ||
| - io.netty:netty-buffer:4.1.100.Final | ||
| - io.netty:netty-codec:4.1.100.Final | ||
| - io.netty:netty-common:4.1.100.Final | ||
| - io.netty:netty-handler:4.1.100.Final | ||
| - io.netty:netty-resolver:4.1.100.Final | ||
| - io.netty:netty-transport:4.1.100.Final | ||
| - io.netty:netty-transport-classes-epoll:4.1.100.Final | ||
| - io.netty:netty-transport-native-epoll:4.1.100.Final | ||
| - io.netty:netty-transport-native-unix-common:4.1.100.Final | ||
| - org.apache.commons:commons-compress:1.26.1 | ||
| - org.apache.commons:commons-configuration2:2.10.1 | ||
| - org.apache.commons:commons-lang3:3.18.0 | ||
| - org.apache.commons:commons-text:1.4 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1 | ||
| - org.apache.hadoop:hadoop-annotations:3.3.4 | ||
| - org.apache.hadoop:hadoop-auth:3.3.4 | ||
| - org.apache.hadoop:hadoop-common:3.3.4 | ||
| - org.apache.kerby:kerb-core:1.0.1 | ||
| - org.apache.kerby:kerby-asn1:1.0.1 | ||
| - org.apache.kerby:kerby-pkix:1.0.1 | ||
| - org.apache.kerby:kerby-util:1.0.1 | ||
| - org.apache.commons:commons-text:1.10.0 | ||
| - org.apache.hadoop:hadoop-annotations:3.4.1 | ||
| - org.apache.hadoop:hadoop-auth:3.4.1 | ||
| - org.apache.hadoop:hadoop-common:3.4.1 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.3.0 | ||
| - org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.3.0 | ||
| - org.apache.kerby:kerb-core:2.0.3 | ||
| - org.apache.kerby:kerb-crypto:2.0.3 | ||
| - org.apache.kerby:kerb-util:2.0.3 | ||
| - org.apache.kerby:kerby-asn1:2.0.3 | ||
| - org.apache.kerby:kerby-config:2.0.3 | ||
| - org.apache.kerby:kerby-pkix:2.0.3 | ||
| - org.apache.kerby:kerby-util:2.0.3 | ||
| - org.codehaus.jettison:jettison:1.5.4 |
There was a problem hiding this comment.
@xx789633 these new ones are introduced by Hadoop 3.4 right?
If we set them as exclusions, would everything fall apart?
4 participants
Purpose
Linked issue: close #1210
Brief change log
Tests
API and Format
Documentation