HADOOP-18073. Upgrade AWS SDK to v2 in S3A #5980
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ool in ObserverReadProxy (apache#5803)
…chan Yoon. Reviewed-by: Tao Li <tomscut@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
…ache#5822). Contributed by farmmamba. Reviewed-by: zhangshuyan <zqingchai@gmail.com> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
…. Contributed by Shuyan Zhang. Reviewed-by: hfutatzhanghb <1036798979@qq.com> Reviewed-by: Tao Li <tomscut@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…ontributed by Zhaohui Wang. Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
…job configuration web UI (apache#5848)
Contributed by Viraj Jasani
…y Ayush Saxena. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
…bserver reads enabled. (apache#5860). Contributed by Simbarashe Dzinamarira.
…. Contributed by Hualong Zhang. Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Contributed By: Ahmar Suhail <ahmarsu@amazon.co.uk>
…BlockManager processMisReplicatesAsync. (apache#5877). Contributed by Haiyang Hu. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…RouterSafemodeService (apache#5876). Contributed by Haiyang Hu. Reviewed-by: hfutatzhanghb <1036798979@qq.com> Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…ks (apache#5904) Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>
…cyDefault Class (apache#5907) Co-authored-by: huangzhaobo <huangzhaobo99@126.com> Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
…pache#5900). Contributed by Shuyan Zhang. Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…t may cause transitive dependency issue with 2.12.7 (apache#5884)
…che so tokens are updated frequently. (apache#5897) Contributed by Hector Sandoval Chaverri. Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
…pache#5932) Signed-off-by: Tao Li <tomscut@apache.org>
…a block logic when set decrease replication. (apache#5913). Contributed by Haiyang Hu. Reviewed-by: Tao Li <tomscut@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Contributed by Rohit Kumar
Contributed by Anmol Asrani
…rPolicyConfiguration Of Queues. (apache#5862) Contributed by Shilun Fan. Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
…ed by Liangjun He. Reviewed-by: Shilun Fan <slfan1989@apache.org> Reviewed-by: Xing Lin <linxingnku@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…ommitted-allowed. (apache#5933) Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>
…che#5929) Contributed by Bilwa S T.
…Contributed by Shuyan Zhang. Reviewed-by: hfutatzhanghb <1036798979@qq.com> Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…ache#5941). Contributed by Shuyan Zhang. Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Contributed by Viraj Jasani
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>
…istent reads. (apache#5951) Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
…Shilun Fan. Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
apache#5938). Contributed by Shuyan Zhang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…5.2 as it may cause transitive dependency issue with 2.12.7" (apache#5969) This reverts commit 35af8b9.
Contributed by Sotetsu Suzugamine
See aws_sdk_v2_changelog.md for details. Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk> Co-authored-by: Alessandro Passaro <alexpax@amazon.co.uk> HADOOP-18073. Address review comments. (apache#31) addresses review comments + yetus errors Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk> Move MultiObjectDeleteException to impl Reinstate old constants Move TransferManager initialization to ClientFactory Add unit tests for BlockingEnumeration Add unit tests for SelectEventStreamPublisher updates new providers in TestS3AAWSCredentialsProvider to V2 update GET range referrer header logic to V2 adds in unit check for bytes HADOOP-18565. Complete outstanding items for the AWS SDK V2 upgrade. (apache#5421) Changes include * use bundled transfer manager * adds transfer listener to upload * adds support for custom signers * don't set default endpoint * removes v1 sdk bundle, only use core package * implements region caching + many more Note: spotbugs is warning about inconsistent synchronization in accessing a new s3a FS field. This will be fixed in a follow-up patch. Contributed by Ahmar Suhail
This removes the AWS V1 SDK as a hadoop-aws runtime dependency. It is still used at compile time so as to build a wrapper class V1ToV2AwsCredentialProviderAdapter which allows v1 credential provider to be used for authentication. All well known credential providers have their classname remapped from v1 to v2 classes prior to instantiation; this wrapper is not needed for them. There is no support for migrating other SDK plugin points (signing, handlers) Access to the v2 S3Client class used by an S3A FileSystem instance is now via a new interface org.apache.hadoop.fs.s3a.S3AInternals; other low-level operations (getObjectMetadata(Path)) have moved. Contributed by Steve Loughran
Upgrades the AWS sdk v2 version to 2.20.28 This * adds multipart COPY/rename in the java async client * removes the aws-crt JAR dependency Contributed by Ahmar Suhail
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Aggregate PR of commits needed to move hadoop trunk to the AWS v2 SDK.
will merge as a chain of commits, just pushing through yetus as a single large PR to see how it reacts
How was this patch tested?
s3 london -Dprefetch -Dscale
Found one regression, HADOOP-18853, which affects one test and can be fixed later.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?