Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-19082: Update AWS SDK V2 to 2.24.6 #6568

Merged
merged 2 commits into from Mar 5, 2024
Merged

HADOOP-19082: Update AWS SDK V2 to 2.24.6 #6568

merged 2 commits into from Mar 5, 2024

Conversation

HarshitGupta11
Copy link
Contributor

Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module.

How was this patch tested?

Integration Test using mvn clean verify -Dparallel-tests -DtestsThreadCount=16 -Dscale -fae

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@steveloughran
Copy link
Contributor

  • you will need to update LICENSE-binary
  • and run the hadoop-aws tests -where?
  • and run the "qualifying an SDK update" steps -where and what problems surfaced?

HADOOP-19082: Update AWS SDK V2 to 2.24.1
cf11a93 
Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module.
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 44m 50s trunk passed
+1 💚 compile 0m 21s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 21s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 mvnsite 0m 26s trunk passed
+1 💚 javadoc 0m 26s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 23s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 shadedclient 80m 10s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 14s the patch passed
+1 💚 compile 0m 13s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 13s the patch passed
+1 💚 compile 0m 13s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 14s the patch passed
+1 💚 javadoc 0m 13s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 13s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 shadedclient 36m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 17s hadoop-project in the patch passed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
122m 33s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/artifact/out/Dockerfile
GITHUB PR #6568
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname Linux 187f99e9288d 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 6bde82e
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/testReport/
Max. process+thread count 702 (vs. ulimit of 5500)
modules C: hadoop-project U: hadoop-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

+1 pending you state which s3 endpoint you ran the tests and what the parameters to your test run were.

@ahmarsuhail
Copy link
Contributor

ahmarsuhail commented Feb 22, 2024 •
edited

@HarshitGupta11 could you upgrade to the latest version, or at least 2.24.6 instead? We found an issue with the S3 async client not implementing abort which is fixed in 2.24.6. This doesn't really affect us currently, but if someone is using Client side encryption, then it might cause issues. Related to #6164

@steveloughran
Copy link
Contributor

@HarshitGupta11 need to know where you qualified the SDK...

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 1s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 42s Maven dependency ordering for branch
+1 💚 mvninstall 31m 41s trunk passed
+1 💚 compile 17m 4s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 compile 15m 42s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 mvnsite 20m 45s trunk passed
+1 💚 javadoc 8m 37s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 7m 42s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 shadedclient 48m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 1m 7s Maven dependency ordering for patch
+1 💚 mvninstall 29m 25s the patch passed
+1 💚 compile 16m 39s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javac 16m 39s the patch passed
+1 💚 compile 15m 42s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 15m 42s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 15m 29s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 javadoc 8m 26s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 7m 43s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 shadedclient 49m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 733m 15s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 1m 38s The patch does not generate ASF License warnings.
1018m 4s
Reason Tests
Failed junit tests hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.protocol.TestBlockListAsLongs
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/3/artifact/out/Dockerfile
GITHUB PR #6568
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname Linux 8ba8950c227c 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 752b1f3
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/3/testReport/
Max. process+thread count 4275 (vs. ulimit of 5500)
modules C: hadoop-project . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/3/console
versions git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@HarshitGupta11
Copy link
Contributor Author

@HarshitGupta11 could you upgrade to the latest version, or at least 2.24.6 instead? We found an issue with the S3 async client not implementing abort which is fixed in 2.24.6. This doesn't really affect us currently, but if someone is using Client side encryption, then it might cause issues. Related to #6164

Hi yeah, I have updated the ticket and ran the hadoop test suite again a normal and directory bucket in us-west-2

@HarshitGupta11
Copy link
Contributor Author

@HarshitGupta11 need to know where you qualified the SDK...

Still running all the tests as mentioned, I have run the test suite under normal conditions in both kind of buckets but there are a couple of changes needed with CSE-KMS and access point buckets, trying to figure those out.

@steveloughran
Copy link
Contributor

ok, the CSE stuff is going to fail, don't worry

@HarshitGupta11
Copy link
Contributor Author

Ran all the tests as per the Qualifying sdk update, all seems to be fine.

steveloughran
steveloughran approved these changes Mar 5, 2024
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

welcome to the "qualifying an aws release" work

@steveloughran steveloughran changed the title HADOOP-19082: Update AWS SDK V2 to 2.24.1 HADOOP-19082: Update AWS SDK V2 to 2.24.6 Mar 5, 2024
@steveloughran steveloughran merged commit d974a12 into apache:trunk Mar 5, 2024
1 of 4 checks passed
asfgit pushed a commit that referenced this pull request Mar 5, 2024
@HarshitGupta11 @steveloughran
HADOOP-19082: S3A: Update AWS SDK V2 to 2.24.6 (#6568)
397c10a 
Update the AWS SDK to 2.24.6 from 2.23.5 for latest updates in packaging w.r.t. IMDS module.

Contributed by Harshit Gupta
@dongjoon-hyun dongjoon-hyun mentioned this pull request Apr 2, 2024
Closed
dongjoon-hyun added a commit to apache/spark that referenced this pull request Apr 2, 2024
@dongjoon-hyun
[SPARK-47695][BUILD] Upgrade AWS SDK v2 to 2.24.6
e6144e4 
### What changes were proposed in this pull request?

This PR aims to upgrade AWS SDK v2 to 2.24.6.

### Why are the changes needed?

Like HADOOP-19082 (Apache Hadoop 3.4.1), Apache Spark 4.0.0 had better use the latest one.
- apache/hadoop#6568

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #45821 from dongjoon-hyun/SPARK-47695.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steveloughran steveloughran steveloughran approved these changes

Assignees
No one assigned
Labels
build trunk
Projects
None yet
Milestone
No milestone
4 participants
@HarshitGupta11 @steveloughran @hadoop-yetus @ahmarsuhail