HADOOP-19181. S3A: IAMCredentialsProvider throttling results in auth failures #8118

steveloughran · 2025-12-03T13:37:28Z

Hard-code a 60s advance time for credential renewal, so even if a large number of processes simultaneously trying to renew their tokens through IAM requests may trigger throttling, the asynchronous refresh thread has 60s of backoff and retry before the existing credentials become invalid.

How was this patch tested?

rerunning store tests, though need to also deploy in ec2 to make sure there are no regressions in normal use.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…failures Hard-code a 60s advance time for credential renewal, so even if a large number of processes simultaneously trying to renew their tokens through IAM requests may trigger throttling, the asynchronous refresh thread has 60s of backoff and retry before the existing credentials become invalid.

ahmarsuhail · 2025-12-03T14:22:20Z

+1, LGTM

wondering if we should add a config to control this though

hadoop-yetus · 2025-12-03T14:40:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 17s		trunk passed
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 18s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 25s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 21s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	0m 47s	/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html	hadoop-tools/hadoop-aws in trunk has 188 extant spotbugs warnings.
+1 💚	shadedclient	14m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 22s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 20s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	0m 48s		the patch passed
+1 💚	shadedclient	14m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 1s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 21s		The patch does not generate ASF License warnings.
		61m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8118/1/artifact/out/Dockerfile
GITHUB PR	#8118
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 4bd191d3a039 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3c12786`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8118/1/testReport/
Max. process+thread count	634 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8118/1/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2025-12-03T14:45:47Z

Do plan to add a test to create an instance of the class, which will either return no credentials or (in EC2), actually work.

Tests run against s3 express

one failure for @ahmarsuhail to worry about.

[ERROR]   ITestS3AAnalyticsAcceleratorStreamReading.testSequentialStreamsNoDuplicateGets:402 [Counter named action_http_get_request with expected value 1] 
Expecting:                                                                                                                                                                                          
 <2L>                                                                                                                                                                                               
to be equal to:                                                                                                                                                                                     
 <1L>

and two failures in assume roles of malformed roles. Looks like STS has changed its error text. fix: remove the probes for specific text

  ITestAssumeRole.testAssumeRoleFSBadPolicy:251->expectFileSystemCreateFailure:164  Expected to find 'JSON' but got unexpected exception: org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on /: software.amazon.awssdk.services.sts.model.MalformedPolicyDocumentException: Unexpected IOException: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: java.io.StringReader@16a63c4d; line: 1, column: 0])                                                                                           
 at [Source: java.io.StringReader@16a63c4d; line: 1, column: 2] (Service: Sts, Status Code: 400, Request ID: 47f1da5d-c400-4269-866a-7324cb167a1a) (SDK Attempt Count: 1):MalformedPolicyDocument: Unexpected IOException: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: java.io.StringReader@16a63c4d; line: 1, column: 0])                                                 
 at [Source: java.io.StringReader@16a63c4d; line: 1, column: 2] (Service: Sts, Status Code: 400, Request ID: 47f1da5d-c400-4269-866a-7324cb167a1a) (SDK Attempt Count: 1)                           
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:271)                                                                                                                  
        at org.apache.hadoop.fs.s3a.S3AUtils.getInstanceFromReflection(S3AUtils.java:705)                                                                                                           
        at org.apache.hadoop.fs.s3a.auth.CredentialProviderListFactory.createAWSV2CredentialProvider(CredentialProviderListFactory.java:303)                                                        
        at org.apache.hadoop.fs.s3a.auth.CredentialProviderListFactory.buildAWSProviderList(CredentialProviderListFactory.java:249)                                                                 
        at org.apache.hadoop.fs.s3a.auth.CredentialProviderListFactory.createAWSCredentialProviderList(CredentialProviderListFactory.java:142)                                                      
        at org.apache.hadoop.fs.s3a.S3AFileSystem.createClientManager(S3AFileSystem.java:1151)                                                                                                      
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:729)                                                                                                                
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3616)                                                                                                                   
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:555)                                                                                                                                 
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:373)                                                                                                                                   
        at org.apache.hadoop.fs.s3a.auth.ITestAssumeRole.lambda$expectFileSystemCreateFailure$0(ITestAssumeRole.java:166)                                                                           
        at org.apache.hadoop.fs.s3a.S3ATestUtils.lambda$interceptClosing$0(S3ATestUtils.java:753)

And here

  ITestAssumeRole.testAssumeRoleFSBadPolicy2:262->expectFileSystemCreateFailure:164  Expected to find 'Syntax errors in policy' but got unexpected exception: org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on /: software.amazon.awssdk.services.sts.model.MalformedPolicyDocumentException: Unexpected IOException: Unexpected character (''' (code 39)): was expecting double-quote to start field name                                                                                                               
 at [Source: java.io.StringReader@21d2e81f; line: 1, column: 3] (Service: Sts, Status Code: 400, Request ID: 7b22250d-1238-4609-92f2-280531aacf11) (SDK Attempt Count: 1):MalformedPolicyDocument: Unexpected IOException: Unexpected character (''' (code 39)): was expecting double-quote to start field name                                                                                         
 at [Source: java.io.StringReader@21d2e81f; line: 1, column: 3] (Service: Sts, Status Code: 400, Request ID: 7b22250d-1238-4609-92f2-280531aacf11) (SDK Attempt Count: 1)

this is all really good for production use: callers are getting errors back from the parser (notable that this change coincides with re-invent). But our tests fail...

…failures * Add a test case to explicitly instantiate the class and ask for credentials doesn't care about whether credentials are returned (EC2 runs) or if NoAwsCredentialsException is returned -any other exception is raised as a failure. * Make UNIT test suite subclass of HadoopTestBase to avoid instantiating s3a fs. HADOOP-19748: S3A: ITestAssumeRole tests failing now STS returns detailed error messages

hadoop-yetus · 2025-12-03T16:13:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 38s		trunk passed
+1 💚	compile	0m 23s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 28s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 19s		trunk passed
+1 💚	mvnsite	0m 32s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	0m 47s	/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html	hadoop-tools/hadoop-aws in trunk has 188 extant spotbugs warnings.
+1 💚	shadedclient	14m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 20s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	0m 48s		the patch passed
+1 💚	shadedclient	14m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 5s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 21s		The patch does not generate ASF License warnings.
		62m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8118/2/artifact/out/Dockerfile
GITHUB PR	#8118
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b5ed430a0142 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `00e3685`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8118/2/testReport/
Max. process+thread count	611 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8118/2/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

github-actions bot added trunk TOOLS AWS labels Dec 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19181. S3A: IAMCredentialsProvider throttling results in auth failures #8118

HADOOP-19181. S3A: IAMCredentialsProvider throttling results in auth failures #8118

steveloughran commented Dec 3, 2025

Uh oh!

ahmarsuhail commented Dec 3, 2025

Uh oh!

hadoop-yetus commented Dec 3, 2025

Uh oh!

steveloughran commented Dec 3, 2025

Uh oh!

hadoop-yetus commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-19181. S3A: IAMCredentialsProvider throttling results in auth failures #8118

Are you sure you want to change the base?

HADOOP-19181. S3A: IAMCredentialsProvider throttling results in auth failures #8118

Conversation

steveloughran commented Dec 3, 2025

How was this patch tested?

For code changes:

Uh oh!

ahmarsuhail commented Dec 3, 2025

Uh oh!

hadoop-yetus commented Dec 3, 2025

Uh oh!

steveloughran commented Dec 3, 2025

Uh oh!

hadoop-yetus commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants