role_arn and source_profile in ~/.aws/config not supported when using DefaultAWSCredentialsProviderChain

[nerdyness]

Describe the bug

I have the following snippet in my ~/.aws/config file and am using the DefaultAWSCredentialsProviderChain. When I use AWS_PROFILE=role the Java SDK doesn't find my credentials yet the AWS CLI works just fine. This is true whether I use source_profile = creds or source_profile = creds_p.

Either of the "credential" profiles (creds or creds_p) on their own do work for the Java SDK and the AWS CLI, e.g.: AWS_PROFILE=creds. The chaining that is needed for role_arn to work (via source_profile) is not supported in the Java SDK.

[profile role]
role_arn = arn:aws:iam::123456789112:role/credtest
source_profile = creds

[profile creds]
aws_access_key_id = AKIA3Bxxx
aws_secret_access_key = v3yZ+xxx

[profile creds_p]
credential_process = /Users/my-user/.aws/creds.sh --username my-user

Expected Behavior

I'd expect the Java SDK to pick up the credentials as defined in the ~/.aws/config file.

Current Behavior

As soon as I combine source_profile and one of the other authentication mechanisms, the Java SDK can't find my credentials any more yet the AWS CLI works as expected.

Steps to Reproduce

1. Create a sample app that uses the aws-sdk-java v1
2. Configure your ~/.aws/config with two profiles, where one uses credentials and the other uses role_arn and source_profile as illustrated above.
3. Set your shell to use the role_arn profile and try something like aws s3 ls to verify your credentials are working.
4. Try to run the app and you will find it can't find your credentials.

Context

This is a huge problem for my customer as their business runs on Java and they frequently fire up their microservices locally with credentials into AWS to get access to real life resources.

They were just acquired by another company and need to switch to this new way of authenticating. An upgrade to aws-sdk-v2 is not straight forward and we can't negotiate on authentication methods as the rest of the company that acquired us works like that (but using a non-Java SDK).

Your Environment

• AWS Java SDK version used: 1.11.837 & 2.15.15
• JDK version used: OpenJDK Runtime Environment Corretto-16.0.1.9.1 (build 16.0.1+9)
• Operating System and version: MacOS Big Sur 11.4

[debora-ito]

Hi @nerdyness thank you for reaching out.

I think the issue is related to #803, and if it is that thread provides some workarounds. I'll work to repro your use case.

[nerdyness]

Hi @debora-ito,
Thank you for getting back to me. I can see how both issues are related to the way the Java SDK handles AWS credentials in ~/.aws/config & ~/.aws/credentials, but the issues are separate and the proposed work-arounds won't work in this case.

#803 talks about the way credentials need to be double-defined with and without the profile prefix and how the SDK is incompatible in v1 of the SDK but now fixed in v2.

This bug (#2602) is talking about the source_profile directive not working, which is an issue in both v1 and v2 of the Java SDK. I've tested my credentials file using

• the AWS CLI
• the Ruby SDK
• and the Python SDK

and it works for all 3 but NOT the Java SDK, v1 or v2.

Right now, my customer, who runs on well over 50 Java microservices would have to change their code in every microservice they run just because the way they source credentials in their dev environment has changed. This is clearly something that needs to be addressed further upstream.

[debora-ito]

@nerdyness ok, so I've run some tests locally. This scenario won't work on v1 for the same reasons pointed out in #803, and as explained in that thread this won't be changed due to backwards compatibility. Also, in v1 this doesn't work:

"Either of the "credential" profiles (creds or creds_p) on their own do work for the Java SDK and the AWS CLI, e.g.: AWS_PROFILE=creds"

so here I'm assuming the described behavior is using Java v2.

Now, chaining role_arn and source_profile in the config file should work in v2, and I confirmed in my local tests that they do. Have you or your customers tested in v2? I see 2.15.15 as version used. If so can they share the logs with the error or explain how they are aware that the source_profile is not working?

[nerdyness]

That's strange that you couldn't reproduce this, but I'm not a Java developer so there is a high chance I'm messing this up and you can help me, @debora-ito 😄

Following these getting started steps, I get:

$ AWS_PROFILE=role mvn exec:java -Dexec.mainClass="com.example.myapp.App"
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< com.example.myapp:myapp >-----------------------
[INFO] Building myapp 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ myapp ---
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[WARNING]
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable t
o load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variabl
e AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): To use assumed roles in the 'test2' profile, the 'sts' service module must be on the class path., ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvide
r(): Unable to load credentials from service endpoint.]
    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build (SdkClientException.java:98)
    at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials (AwsCredentialsProviderChain.java:112)
    at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials (LazyAwsCredentialsProvider.java:45)
    at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials (DefaultCredentialsProvider.java:104)
    at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext (AwsClientHandlerUtils.java:76)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext (AwsSyncClientHandler.java:68)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1 (BaseSyncClientHandler.java:97)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess (BaseSyncClientHandler.java:167)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute (BaseSyncClientHandler.java:94)
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute (SdkSyncClientHandler.java:45)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute (AwsSyncClientHandler.java:55)
    at software.amazon.awssdk.services.s3.DefaultS3Client.createBucket (DefaultS3Client.java:1055)
    at com.example.myapp.App.tutorialSetup (App.java:47)
    at com.example.myapp.App.main (App.java:26)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:831)
[WARNING] thread Thread[idle-connection-reaper,5,com.example.myapp.App] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[idle-connection-reaper,5,com.example.myapp.App] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to  via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=com.example.myapp.App,maxpri=10]
java.lang.IllegalThreadStateException
    at java.lang.ThreadGroup.destroy (ThreadGroup.java:795)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:293)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:78)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:567)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.448 s
[INFO] Finished at: 2021-07-02T08:52:07+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project myapp: An exception occured while executing the Java class. Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCrede
ntialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or
system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): To use assumed roles in the 'test2' profile, the 'sts' service module must be on the class path., ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or
 AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

as opposed to:

$ AWS_PROFILE=creds mvn exec:java -Dexec.mainClass="com.example.myapp.App"
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< com.example.myapp:myapp >-----------------------
[INFO] Building myapp 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ myapp ---
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Creating bucket: bucket1625311176637
bucket1625311176637 is ready.

Uploading object...
Upload complete

Cleaning up...
Deleting object: key
key has been deleted.
Deleting bucket: bucket1625311176637
bucket1625311176637 has been deleted.

Cleanup complete

Closing the connection to Amazon S3
Connection closed
Exiting...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.657 s
[INFO] Finished at: 2021-07-03T13:19:39+02:00
[INFO] ------------------------------------------------------------------------

With SDK version 2.15.15:

$ grep -A3 software.amazon.awssdk pom.xml
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>bom</artifactId>
        <version>2.15.15</version>
        <type>pom</type>

And I've now also tried 2.16.60 as the sample app has been updated since I raised the issue to use this newer version.

Could you share a sufficiently redacted version of your ~/.aws/c* files so I can compare if I'm doing something wrong there?

[debora-ito]

The error message

ProfileCredentialsProvider(): To use assumed roles in the 'test2' profile, the 'sts' service module must be on the class path

shows the CredentialProvider chain tried to find the STS module in the classpath but couldn't find it. To be able to call AssumeRole, STS need to be added as a dependency:

<dependencies>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>sts</artifactId>
    </dependency>
</dependencies>

[nerdyness]

Hello @debora-ito 👋
Thanks for your help and pointing this out. You're absolutely right! It works in v2 if I include all dependencies 🤦 but doesn't in v1 and as you pointed out, you don't want to break backwards compatibility.

Have you considered honoring an environment variable like AWS_JAVA_SDK_V1_NEW_FEATURES=1 or some other mechanism to backport those features into v1? I think a lot of people would be very happy about this.

[debora-ito]

We have no plans to backport v2 features into v1 right now. It's the other way around really, we are working to release new features and closing the 1.11.x feature parity gap in 2.x so customers can migrate to 2.x with more confidence and support.

Let us know if you have more questions.