aws-cli role assumption profiles are incompatible with SDK

[uriahcarpenter]

The CLI userguide instructs you to add delegated profiles to the file ~/.aws/config. However, profiles added here are incompatible with the SDK.

JavaDoc in com.amazonaws.profile.path.AwsProfileFileLocationProvider makes it clear that only one config file will be loaded (~/.aws/credentials or ~/.aws/config).

The confusion may be that CLI profiles that use keys are defined in ~/.aws/credentials but profiles that use source_profile should be placed in ~/.aws/config. More details in the CLI configuration section.

The workaround for this problem is to define the role assumption profile in ~/.aws/credentials without the profile prefix. However, there are other tools that do not expected delegated profiles in this file, so this solution is imperfect.

~/.aws/credentials

[stage]
role_arn = arn:aws:iam:: 123456789012:role/delegated-admin
source_profile = default

It would be very helpful if the Java SDK allowed role assumption profiles to be defined using the documented CLI format.

[zhangzhx]

Thanks for reaching out to us. We'll talk about this with other SDK/CLI teams to try to reach to a consistent behavior.

[uriahcarpenter]

Our workaround for this issue is to have duplicate profiles in ~/.aws/config; one with the profile prefix and one without.

[profile stage]
region = us-east-1
output = json

[stage]
region = us-east-1
output = json

[alexmolodchikov]

Hey guys, is there any progress with this ticket?

We are using Boto3 SDK + Assuming Roles + Athena.
Everything works fine up to the moment when we are trying to connect to Athena (we are creating SQL Alchemy engine). Our AWS CLI is configured to use both ~/.aws/config and ~/.aws/credentials.

I think the problem could be related to Athena using jdbc and as a result it falls back Java AWS SDK and Python being a wrapper around. As a result we can reach all AWS resources but Athena.
With Athena it either doesn't assume roles and as a result it tries to locate KMS key under wrong account or if I pass environment variable: AWS_PROFILE={profile} - it fails with an error: Unable to load AWS credentials from any provider in the chain

@declankeyesbevan

Thank you.

[alexmolodchikov]

I've read a bit more on the topic and I'm not sure that it is even possible at this moment of time to make python and java work together (use cross accounts, boto3 and jdbc). Here is the error:
SdkClientExceptionPyRaisable: com.amazonaws.athena.jdbc.shaded.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain

[steinybot]

This behaviour is quite annoying.

@zhangzhx did you ever talk this one over with the other SDK/CLI contributors? To me this is more of a bug than a feature, here's why:

The only thing should go into the credentials file is the aws_access_key_id and the aws_secret_access_key for the source profiles, basically anything that should be kept secret. The config file should contain any other non-sensitive configuration such as the region, role_arn, source_profile, output etc. That way one can share config files and never modify the credentials except when rotating key material. I can also add new roles trivially by simply modifying the config. This is how the CLI works and is the better of the two behaviours.

The fact that the CLI has a different naming scheme for the profile heading is a separate issue. @uriahcarpenter said that duplicating the profiles in the config file is a workaround but it does not solve the real issue of how the credentials are found.

[steinybot]

Example of both problems

~/.aws/credentials:

[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx

[dev]
role_arn = arn:aws:iam::12345:role/dev
source_profile = default

[sandbox]
role_arn = arn:aws:iam::67890:role/admin
source_profile = default

~/.aws/config:

[default]
region = us-east-1

[dev]
region = us-east-1
source_profile = default

[profile dev]
region = us-east-1
role_arn = arn:aws:iam::12345:role/dev
source_profile = default

[sandbox]
region = us-east-1
role_arn = arn:aws:iam::67890:role/admin
source_profile = default

[profile sandbox]
region = us-east-1
source_profile = default

Preferred solution

~/.aws/credentials:

[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx

~/.aws/config:

[default]
region = us-east-1

[dev]
region = us-east-1
role_arn = arn:aws:iam::12345:role/dev
source_profile = default

[sandbox]
region = us-east-1
role_arn = arn:aws:iam::67890:role/admin
source_profile = default

[adrian-baker]

We're baffled as to why this is labelled a feature rather than a bug - the Java SDK is clearly incompatible with the config + credentials formats supported and documented by https://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html .

[millems]

It's unfortunately a very difficult issue to fix while maintaining backwards-compatibility. It has been fixed in the 2.0 preview of the SDK. The CLI's behavior isn't something that can be easily reproduced without a major version bump, because it's fundamentally at odds with the 1.11 behavior.

In fact, the behavior that the reasonable person would expect is also at odds with the CLI's implementation. For example, the CLI currently merges duplicate profiles between configuration and credential files for assume-role credentials and regions, including non-credential properties like the source_profile. It uses separate logic for session and standard credentials where it looks at each file separately.

The feature request for this issue is to create a "cli-compatible mode" for 1.11 that a customer can opt-into, outside of the current default behavior. That would keep the SDK's non-cli-compliant current behavior as the default, but give a workaround other than duplicating profiles.

[adrian-baker]

It's unfortunately a very difficult issue to fix while maintaining backwards-compatibility
...
the behavior that the reasonable person would expect is also at odds with the CLI's implementation

Appreciate those points, but don't think either have any bearing on whether this is a bug or not.

Here's my hacky workaround, which avoids having to hack the credentials files themselves (not really feasible across a lot of clients), and supports switching roles, which I couldn't get to work even with duplicate profiles.

public AWSCredentialsProvider getDefaultCredentials() {

    final String profileName = AwsProfileNameLoader.INSTANCE.loadProfileName();
    final AllProfiles allProfiles = new AllProfiles(Stream.concat(
            BasicProfileConfigLoader.INSTANCE.loadProfiles(
                    AwsProfileFileLocationProvider.DEFAULT_CONFIG_LOCATION_PROVIDER.getLocation()).getProfiles().values().stream(),
            BasicProfileConfigLoader.INSTANCE.loadProfiles(
                    AwsProfileFileLocationProvider.DEFAULT_CREDENTIALS_LOCATION_PROVIDER.getLocation()).getProfiles().values().stream())
            .map(profile -> new BasicProfile(profile.getProfileName().replaceFirst("^profile ", ""), profile.getProperties()))
            .collect(Collectors.toMap(profile -> profile.getProfileName(), profile -> profile,
                    (left, right) -> {
                        final Map<String, String> properties = new HashMap<>(left.getProperties());
                        properties.putAll(right.getProperties());
                        return new BasicProfile(left.getProfileName(), properties);
                    })));

    final BasicProfile profile = Optional.ofNullable(allProfiles.getProfile(profileName))
            .orElseThrow(() -> new RuntimeException(String.format("Profile '%s' not found in %s", 
                    profileName, allProfiles.getProfiles().keySet())));
    if (profile.isRoleBasedProfile()) {
        return new ProfileAssumeRoleCredentialsProvider(STSProfileCredentialsServiceLoader.getInstance(), allProfiles, profile);
    } else {
        return new ProfileStaticCredentialsProvider(profile);
    }
}

[millems]

Nice workaround!

Whether or not it's considered a bug, I can't see a way to fix it in a backwards-compatible way, because customers could be intentionally or unintentionally relying on this behavior. It's been around for a long time.

This is a particularly egregious deviation from the CLI's behavior, but there are others that we'd want to fix as part of any solution we'd come up with.

A higher-work, higher-accuracy workaround would be to back-port the 2.0 profile implementation to 1.11 and make it opt-in, which would be my suggestion for how to fix this.

[david-katz]

@adrian-baker thanks a lot for the workaround! Would it be ok if I use it in a closed-source project? Perhaps you could even license it explicitly (under MIT for instance)?

[adrian-baker]

Of course. I've put it into a gist with MIT license.

[ahgittin]

Note a minor niggle with the code above -- while expecting users to drop profile makes it more consistent with the CLI it will cause any region = XXX setting in the [profile xxx] to be ignored.

If you want region to be respected remove the .map(...) line and pass profile xxx (or go all the way and replace the region provider).

Great workaround though -- TY.

[oripwk]

For anyone who needs a Scala version of @adrian-baker's snippet:

class AssumeRoleCredentialsProvider extends AWSCredentialsProvider {

  private lazy val delegate: ProfileAssumeRoleCredentialsProvider = {
    val profiles = Seq(
      AwsProfileFileLocationProvider.DEFAULT_CONFIG_LOCATION_PROVIDER,
      AwsProfileFileLocationProvider.DEFAULT_CREDENTIALS_LOCATION_PROVIDER
    )
      .map(_.getLocation)
      .map(BasicProfileConfigLoader.INSTANCE.loadProfiles)
      .flatMap(_.getProfiles.values.asScala)
      .map(profile => new BasicProfile(profile.getProfileName.stripPrefix("profile "), profile.getProperties))
      .groupBy(_.getProfileName)
      .mapValues { profiles =>
        new BasicProfile(
          profiles.head.getProfileName,
          profiles.flatMap(_.getProperties.asScala.toSeq).toMap.asJava
        )
      }

    val profileName = AwsProfileNameLoader.INSTANCE.loadProfileName
    val profile = profiles.getOrElse(profileName, throw new RuntimeException(
      s"Profile $profileName' not found in ${profiles.keys}"
    ))
    new ProfileAssumeRoleCredentialsProvider(
      STSProfileCredentialsServiceLoader.getInstance,
      new AllProfiles(profiles.asJava),
      profile
    )
  }

  override def getCredentials: AWSCredentials = delegate.getCredentials
  override def refresh(): Unit = delegate.refresh()
}

[RudolfVonKrugstein]

Hey, thanks for snippet. Now I am going even further, I want to a 2 assume roles in my chain:

~/.aws/credentials:

[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx

~/.aws/config:

[default]
region = us-east-1

[profile differentaccount]
region = us-east-1
role_arn = arn:aws:iam::12345:role/AccountAccess
source_profile = default

[profile dev]
region = us-east-1
role_arn = arn:aws:iam::12345:role/dev
source_profile = differentaccount

The code snippet does not seem to support that. And I am unsure how to adjust it.

The ProfileAssumeRoleCredentialsProvider would, I guess, have to be loaded from a different ProfileAssumeRoleCredentialsProvider. But how do I chain this?

[l1x]

Our workaround for this issue is to have duplicate profiles in ~/.aws/config; one with the profile prefix and one without.

[profile stage]
region = us-east-1
output = json

[stage]
region = us-east-1
output = json

This is not a working workaround for me.

[jtheuer]

Any plans to align the behavior of the AWS SDK with the other SDKs? I wonder why you share the same config file with other SDKs but use it differently. This is really confusing for me as a user.

[l1x]

Any plans to align the behavior of the AWS SDK with the other SDKs? I wonder why you share the same config file with other SDKs but use it differently. This is really confusing for me as a user.

I just gave up on using the Java SDK at all. At this stage it can be considered broken. Using anything else Javascript, C# or Python gives you a much better option. Luckily the tools that we write are pretty small so it is easy to use the other SDKs.

[millems]

@jtheuer Because of this issue, the Java SDK team worked with the other AWS SDKs back in 2017 to write up a (long) profile file standard on which all SDKs can align going forward. That standard was written to (with a few exceptions) directly implement the CLI behavior. That standard was followed for the Java SDK 2.x, so it's quite compatible with the CLI.

If you're willing to add the Java SDK 2.x to your classpath (software.amazon.awssdk:auth), the following adapter should give you CLI-compatible behavior in 1.11.x:

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.BasicSessionCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;

package com.example;

public class CliCompatibleCredentialsProvider implements AWSCredentialsProvider {
    private final ProfileCredentialsProvider delegate;

    public CliCompatibleCredentialsProvider() {
        this.delegate = ProfileCredentialsProvider.create();
    }

    public CliCompatibleCredentialsProvider(String profileName) {
        this.delegate = ProfileCredentialsProvider.create("my-profile-name");
    }

    @Override
    public AWSCredentials getCredentials() {
        AwsCredentials credentials = delegate.resolveCredentials();

        if (credentials instanceof AwsSessionCredentials) {
            AwsSessionCredentials sessionCredentials = (AwsSessionCredentials) credentials;
            return new BasicSessionCredentials(sessionCredentials.accessKeyId(),
                                               sessionCredentials.secretAccessKey(),
                                               sessionCredentials.sessionToken());
        }

        return new BasicAWSCredentials(credentials.accessKeyId(), credentials.secretAccessKey());
    }

    @Override
    public void refresh() {
        throw new UnsupportedOperationException();
    }
}

We'd ideally like to implement the CLI-compatible standard natively in 1.11.x of the SDK, but because of resource limitations are focusing our efforts on getting 2.x up to feature parity with 1.11.x.

[elonderin]

hi, i just wanted to post our case and confirm that i do the right thing:

Our setup is like so:

• We have a company wide account were all the actual users live. (tier 1)
• These are granted assume rights/trust-relationships into IAM roles into separate AWS project accounts such as account-admin and poweruser etc. (tier 2)
• only these are allowed to assume application specific roles, such as is assigned to an ECS task (ie the one that matters for running the app). (tier 3)

this is what i do now for local dev so that my app runs locally with the same role as in AWS:

1. we do aws sts get-session-token and store these creds in the ~/.aws/credentials under the default profile, eg:

[default]
aws_secret_access_key=....
aws_access_key_id=...
aws_session_token=...

2. /config is setup like so:

  [profile test]
  role_arn = <tier 2 role arn>
  source_profile = default
  ...

3. in CLI we define export AWS_PROFILE=test

This all works fine but makes the app run locally under the tier 2 user, which is undesirable for several reasons. I have not been able to setup a profile in the config such that it would allow me to select it and assume directly a tier 3 role.

So what i do now in my app is this:

4. get a credential provider for the tier 2 role (with the above workaround)

    final AWSCredentialsProvider providerFromProfile =
        new ProfileAssumeRoleCredentialsProvider(STSProfileCredentialsServiceLoader.getInstance(), allProfiles, activeProfile);

    log.info("Assumed intermediate role: {}", activeProfile.getRoleArn());

5. then i manually create a 2nd provider like so for the tier 3 app role.

    final AWSSecurityTokenService appRoleSts = AWSSecurityTokenServiceClientBuilder.standard()
          .withRegion(activeProfile.getRegion())
          .withCredentials(providerFromProfile)
          .build();

    final var providerForAppRole = new STSAssumeRoleSessionCredentialsProvider
         .Builder("my-app-role", activeProfile.getRoleSessionName())
         .withStsClient(appRoleSts)
         .build();

6. use this provider when building AWS clients such as for S3.

Note:

what confused me a little was, that with this i got many more of the known warnings:

Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.

After debugging the code, i noticed that the internal logic will always parse the config/credentials as part of the CTOR/init (static and field initialization) of credential providers -- even if they are not using the information.

@RudolfVonKrugstein this case might be similar to ur's but i'm not sure

[petersiemen]

To sum it up: The problem (at least for me) is that the AWS Java SDK will only know the profiles listed in .aws/credentials.
In my situation I only have one profile defined in .aws/credentials. This is the AWS account of my organization where I administer users. All other AWS subaccounts are switch into using roles.

So my .aws/credentials looks like

[myorg-master]
aws_access_key_id = XYZ...
aws_secret_access_key = XYZ...

My .aws/config looks like

[profile myorg-master]
region = eu-central-1

[profile myorg-development]
region = eu-central-1
role_arn = arn:aws:iam::123456789:role/OrganizationAccountAccessRole
source_profile = myorg-master

[profile myorg-production]
region = eu-central-1
role_arn = arn:aws:iam::987654321:role/OrganizationAccountAccessRole
source_profile = profile myorg-master

When I pass AWS_PROFILE=myorg-development to the JVM process AWS Java SDK will not have the profile available.
Only myorg-master

My solution/workaround to this problem is to not use .aws/credentials at all and tell the AWS Java SDK to look up the credentials from .aws/config together with the other configuration.
There is a setting for it: AWS_CREDENTIAL_PROFILES_FILE=$HOME/.aws/config

So in my solution i keep .aws/crendentials empty as the desert.
My .aws/config looks like

[profile myorg-master]
region = eu-central-1
aws_access_key_id = XYZ...
aws_secret_access_key = XYZ..

[myorg-master]
region = eu-central-1
aws_access_key_id = XYZ...   # same as above (see comment below)
aws_secret_access_key = XYZ..  # same as above (see comment below)

# used by AWS Java SDK
# - otherwise you would have to pass `profile myorg-development` to the JVM process, using this double definition AWS_PROFILE=myorg-development can be used in the SDK
# - source_profile needs to reference 'profile myorg-master' instead of 'now-master'
[myorg-development]
region = eu-central-1
role_arn = arn:aws:iam::123456789:role/OrganizationAccountAccessRole
source_profile = profile myorg-master

[profile myorg-development]
region = eu-central-1
role_arn = arn:aws:iam::123456789:role/OrganizationAccountAccessRole
source_profile = myorg-master

[myorg-production]
region = eu-central-1
role_arn = arn:aws:iam::987654321:role/OrganizationAccountAccessRole
source_profile = profile myorg-master

[profile myorg-production]
region = eu-central-1
role_arn = arn:aws:iam::987654321:role/OrganizationAccountAccessRole
source_profile = myorg-master

When I pass
AWS_CREDENTIAL_PROFILES_FILE=$HOME/.aws/config
and
AWS_PROFILE=myorg-development
to the JVM process the AWS Java SDK will be able to find the profile using the DefaultAWSCredentialsProviderChain.

Extra:
Because I do not want to fuck up terraform
I need to have export AWS_SHARED_CREDENTIALS_FILE=$HOME/.aws/config in my .bashrc.
as well.

[elruwen]

Hi!

Other SDKs seem to swap between the behaviours depending on the environment variable AWS_SDK_LOAD_CONFIG=true. I wouldn't even mind if there is a java version of that environment variable so people can opt in.

I think everybody would be happy with that. And I would not consider that a breaking change if you need to opt in via an environment variable.

[AnweshaSen]

Hii, I am facing an issue some kind similar, i.e: my config file looks like:

[default]
region =
output =

[profile1]
role_arn = <role_arn>
source_profile = default
mfa_serial = <mfa_serial>
region =
output =

am trying to access AmazonElasticMapReduce service which this profile1 has access to. Now if I write ProfileCredentialsProvider("profile1").getCredentials(), it shows me an error like : "Exception in thread "main" com.amazonaws.AmazonClientException: Cannot load credentials from .aws/credentials file. Make sure that the credentials file exists and the profile name is specified within it."

My CLI is working perfectly just by changing "profile prifile1" in .aws/config.
Am using AWS SDK 1.x.x. How to solve this issue?

[nimeshpande]

JAVA SDK uses either one of the file, Config or credentials so the work around is
Mention all the properties in the credentials file under the profile as

[Profile_name]
region=...
output=..
aws_secret_access_key=....
aws_access_key_id=...
aws_session_token=...
role_arn =...
source_profile=...

[replace ... with your actual value]

in the classpath add dependency of aws-java-sdk-sts to class path
use profileCredentialsProvider for passing the profile value as follows.
ProfileCredentialsProvider creds = new ProfileCredentialsProvider("profile_name");
add the creds when generating any aws service's clientbuilder

Hope this helps......

[millems]

Today's release should hopefully help with some of these issues. The 1.x SDK will now fall back to "profile X" if there is no "X" profile defined and the SDK is configured to look for profile X. There is no longer a warning if a "profile X" exists in the file.

[elruwen]

My setup:

~/.aws/config

[default]
output = json

[profile source1]
output = json

[profile source2]
output = json

[profile account1a]
role_arn = arn:aws:iam::XXXXX:role/myrole
source_profile = source1

[profile account1b]
role_arn = arn:aws:iam::XXXXX:role/myrole
source_profile = source1


[profile account2a]
role_arn = arn:aws:iam::XXXXX:role/myrole
source_profile = source2

[profile account2b]
role_arn = arn:aws:iam::XXXXX:role/myrole
source_profile = source2

~/.aws/credentials

[source1]
aws_access_key_id=XXXXXXX
aws_secret_access_key=XXXXXXXX
aws_session_token="XXXXXX"
aws_session_expiration=2021-XXxXX

[source2]
aws_access_key_id=XXXXXXX
aws_secret_access_key=XXXXXXXX
aws_session_token="XXXXXX"
aws_session_expiration=2021-XXxXX

Before version 79 I always got the error message for 4 times (one for each account in config). Now the message is gone, but the only profiles available are source1 and source2.

So setting a profile via environment variable is still not working.

This is at the moment expected because the SDK uses only one file?

[millems]

Yes, you still need to use v2 to get CLI-compatible behavior. v1 still only uses one file.

[reegnz]

Can we get the v2 behaviour hidden behind a flag? As mentioned above, AWS_SDK_LOAD_CONFIG=true worked for aws-sdk-go, I think it would be backward compatible and it would most definitely be a welcome change.

I have a lot of code depending on SDKv1, and a rewrite to v2 will most likely never happen, so being able to have the SDK 'just work' by setting an env variable would be great.