Missing 'profile ' prefix when looking up key values in ~/.aws/config file.

[pagil]

SUMMARY

The following article introduced new API which should be used to load AWS credentials and region:
https://aws.amazon.com/blogs/developer/client-constructors-now-deprecated/

However, when the following API is used:

AmazonS3ClientBuilder.defaultClient();

Exception com.amazonaws.SdkClientException: Unable to load region information from any provider in the chain is thrown.

Steps to reproduce

1. Create a new gradle Java project.
2. Add the following dependencies:

    // aws java sdk
    compile('com.amazonaws:aws-java-sdk-s3:1.11.84')
    compile('com.amazonaws:aws-java-sdk-core:1.11.84')

3. Create a new class and add the following main method in it:

    public static void main(String[] args) {
        System.out.println(System.getenv("AWS_PROFILE"));
        AmazonS3ClientBuilder.defaultClient();
    }

4. Make sure you set environment variable AWS_PROFILE=test.

5. Add the following lines into your ~/.aws/config file:

[profile test]
region=ap-southeast-2

6. (Optional and not relevant to the bug) Add the following lines into your ~/.aws/credentials file:

[test]
aws_access_key_id=xyz
aws_secret_access_key=xyz

7. Run the class with the environment variable AWS_PROFILE=test and check the console.

I get the following console output:

test
16:34:45.691 [main] DEBUG com.amazonaws.AmazonWebServiceClient - Internal logging successfully configured to commons logger: true
16:34:45.769 [main] DEBUG com.amazonaws.metrics.AwsSdkMetrics - Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
16:34:46.560 [main] WARN com.amazonaws.auth.profile.internal.BasicProfileConfigLoader - The legacy profile format requires the 'profile ' prefix before the profile name. The latest code does not require such prefix, and will consider it as part of the profile name. Please remove the prefix if you are seeing this warning.
16:34:46.560 [main] WARN com.amazonaws.auth.profile.internal.BasicProfileConfigLoader - The legacy profile format requires the 'profile ' prefix before the profile name. The latest code does not require such prefix, and will consider it as part of the profile name. Please remove the prefix if you are seeing this warning.
16:34:46.560 [main] WARN com.amazonaws.auth.profile.internal.BasicProfileConfigLoader - The legacy profile format requires the 'profile ' prefix before the profile name. The latest code does not require such prefix, and will consider it as part of the profile name. Please remove the prefix if you are seeing this warning.
16:34:46.560 [main] WARN com.amazonaws.auth.profile.internal.BasicProfileConfigLoader - The legacy profile format requires the 'profile ' prefix before the profile name. The latest code does not require such prefix, and will consider it as part of the profile name. Please remove the prefix if you are seeing this warning.
16:34:46.560 [main] WARN com.amazonaws.auth.profile.internal.BasicProfileConfigLoader - The legacy profile format requires the 'profile ' prefix before the profile name. The latest code does not require such prefix, and will consider it as part of the profile name. Please remove the prefix if you are seeing this warning.
Exception in thread "main" com.amazonaws.SdkClientException: Unable to load region information from any provider in the chain
16:35:03.368 [main] DEBUG com.amazonaws.regions.InstanceMetadataRegionProvider - Ignoring failure to retrieve the region: Unable to contact EC2 metadata service.
	at com.amazonaws.regions.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:56)
	at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:367)
	at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:337)
	at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:38)
	at com.amazonaws.services.s3.AmazonS3ClientBuilder.defaultClient(AmazonS3ClientBuilder.java:61)
	at com.test.AppTest.<init>(AppConfig.java:14)
	at com.test.AppTest.main(AppConfig.java:25)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

Process finished with exit code 1

IMPORTANT

Region cannot be loaded from ~/.aws/config file because aws-java-sdk-core searches the values in the ~/.aws/config file by key test not profile test. Prefix 'profile ' is missing.

This is a bug, because ~/.aws/config file keys should have 'profile ' prefix if they are different from default. For example, this article demonstrates the format of the ~/.aws/config file:

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

WORKAROUNDS

1. Add the following lines into ~/.aws/config file:

[test]
region=ap-southeast-2

Note the difference: `[test]` instead of `[profile test]`. This is the format of the `~/.aws/credentials` file.

2. Set environment variable AWS_REGION=test in addition to AWS_PROFILE=test.

3. Use old deprecated API:

new AmazonS3Client(new DefaultAWSCredentialsProviderChain());

NOTES

I have also tried the newest versions of the libraries:

    // aws java sdk
    compile('com.amazonaws:aws-java-sdk-s3:1.11.106')
    compile('com.amazonaws:aws-java-sdk-core:1.11.106')

The issue is reproducible with the newest available versions of the libraries.

[uriahcarpenter]

A workaround of duplicating sections is described in #803

[spfink]

I'm not sure that we can fix this without the possibility of breaking customers.

We need to continue to support the current behavior and if we add a check that looks up "profile test" if "test" was inputted and returned null then we could start reading profiles that are out of date or unwanted.

I'll sync with the team to see what we want to do here.

[twiggy]

Just had this issue. I'd suggest you check [profile xxxx] first and if not found then try [xxxx] or the other way around. anyone using named profiles from the java sdk over default ( guessing a small amount ) will likely already be dealing with this. Most of the time i'm just dealing with this locally and on prod it finds the region of the machine its running on.

In general all this "automatic" stuff would be nice to turn off as it can be super confusing what is going on. Some kind of explicit way to trigger these paths outside of just calling .build() on service outside of a try/catch would be nice.

[millems]

We will be fixing the "profile" prefix behavior for credential and region loading in V2 of our SDK to be consistent with the way the CLI works: aws/aws-sdk-java-v2#31

We're not able to make this change in 1.11 because of the risk of breaking customers that rely on the current behavior.