AccessDenied with PutObject (yet works with CLI)

[kemitix]

Using aws-java-sdk-s3 1.11.562

Error when using SDK to putObject, but works with CLI (i.e. I have IAM permissions, and my ~/.aws/credentials file is properly configured)

This works fine:

aws s3 cp src/main/scala/net/kemitix/s3thorp/MainS3.scala s3://my-bucket/MainS3.scala

This causes the error below:

  private def putObject: Unit =
    AmazonS3ClientBuilder.defaultClient
      .putObject(
        new PutObjectRequest("my-bucket", "MainS3.scala",
          new File("src/main/scala/net/kemitix/s3thorp/MainS3.scala")))

[error] (run-main-0) com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 6FE77FE664905747; S3 Extended Re
quest ID: Bu64wjvX58JuNNy3W1I4E7pmkMi4xFwVmHxvckaFkEXAB1IZuT/kH9uv9ZIuSV6lwTgrMe9PeYs=), S3 Extended Request ID: Bu64wjvX58JuNNy3W1I4E7pmkMi4xFwVmHxvckaFkEXAB1IZuT/kH9uv9ZIuSV6lwTgrMe9PeYs=     
[error] com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 6FE77FE664905747; S3 Extended Request ID: Bu6
4wjvX58JuNNy3W1I4E7pmkMi4xFwVmHxvckaFkEXAB1IZuT/kH9uv9ZIuSV6lwTgrMe9PeYs=), S3 Extended Request ID: Bu64wjvX58JuNNy3W1I4E7pmkMi4xFwVmHxvckaFkEXAB1IZuT/kH9uv9ZIuSV6lwTgrMe9PeYs=                  
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
[error]         at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
[error]         at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
[error]         at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
[error]         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920)
[error]         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866)
[error]         at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:389)
[error]         at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5800)
[error]         at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1789)
[error]         at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1749)
[error]         at net.kemitix.s3thorp.MainS3$.putObject(MainS3.scala:28)
[error]         at net.kemitix.s3thorp.MainS3$.main(MainS3.scala:15)
[error]         at net.kemitix.s3thorp.MainS3.main(MainS3.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)

[dagnir]

Hi @kemitix, if you haven't already, can you try explicitly providing your credentials to the client using the .withCredentials() method?

        AmazonS3 s3 = AmazonS3ClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY)))
                .build();

It's possible the Java SDK is picking up a different set of credentials from your environment than the CLI is.

[kemitix]

@dagnir That does work.

My ~/.aws/ files are configured according to this.

[dagnir]

Okay I see, it looks like the SDK is not able to load the credentials from your .aws/credentials file. The Java SDK is not fully compatibly with the the profile loading logic of the CLI, see #803 and #1707 for some discussions on this issue.

Can you give show us how you configured your file? Please do no post your actual credentials.

Another option may be to try upgrading to V2 (https://github.com/aws/aws-sdk-java-v2) if that's an option for you, since it's much more compatible with the CLI.

[kemitix]

I can't use V2 as it doesn't support multi-part uploads or TransferManager yet.

Here are my redacted credentials:

[default]
; same as kemitix profile
aws_access_key_id = ak1
aws_secret_access_key = sk1
region = eu-west-1

[kemitix]
aws_access_key_id = ak1
aws_secret_access_key = sk1
region = eu-west-1

[other-account]
aws_access_key_id = ak2
aws_secret_access_key = sk2
region = eu-west-1

[an-assumedrole-account]
source_profile = other-account
role_arn = arn:aws:iam::123456123456:role/a-role-to-assume
region = eu-west-2

There are a few more profiles, but they all follow the same pattern. I've been trying to use the default or kemitix profiles. e.g. export AWS_PROFILE=kemitix

[dagnir]

Do you by any chance have credentials defined via the environment variables or Java system properties that the default credentials chain may be picking up first over the credentials file?

aws-sdk-java/aws-java-sdk-core/src/main/java/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.java

Lines 19 to 39 in ec9fd59

	/**
	* AWS credentials provider chain that looks for credentials in this order:
	* <ul>
	* <li>Environment Variables -
	* <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code>
	* (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET),
	* or <code>AWS_ACCESS_KEY</code> and <code>AWS_SECRET_KEY</code> (only recognized by Java SDK)
	* </li>
	* <li>Java System Properties - aws.accessKeyId and aws.secretKey</li>
	* <li>Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI</li>
	* <li>Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set
	* and security manager has permission to access the variable,</li>
	* <li>Instance profile credentials delivered through the Amazon EC2 metadata service</li>
	* </ul>
	*
	* @see EnvironmentVariableCredentialsProvider
	* @see SystemPropertiesCredentialsProvider
	* @see ProfileCredentialsProvider
	* @see EC2ContainerCredentialsProviderWrapper
	*/
	public class DefaultAWSCredentialsProviderChain extends AWSCredentialsProviderChain {

[kemitix]

No, I have no AWS_* environment variables set, and no aws.* Java system properties are defined.

[kemitix]

Solved it. I tried using the debugger on the com.amazonaws.auth.profile.ProfileCredentialsProvider#getCredentials method. (Should have tried this sooner, I know)
Stepping through I spotted this exception being caught

com.amazonaws.auth.profile.ProfileCredentialsProvider@2e957f1: Invalid property format: no '=' character is found in the line [; same as kemitix profile].

So, the problem turns out that ; is not a valid comment character, but that the reporting of that as an error is swallowed by the SDK.
Is it worth opening a new issue to request that such errors not be swallowed?
Thanks for your help @dagnir

[subhambiswas97]

Seems to be a sdk support issue. Got solved by updating to aws-java-sdk-s3 1.11.1006.