S3 V4 Signatures Frankfurt getObjectMetadata

[adrianbk]

Is there something different about AmazonS3Client.getObjectMetadata(..) such that V4 signatures are not being sent for HEAD requests?

I'm using the following code to verify that I can interact with buckets in all regions without setting specific region endpoints but it fails on getObjectMetadata for the EU_Frankfurt region. The strange thing is that only getObjectMetadata fails - creating a bucket, deleting a bucket, get and put work fine.

I've tried 'SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY' but no difference.

Stacktrace below.

SDK versions: 1.9.19 &1.9.20

      AmazonS3Client amazonS3Client = new AmazonS3Client(basicAWSCredentials)
      (Region.values() - [Region.US_GovCloud, Region.CN_Beijing]).each { Region region ->
        String bucketName = "${BUCKET_NAME}-${region ?: region.name}"

        println("Creating bucket: ${bucketName}")
        CreateBucketRequest createBucketRequest = new CreateBucketRequest(bucketName, region)
        amazonS3Client.createBucket(createBucketRequest)

        println "-- uploading"
        InputStream inputStream = new FileInputStream(file)
        ObjectMetadata metadata = new ObjectMetadata()
        metadata.setContentLength(file.length())
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, file.name, inputStream, metadata)
        amazonS3Client.putObject(putObjectRequest)

        //FAILING HERE FOR EU_Frankfurt
        println "------Getting object metadata"
        GetObjectMetadataRequest metadataRequest = new GetObjectMetadataRequest(bucketName,file.name)
        ObjectMetadata objectMetadata = amazonS3Client.getObjectMetadata(metadataRequest)

        println "------Getting object"
        GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, file.name)
        getObjectRequest.setRange(0, 0);
        S3Object object = amazonS3Client.getObject(getObjectRequest)
        InputStream objectData = object.getObjectContent();
        objectData.close();

        println "-- Deleting object"
        DeleteObjectRequest deleteObjectRequest = new DeleteObjectRequest(bucketName, putObjectRequest.getKey())
        amazonS3Client.deleteObject(deleteObjectRequest)

        println("Deleting bucket: ${bucketName}")
        DeleteBucketRequest deleteBucketRequest = new DeleteBucketRequest(bucketName)
        amazonS3Client.deleteBucket(deleteBucketRequest)
      }

Creating bucket: testv4signatures-us-west-1
-- uploading
------Getting object metadata
------Getting object
-- Deleting object
Deleting bucket: testv4signatures-us-west-1
Creating bucket: testv4signatures-us-west-2
-- uploading
------Getting object metadata
------Getting object
-- Deleting object
Deleting bucket: testv4signatures-us-west-2
Creating bucket: testv4signatures-eu-west-1
-- uploading
------Getting object metadata
------Getting object
-- Deleting object
Deleting bucket: testv4signatures-eu-west-1
Creating bucket: testv4signatures-eu-central-1
-- uploading
Feb 19, 2015 12:06:34 AM com.amazonaws.services.s3.AmazonS3Client shouldRetryWithAuthParam
WARNING: Attempting to re-send the request to testv4signatures-eu-central-1.s3-external-1.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.
------Getting object metadata

com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1AD4BF78493A5206), S3 Extended Request ID: 78CyvR+iqPNnh1/UQ/xM/DkcwTCwVtPniVLlvfMyj96djN6sFHh28KE+jFx0TWITDzezDW6+LzQ=
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3736)
    at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)

[hansonchar]

AmazonS3Client.getObjectMetadata(..) is the only request that uses http method HEAD which apparently couldn't trigger a 307 redirect for the retry to work.

[david-at-aws]

Signature version 4 requires that you know the region of the bucket you're talking to when creating the signature. If you don't configure your S3 client for a particular region, we use the old S3Signer (unless you've set the ENFORCE_S3_SIGV4_SYSTEM_PROPERTY, in which case we try using us-east-1 and throw you an exception if that turns out to be wrong; ENABLE_... opts in to SigV4 if you've configured a region).

Since eu-central-1 requires Signature version 4, using the S3Signer yields a 400 response from the service. When this response has a body (as most responses do), it contains an error message that tells us the request failed because we need to use SigV4; in this case we do a complicated dance to try to figure out what region the bucket is in and then retry the request with a V4 signature. That's the source of the WARNING you're seeing when doing putObject() on the eu-central-1 bucket.

However, as @hansonchar mentioned, getObjectMetadata() translates into a HEAD request. Per the HTTP spec the response to a HEAD request cannot have a body, so we don't get an error message identifying why the request was invalid. Since there are plenty of other reasons a request could be invalid and get a 400, we decided not to do the (rather expensive) region-discovery dance in this case, and to simply throw an exception.

Our recommendation is to always use a client that's configured for the region of the bucket you're accessing so you can avoid this retry dance altogether and sign the request appropriately on the first try.

[adrianbk]

Thanks @david-at-aws @hansonchar

I 'm planning on using virtual-hosted–style urls so that things just work without having to explicitly specify where the bucket resides. i.e. relying on the "complicated dance to try to figure out what region the bucket is in and then retry the request with a V4 signature". For now I can avoid using getObjectMetadata and instead do a lightweight getObject (request.setRange(0, 0))

[david-at-aws]

The complicated dance will significantly increase both the latencies you see (three round-trips instead of one) and your exposure to the vagaries of Internet weather (one of those round-trips is always to us-east-1, even if you and the bucket are right next door to each other in eu-central-1). For some applications this may be perfectly acceptable, just want to make sure you know what you're signing up for. :)

[adrianbk]

Yep, I think it's fine in this case. I want to minimise end user configuration (regions, signature versions, etc.) Here's what I'm aiming for

1 Specify a virtual-hosted–style url which will work in all regions regardless of signature versions
2. When the "dance" latencies become an issue the region can be included in the url e.g. ://bucket.s3-aws-region.amazonaws.com.

[david-at-aws]

Sounds reasonable to me. If you haven't seen it already, the AmazonS3URI class will parse region (if applicable), bucket, and key from a URI of the form http[s]://{bucket}.s3[.-]{region}.amazonaws.com/{key}.

Sounds like you're all set, so I'm going to close this. Please feel free to reopen if you have any other questions!

[adrianbk]

Thanks. AmazonS3URI will certainly help 👍

[jain-aanchal]

I am facing the same problem. Where and how do I set ENFORCE_S3_SIGV4_SYSTEM_PROPERTY property ?

[manikandanrs]

It is a system property that you need to set.