aws-cpp-sdk Issue GetBucketLocation returns Access Denied - when not called by bucket owner

[jimmy-edicogenome]

We are running on Centos and Ubuntu and seen the issue on both the systems.
GCC version 4.9.3

Here is the scenario - The data that we want to process in our application is available on S3 buckets. The customers control these S3 buckets. The buckets can be in any region (typically they are is us-east-1, us-west-2 and eu-west-2). We try to stream the data from S3 directly to ensure this works from any location we try GetBucketLocation - the issue is GetBucketLocation returns Access Denied if the client running the instance is not the owner of the bucket.
Not sure how to get around this limitation

Thanks

Jimmy

[jimmy-edicogenome]

These are the Actions allowed on the bucket
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:ListMultipartUploadParts",
"s3:PutObject"

[jimmy-edicogenome]

I also would add I can access buckets in us-east-1 without any issues. I get intermittent failures on both us-west (oregon) and eu-west (ireland)

[wps132230]

Have you tried this? Add the following policy to the bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddCannedAcl",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-id>:root"
            },
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket-name>"
            ]
        }
    ]
}

It works for me, but I am not sure if we are in the exact same scenario.

[marcomagdy]

the issue is GetBucketLocation returns Access Denied if the client running the instance is not the owner of the bucket.

The documentation states that only bucket owners are allowed to make this call.

This implementation of the GET operation uses the location subresource to return a bucket's region. You set the bucket's region using the LocationConstraint request parameter in a PUT Bucket request. For more information, see PUT Bucket.
To use this implementation of the operation, you must be the bucket owner.

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html

[jimmy-edicogenome]

Hi

The issue I have is we stream directly from S3 and we cannot stream from buckets in different regions without getting the bucket location and setting it in the Aws::Client::ClientConfiguration structure.

For e.g. if my bucket is in US-WEST-2 but I started my instance in us-east-1 I have no recourse to get the bucket location. Sounds like the only way around is to use pre signed url's or make the bucket owner the same as the instance owner.

[jimmy-edicogenome]

Hi wps132230

Yes I have a similar policy on my bucket.

    {
        "Effect": "Allow",
        "Principal": {
            "AWS": [
                "arn:aws:iam::pubweb",
                "arn:aws:iam::job",
                "arn:aws:iam::batch"
            ]
        },
        "Action": [
            "s3:AbortMultipartUpload",
            "s3:GetBucketLocation",
            "s3:GetObject",
            "s3:GetObjectVersion",
            "s3:ListBucket",
            "s3:ListBucketVersions",
            "s3:ListMultipartUploadParts",
            "s3:PutObject"
        ],
        "Resource": [
            "arn:aws:s3:::data-1",
            "arn:aws:s3:::data-1/*"
        ]
    },

[marcomagdy]

Yeah it's a service limitation.
Depending on the nature of your application, you could store the location of the buckets in some lookup table. Say a global table in DynamoDB.

[jimmy-edicogenome]

Thanks Marco for the response - we are going to use the presigned URL's for now. One suggestion I have seen at other places is to use head bucket. In the C++ sdk head bucket seems to return nothing if it is successful. Looking through the code it looks like some of the information exists in the XmlOutcome but is cleared out before we return.
Here is the code from aws-cpp-sdk-s3/source/S3Client.cpp

HeadBucketOutcome S3Client::HeadBucket(const HeadBucketRequest& request) const
{
Aws::StringStream ss;
Aws::Http::URI uri = ComputeEndpointString(request.GetBucket());
uri.SetPath(uri.GetPath() + ss.str());
XmlOutcome outcome = MakeRequest(uri, request, HttpMethod::HTTP_HEAD);
if(outcome.IsSuccess())
{
return HeadBucketOutcome(NoResult());
}
else
{
return HeadBucketOutcome(outcome.GetError());
}
}

Looks like it is HeadBucketOutcome is cleared - can this still return the metadata from the bucket. I can then parse it out for the location and avoid using GetBucketLocation.

Thanks

--
Jimmy

[jimmy-edicogenome]

Just for reference this is the post I found aws/aws-sdk-go#720
And using boto we see this

client = boto3.client('s3')
info = client.head_bucket(Bucket="data-1")

print info['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region']
us-east-1 <= Good result

info2 = client.head_bucket(Bucket="data-eu-west-1")
print info2['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region']
eu-west-1 <== Good result

[marcomagdy]

Unfortunately we don't have the values of the HTTP headers exposed. But I can see how that can be useful in such cases. I'll add an item to our backlog.

[jimmy-edicogenome]

Hi

Thanks very much for adding this to your backlog. It is indeed a very useful feature.

--
Jimmy