s3api get-bucket-policy does not return valid policy

[quiver]

problem

$s3api get-bucket-policy returns a JSON policy, but

• s3api put-bucket-policy
• s3 manage console's Bucket Policy Editor

does not accept its returned policy data.

How to reproduce

create a bucket and set bucket policy

$ aws s3 mb 13b7
$ cat policy.json
{
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
            "AWS": "*"
         },
         "Action": "s3:GetObject",
         "Resource": "arn:aws:s3:::13b7/*"
      }
   ]
}
$ aws s3api put-bucket-policy --bucket 13b7 --policy file://policy.json

Retrieve bucket policy

$ aws s3api get-bucket-policy --bucket 13b7 --debug
...
2014-07-19 14:37:46,736 - MainThread - botocore.response - DEBUG - Response Headers:
...
content-type: application/json
2014-07-19 14:37:46,737 - MainThread - botocore.response - DEBUG - Response Body:
{"Version":"2008-10-17","Statement":[{"Sid":"","Effect":"Allow","Principal":{"AWS":"*"},"Action":"s3:GetObject","Resource":"arn:aws:s3:::13b7/*"}]}
...
2014-07-19 14:37:46,738 - MainThread - awscli.errorhandler - DEBUG - HTTP Response Code: 200
{
    "Policy": "{\"Version\":\"2008-10-17\",\"Statement\":[{\"Sid\":\"\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::13b7/*\"}]}"
}

As you see, API server's Response Body and aws cli's Response Body is different.
aws cli's response is { "Policy" : "api-server's-response-as-string"}

put s3's bucket policy back to the original bucket

$ aws s3api get-bucket-policy --bucket 13b7 > returned_policy.json
$ cat returned_policy.json
{
    "Policy": "{\"Version\":\"2008-10-17\",\"Statement\":[{\"Sid\":\"\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::13b7/*\"}]}"
}
$ aws s3api put-bucket-policy --bucket 13b7 --policy file://returned_policy.json

A client error (MalformedPolicy) occurred when calling the PutBucketPolicy operation: Invalid policy element

aws s3api get-bucket-policy should return API server's response body as is.

Ref #678

[jamesls]

Yeah in this case we have the JSON policy within a JSON string so the quotes are escaped which I agree is not ideal. The only thing that makes this difficult is that this would be a breaking change to the output.

As a workaround, you can use the --output text to get just the JSON policy. Leaving this open for the time being while do a little more digging.

[alilee]

get-bucket-notification and get-bucket-logging produce beautifully formatted JSON. Makes it much easier to debug..

[jamesls]

I'd like to follow up on this issue. We can't really change the existing behavior because we need to stay backwards compatible, so we have a few options:

• Add an example to this command about how you can round trip policies. Something like:

To print the JSON policy for the bucket:

$ aws s3api get-bucket-policy --query Policy --output text --bucket bucket
{"Version":"2012-10-17","Statement":[{"Sid":"AddPerm","Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::bucket/*"}]}

You can then also save this policy to a file, edit the file, and reapply the policy to your S3 Bucket:
$ aws s3api get-bucket-policy --query Policy --output text --bucket bucket > mypolicy.json
$ # edit your policy...
$ aws s3api put-bucket-policy --bucket bucket --policy file://mypolicy.json

• Add an opt-in option for this command to just pretty print the policy:

$ aws s3api get-bucket-policy --bucket bucket --pretty-print
{
  "Statement": [
    {
      "Action": "s3:GetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Resource": "arn:aws:s3:::bucket/*",
      "Sid": "AddPerm"
    }
  ],
  "Version": "2012-10-17"
}

I'm leaning towards the first option, but I'm open to either.

Thoughts? cc @kyleknap @mtdowling @rayluo @JordonPhillips

[jamesls]

For now we can start with an example: #1583

If we decide to add option 2, we can always update this example as needed.