Unable to create S3 buckets after the new S3 defaults

[jwitrick]

What happened?

Recently AWS updated their default AWS S3 bucket settings and since then I am unable to create S3 buckets due to ACL errors.

How can we reproduce it?

Not sure if the settings are global or not. but in us-east-2 when i try to create an s3 bucket I get an error about ACL errors (Note: same code works in us-west-2).

Here is the example code i was using:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: crosspln-buckettest-failed-acl
  annotations:
    # crossplane.io/external-name: crosspln-buckettest-failed-acl
spec:
  forProvider:
    acl: private
    locationConstraint: us-east-2
    publicAccessBlockConfiguration:
      blockPublicPolicy: false
      blockPublicAcls: true
      ignorePublicAcls: false
      restrictPublicBuckets: false
    objectLockEnabledForBucket: false
    serverSideEncryptionConfiguration:
      rules:
      - applyServerSideEncryptionByDefault:
          sseAlgorithm: AES256
    corsConfiguration:
      corsRules:
      - allowedMethods:
        - "GET"
        allowedOrigins:
        - "*"
        allowedHeaders:
        - "*"

The error seems to be when trying to put an acl on bucket. Here is a small bit from the AWS CloudTrail event:

"errorCode": "AccessControlListNotSupported",
"errorMessage": "The bucket does not allow ACLs",

What environment did it happen in?

• Crossplane version: 1.11.1
• Provider-aws version: 0.38.0
• Kube version: 1.24
• Kube distro: AWS EKS

Email from AWS:

We are reaching out to inform you that starting in April 2023 Amazon S3 will change the default security configuration for all new S3 buckets. For new buckets created after this date, S3 Block Public Access will be enabled, and S3 access control lists (ACLs) will be disabled.

The majority of S3 use cases do not need public access or ACLs. For most customers, no action is required. If you have use cases for public bucket access or the use of ACLs, you can disable Block Public Access or enable ACLs after you create an S3 bucket. In these cases, you may need to update automation scripts, CloudFormation templates, or other infrastructure configuration tools to configure these settings. To learn more, read the AWS News blog [1] and What's New announcement [2] on this change or visit our user guide for S3 Block Public Access [3] and S3 Object Ownership to disable ACLs [4]. Also, see our user guide for AWS CloudFormation on these settings [5][6].

If you have any questions or concerns, please reach out to AWS Support [7].

[1] https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/
[2] https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/
[3] https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html
[4] https://docs.aws.amazon.com/AmazonS3/latest/userguide/about-object-ownership.html
[5] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-publicaccessblockconfiguration.html
[6] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-ownershipcontrols.html
[7] https://aws.amazon.com/support

[chlunde]

try this:

-            acl: private
+            objectOwnership: BucketOwnerEnforced

[tcd156]

You actually cannot even create a bucket currently without one of the publicAccessBlockConfiguration settings set to true, which is not in line with the AWS API.

[chlunde]

@tcd156 do you have an example with full YAML? I can't reproduce that.

[tcd156]

I'm on 0.38.0, not latest if that change has been included, but literally any bucket with publicAccessBlockConfiguration with all of them being false.

If you set all values as false, you get an error: "Must provide at least one configuration"

[chlunde]

    publicAccessBlockConfiguration:
      blockPublicPolicy: false
      blockPublicAcls: false
      ignorePublicAcls: false
      restrictPublicBuckets: false

@tcd156 Right, I think that's the behaviour for the AWS API, could you please try removing this block instead of all of them being false?

Is this a change? You can argue that provider-aws should "do what you mean" and delete any public access block config if all values are set to false, AWS at least decided to error out instead in this case, so crossplane just mirrors the AWS behaviour.

However, I think if you have a patch for allowing the user to toggle publicAccessBlockConfiguration.blockPublicPolicy, it is not trivial to drop the whole block in a composition patch, so maybe this is a good argument for fixing this in provider-aws?

[tcd156]

Buckets default to no public access, so that has the opposite of the intended effect and blocks access.

[chlunde]

I think there are a few different issues here:

• The "late initialize pattern" in providers means that it's difficult to know if a user wants to delete some attribute/subresource, or if the provider should late-initialize a field
• The fact that this AWS API doesn't allow an all-false value makes it even more difficult to workaround the late init issue.
• While new buckets default to "no public access", I believe this means "new buckets get a public block access policy", not that a "missing policy means public access is blocked"

So if a user does

    publicAccessBlockConfiguration: {}

or

    publicAccessBlockConfiguration: null

provider-aws will currently late-init the field instead of deleting the public access block configuration (opening the bucket).

So, to preserve the current behaviour, I believe we would have to special-case handle this:

    publicAccessBlockConfiguration:
      blockPublicPolicy: false
      blockPublicAcls: false
      ignorePublicAcls: false
      restrictPublicBuckets: false

to trigger DeletePublicAccessBlock

But this is not 100% in line with the AWS API.

[Red-M]

Running into this as well, currently making it impossible to create/manage public S3 buckets.

[Rudya93]

@haarchri Hello, still get the same issue:
operation error S3: PutBucketAcl, https response error StatusCode: 400, RequestID: CRNFPHETCA20TWN1, HostID: 31fH67/080EIp5wyXcr7L5TY4ZpUYVTCEw20bgrWsUq2AUvwYgj8ISpVnznPVQ3ucGnfiQZa3hs=, api error MissingSecurityHeader: Your request was missing a required header

Using this publicAccessBlockConfiguration: {}, or not specify it
on latest provider 0.40 Successfully pulled image "xpkg.upbound.io/crossplane-contrib/provider-aws:v0.40.0" in 7.115975104s (7.115993244s including waiting)

[Rudya93]

The same error with
publicAccessBlockConfiguration: blockPublicPolicy: false blockPublicAcls: false ignorePublicAcls: false restrictPublicBuckets: false

[Ehco1996]

i opened a new issue to track this #1760