S3 Buckets fail to become Synced and Ready

[lbowe]

What happened?

When creating an S3 bucket with tags, the CRD never shows Synced or Ready as True.

Looking at the CRD, there is an error which states:
update failed: cannot create or update: api error NoSuchTagSet: The TagSet does not exist

An example of what the yaml for the bucket looks like:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: test-bucket
  namespace: test-namespace
spec:
  deletionPolicy: "Orphan"
  forProvider:
    acl: "private"
    locationConstraint: us-west-2
    objectOwnership: BucketOwnerEnforced
    objectLockEnabledForBucket: false
    publicAccessBlockConfiguration:
      blockPublicAcls: true
      blockPublicPolicy: true
      ignorePublicAcls: true
      restrictPublicBuckets: true
    serverSideEncryptionConfiguration:
      rules:
        - applyServerSideEncryptionByDefault:
            sseAlgorithm: AES256
    tagging:
      tagSet:
        - key: Billing
          value: billing
        - key: Environment
          value: environment
        - key: Owner
          value: owner
        - key: Team
          value: team
  providerConfigRef:
    name: "awsconfig"

In the AWS console, a bucket is created as expected, but the CRD does not reconcile successfully. If I switch the provider-aws version to v0.47.0, everything works as expected and the CRDs reconciles successfully.

I have a suspicion that a bug might have been introduced with this change #2018 that went out in v0.47.1

How can we reproduce it?

Create an S3 bucket with tags using provider-aws version 0.47.1 and check to see if the CRD shows synced = true and ready = true. If you create a bucket without tags or remove the tags from the yaml of the failed bucket, the CRD will show synced = true and ready = true. If you create a bucket with no tags, and then later on add tags, it will fail to become synced = true and throw the error above.

What environment did it happen in?

Crossplane version: 1.14.0
provider-aws version: 0.47.1
EKS 1.27