Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3: resolve bucket ARN in replication config references #763

Merged
merged 4 commits into from Nov 16, 2021

Conversation

chlunde
Copy link
Collaborator

@chlunde chlunde commented Jul 7, 2021

Description of your changes

The replication destination bucket should resolve to an ARN, not a bucket name.

Fixes #745

I have:

  • Read and followed Crossplane's contribution process.
  • Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

With blood, sweat and tears. Fixed three other issues first.

This is a "minimal" configuration (excluding role and policy):

---
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/external-name: repl-src.2021-06-07
  name: repl-src
spec:
  deletionPolicy: Delete
  forProvider:
    acl: private
    locationConstraint: eu-north-1
    paymentConfiguration:
      payer: BucketOwner
    serverSideEncryptionConfiguration:
      rules:
      - applyServerSideEncryptionByDefault:
          sseAlgorithm: AES256
    versioningConfiguration:
      status: Enabled
    replicationConfiguration:
      role: arn:aws:iam::...:role/s3-replication-test # assumable by s3 svc, I just tested with an s3 admin role
      rules:
      - destination:
          storageClass: STANDARD
          bucketRef:
            name: repl-dest
        deleteMarkerReplication:
          status: Disabled
        priority: 0
        id: rule-1
        status: Enabled
---
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/external-name: repl-dest.2021-06-07
  name: repl-dest
spec:
  deletionPolicy: Delete
  forProvider:
    acl: private
    locationConstraint: eu-north-1
    paymentConfiguration:
      payer: BucketOwner
    serverSideEncryptionConfiguration:
      rules:
      - applyServerSideEncryptionByDefault:
          sseAlgorithm: AES256
    versioningConfiguration:
      status: Enabled

Question: Do you prefer the function returning a function for BucketARN?

Copy link
Collaborator

@AaronME AaronME left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chlunde, thank you for addressing #745

I'm getting the following error in my tests:

"cannot create or update: cannot put Bucket replication: MalformedXML: The XML you provided was not well-formed or did not validate against our published schema"

I used the following configuration:

---
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: repl-src
spec:
  deletionPolicy: Delete
  forProvider:
    acl: private
    locationConstraint: us-west-2
    paymentConfiguration:
      payer: BucketOwner
    serverSideEncryptionConfiguration:
      rules:
      - applyServerSideEncryptionByDefault:
          sseAlgorithm: AES256
    versioningConfiguration:
      status: Enabled
    replicationConfiguration:
      role: arn:aws:iam::...:role/role-name # assumable by s3 svc, I just tested with an s3 admin role
      rules:
      - destination:
          storageClass: STANDARD_IA
          bucketRef:
            name: repl-dest
        deleteMarkerReplication:
          Status: Disabled
        priority: 0
        id: rule-1
        status: Enabled
  providerConfigRef:
    name: example
---
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: repl-dest
spec:
  deletionPolicy: Delete
  forProvider:
    acl: private
    locationConstraint: us-west-2
    paymentConfiguration:
      payer: BucketOwner
    serverSideEncryptionConfiguration:
      rules:
      - applyServerSideEncryptionByDefault:
          sseAlgorithm: AES256
    versioningConfiguration:
      status: Enabled
  providerConfigRef:
    name: example

@AaronME AaronME self-assigned this Aug 13, 2021
@chlunde
Copy link
Collaborator Author

chlunde commented Aug 13, 2021

Thanks for testing and reviewing this and the other PRs! I think you need a filter, or the filter PR, but I'm not certain.

I can check on monday.

Copy link
Collaborator

@AaronME AaronME left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chlunde, after merging the filter fix I successfully created the replicating bucket pair. Nice work!

Before merging, could you please:

  1. Include the typo fix from s3: fix deleteMarkerReplication.Status -> deleteMarkerReplication.status #766 in this PR.
  2. Update examples/s3/ with your test cases so we can run conformance testing on these changes.
  3. Rebase on latest master.

Thanks again for this excellent contribution!

The replication destination bucket should resolve to an ARN, not a bucket name.

Fixes crossplane-contrib#745

Signed-off-by: Carl Henrik Lunde <chlunde@ifi.uio.no>
Signed-off-by: Carl Henrik Lunde <chlunde@ifi.uio.no>
@chlunde chlunde self-assigned this Sep 7, 2021
The JSON/YAML field name had an upper case first letter, which is not
consistent with any other resource.

Signed-off-by: Carl Henrik Lunde <chlunde@ifi.uio.no>
@chlunde chlunde force-pushed the s3-bucket-arn branch 4 times, most recently from 479549f to 07c71ef Compare September 7, 2021 20:45
@chlunde
Copy link
Collaborator Author

chlunde commented Sep 7, 2021

I'm back to malformed policy unless i specify a filter, I don't understand why, because the SDK version is the same as before. Added filter to the example.

@chlunde chlunde removed their assignment Sep 7, 2021
@chlunde chlunde requested a review from AaronME September 7, 2021 20:52
Signed-off-by: Carl Henrik Lunde <chlunde@ifi.uio.no>
Copy link
Collaborator

@AaronME AaronME left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chlunde This works for me!

Thanks!

Comment on lines +59 to +60
filter:
prefix: ""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chlunde This works for me both with and without this filter.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, do you want me to remove it or keep it as it is?

@muvaf muvaf merged commit 892b7a0 into crossplane-contrib:master Nov 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

S3 Bucket replication rules for bucket ref resolves to external name instead of bucket ARN
3 participants