Add set-bucket-replication action

[JohnHillegass]

Add set replication action for s3 so we can take action on data exfil threats. A use case for this is:

            policies:
              - name: s3-unapproved-account-replication
                resource: s3
                filters:
                  - type: value
                    key: Replication.ReplicationConfiguration.Rules[].Destination.Account
                    value: present
                  - type: value
                    key: Replication.ReplicationConfiguration.Rules[].Destination.Account
                    value_from:
                      url: 's3:///path/to/file.json'
                      format: json
                      expr: "approved_accounts.*"
                    op: ni
                actions:
                  - type: set-replication
                    state: remove

[JohnHillegass]

Hey @kapilt, I wanted to get this in front of you early. I'll be adding test updates in a few moments. How do you feel about the general approach? This is my first action addition so want to make sure I am considering all of the things.

[kapilt]

your getting a ci failure because the permissions you've got on the action aren't valid, see the iam policy generator its s3:GetReplicationConfiguration etc

[JohnHillegass]

your getting a ci failure because the permissions you've got on the action aren't valid, see the iam policy generator its s3:GetReplicationConfiguration etc

Thanks @kapilt! That was a real head scratcher. Looks like I misread the boto docs.

[kapilt]

your getting a ci failure because the permissions you've got on the action aren't valid, see the iam policy generator its s3:GetReplicationConfiguration etc

Thanks @kapilt! That was a real head scratcher. Looks like I misread the boto docs.

iam stuff isn't actually documented in the sdks, its sometimes the method name, sometimes its oddball, been trying to work with some upstream folks to get this stuff in a machine readable format.

[JohnHillegass]

For the codecov check 20.69% of diff hit (target 89.69%), does this mean that only 20% of what I added is covered in tests and it should be 90+?

[kapilt]

In a nutshell yes, atm there aren’t any tests in this pr

[JohnHillegass]

@kapilt I was unfamiliar with how placebo works but was able to find some good guidance in the custodian docs. All tests pass now. Thanks for the guidance on this!

[kapilt]

please sanitize recorded data of any account ids, for s3 in particular it maybe useful to use a different source to minimize the amount of recorded data.

[kapilt]

@kapilt I believe this is good to go now. Thanks for your feedback along the way. I'll make sure to incorporate it into future contributions.

@JohnHillegass if you have a chance, given you've got a fresh perspective on it, i'd appreciate if you could have a look at the developer docs (https://github.com/cloud-custodian/cloud-custodian/tree/master/docs/source/developer) to see if there's any improvements/additions that should be made to make the contributing/onboarding process easier. thanks

[JohnHillegass]

@kapilt I believe this is good to go now. Thanks for your feedback along the way. I'll make sure to incorporate it into future contributions.

@JohnHillegass if you have a chance, given you've got a fresh perspective on it, i'd appreciate if you could have a look at the developer docs (https://github.com/cloud-custodian/cloud-custodian/tree/master/docs/source/developer) to see if there's any improvements/additions that should be made to make the contributing/onboarding process easier. thanks

Sure thing! It has been a journey with many learnings for me :) I'd be glad to take a look and fill in any areas that might need additional clarity.

[kapilt]

lgtm, thanks

[Ykamdar]

@kapilt finding it difficult to wrap my head around this policy definition.

What does this part in policy mean -

- type: value key: Replication.ReplicationConfiguration.Rules[].Destination.Account value: present - type: value key: Replication.ReplicationConfiguration.Rules[].Destination.Account value_from: url: 's3:///path/to/file.json' format: json expr: "approved_accounts.*" op: ni

Is there a way to get this policy for a specific bucket only, i am trying to implement this in one of my AWS accounts wherein i only want to enable replication for certain buckets but i dont see the option to specify the bucket name anywhere here in the policy file.

Can you please guide ? This is crucial for me and i have really tried and failed to understand this particular thing.