Operation Aborted but Subsequent Action Executed

[KISStian]

Describe the bug
While running a CloudTrail based policy to turn on default S3 bucket encryption, the operation aborted due to a conflicting operation already in progress. Rather than execute a retry or cause the lambda to fail, it continued running and executed the following/last action which was to send out a notification.

To Reproduce
Create a new S3 bucket and let the CloudTrail based policies (encrypt unencrypted buckets, apply bucket policy, and set public access block) run.

Expected behavior
The lambda will retry a few times, and if it still takes an exception, the lambda will terminate and not move on to subsequent actions.

Background (please complete the following information):

• Python Version: 3.8
• Custodian Version: 0.9.10
• Cloud Provider: AWS
• Policy: [please exclude any account/sensitive information]

################################################
# Remediate Default Bucket Encryption (AES256) #
################################################

- name: s3-remediate-unencrypted-bucket
  resource: s3
  region: us-east-1
  conditions:
  - not:
    - type: value
      key: account.tags[]
      op: contains
      value: "enforce:no"
  - not:
    - type: value
      key: account_id
      op: in
      value:
      ##################################################
      # The following list contains account exceptions #
      ##################################################
      # ProjectXX #
      #############
      - "xxxxxxxxxxxx"
  mode:
    <<: *mode-cloudtrail
    tags:
      <<: *mode-tags
    events:
    - CreateBucket
    - source: s3.amazonaws.com
      event: DeleteBucketEncryption
      ids: "requestParameters.bucketName" 
  filters:
  - type: bucket-encryption
    state: false
  - not:
    - type: value
      key: Name
      op: in
      value:
      #################################################
      # The following list contains bucket exceptions #
      #################################################
      # ProjectXX #
      #############
      - bucket-name
  actions:
  - type: set-bucket-encryption
    crypto: AES256
  - <<: *notify-users
    subject: "S3: Unencrypted Bucket - Account {{ account }} - Region {{ region }}"
    violation_desc:  "*WARN*The following S3 buckets have been identified as having default bucket encryption disabled:"
    action_desc: "Actions Taken: In order to comply with our organizational policies, default bucket encryption has been enabled and set to SSE-S3 (AES-256)."

Additional context
Run log:

021-01-20 20:53:03,500 - custodian.output - DEBUG - metric:ResourceCount Count:1 policy:s3-remediate-unencrypted-bucket restype:s3 scope:policy
2021-01-20 20:53:03,501 - custodian.policy - INFO - Invoking actions [<c7n.resources.s3.SetBucketEncryption object at 0x7f58e4c9c040>, <c7n.actions.notify.Notify object at 0x7f58e4c9cf40>]
2021-01-20 20:53:03,501 - custodian.policy - INFO - policy:s3-remediate-unencrypted-bucket invoking action:setbucketencryption resources:1
2021-01-20 20:53:03,735 - custodian.actions - ERROR - Message: An error occurred (OperationAborted) when calling the PutBucketEncryption operation: A conflicting conditional operation is currently in progress against this resource. Please try again. Bucket: xxxxxxxxxxxxxxx
2021-01-20 20:53:03,736 - custodian.policy - INFO - policy:s3-remediate-unencrypted-bucket invoking action:notify resources:1
2021-01-20 20:53:04,051 - custodian.actions - INFO - sent message:efaf9b7f-b05f-495e-96cc-4ea04c1a68da policy:s3-remediate-unencrypted-bucket template:email.html count:1
2021-01-20 20:53:04,052 - custodian.output - DEBUG - metric:ApiCalls Count:3 policy:s3-remediate-unencrypted-bucket restype:s3

[tjstansell]

Another case for better action error handling ... #3531. Doesn't seem right to completely kill the policy if a single resource fails. What should happen is those resources that failed should stop, but those that succeeded should continue.