aws - ebs - InvalidVolume.NotFound when calling the CreateTags operation

[nitrocode]

Describe the bug

The following exception is thrown

An error occurred (InvalidVolume.NotFound) when calling the CreateTags operation

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Background (please complete the following information):

• OS: lambda
• Python Version: 3.8.1
• Custodian Version: 0.9.4.0
• Tool Version: N/A
• Cloud Provider: aws
• Policy:

vars:
  mode: &mode
    type: periodic
    schedule: "rate(7 days)"
    role: arn:aws:iam::{account_id}:role/cloud-custodian
    tags:
      application: cloud-custodian
      team: sre
      env: production
    execution-options:
      output_dir: s3://output-bucket/cloud-custodian/output

policies:
- name: ebs-mark-unattached-deletion
  resource: ebs
  description: |
    Mark any unattached EBS volumes for deletion in 30 days.
    Volumes set to not delete on instance termination do have
    valid use cases as data drives, but 99% of the time they
    appear to be just garbage creation.
  mode:
    <<: *mode
  filters:
    - "tag:c7n_ignore": absent
    - "tag:c7n_status": absent
    - Attachments: []
    - type: mark-for-op
      tag: c7n_status
      msg: "Unused. If this resource should stay, add the tag c7n_ignore with a value of why it should exist."
      op: delete
      days: 30

• Traceback:

[ERROR] ClientError: An error occurred (InvalidVolume.NotFound) when calling the CreateTags operation: The volume 'vol-05eeb39ae0ab38aa5' does not exist.
Traceback (most recent call last):
  File "/var/task/custodian_policy.py", line 4, in run
    return handler.dispatch_event(event, context)
  File "/var/task/c7n/handler.py", line 176, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 1135, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 532, in run
    return PullMode.run(self)
  File "/var/task/c7n/policy.py", line 323, in run
    results = a.process(resources)
  File "/var/task/c7n/tags.py", line 682, in process
    _common_tag_processer(
  File "/var/task/c7n/tags.py", line 133, in _common_tag_processer
    raise error
  File "/var/lang/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/var/task/c7n/tags.py", line 688, in process_resource_set
    tagger.process_resource_set(client, resource_set, tags)
  File "/var/task/c7n/tags.py", line 430, in process_resource_set
    self.manager.retry(
  File "/var/task/c7n/utils.py", line 373, in _retry
    return func(*args, **kw)
  File "/var/runtime/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)

• custodian version --debug output

Additional context
The workaround for now has been to pin down cloudcustodian to 0.9.1.0.

#5375
#3531

[kapilt]

do you have the full traceback?

[nitrocode]

Yes. I modified the ticket to add it.

[ajkerrigan]

This came up in Gitter today also, courtesy of @cpitstick-argo :

We run a lot of spot instances that wink in and out constantly. So sometimes, Cloud Custodian will attempt to tag a resource that doesn't exist, because in between the time it discovered the resource and tried to tag it, the tagging operation fails, so we get a lot of these types of errors:
An error occurred (InvalidVolume.NotFound) when calling the CreateTags operation: The volume 'vol-XXXXXXXXXXXXX' does not exist.

With a similar sample policy:

- name: ebs-tag-unattached-volume
  resource: ebs
  description: |
    EBS volume is unattached
  filters:
    - Attachments: []
  actions:
    - type: tag
      key: "cloud-custodian-flagged"
      value: "ebs-volume-is-unattached"
    - type: mark-for-op
      tag: cloud-custodian-action
      op: delete
      hours: 0