Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resolver - support decompression when using value_from with s3 #8851

Merged
merged 2 commits into from Aug 31, 2023

Conversation

huynhsontung
Copy link
Contributor

@huynhsontung huynhsontung commented Aug 18, 2023

Add support for value decompression with value_from parameter of a filter when the url is an S3 URI that ends with .gz, .zip, or .gzip

This feature bridges the gap between Custodian output to S3 which is gzip compressed by default where the value_from doesn't support gzip decompression.

value_from already supports decompression for non-s3 URLs. This change handles decompression for S3 URLs as well.

The following policy will be valid after this change:

policies:
  - name: list-in-use-images-master
    resource: ami
    filters:
      - type: value
        key: "ImageId"
        op: not-in
        value_from:
          format: json
          expr: "[].ImageId"
          url: s3://custodian-integrations/reports/<aws-account>/us-east-2/list-in-use-images/2023/08/18/19/resources.json.gz

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Aug 18, 2023

CLA Signed

The committers listed above are authorized under a signed CLA.

Signed-off-by: Tung Huynh <tung75605@gmail.com>
@huynhsontung huynhsontung marked this pull request as ready for review August 18, 2023 23:55
Signed-off-by: Tung Huynh <tung75605@gmail.com>
@huynhsontung huynhsontung changed the title feat: support decompression when using value_from with s3 resolver - support decompression when using value_from with s3 Aug 19, 2023
Copy link
Collaborator

@kapilt kapilt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the pr, lgtm

@kapilt kapilt merged commit 7f4689d into cloud-custodian:main Aug 31, 2023
20 of 21 checks passed
@huynhsontung huynhsontung deleted the from-value-s3-gzip branch August 31, 2023 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants