Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new module: s3_object_info #977

Merged

Conversation

mandar242
Copy link
Contributor

@mandar242 mandar242 commented Aug 24, 2022

SUMMARY

Adding a new module for retrieving s3 object info.

Currently supports following boto APIs
get_object()
get_object_acl()
get_object_attributes()
get_object_legal_hold()
get_object_lock_configuration()
get_object_retention()
get_object_tagging()

ISSUE TYPE
  • New Module Pull Request
COMPONENT NAME

s3_object_info

ADDITIONAL INFORMATION

Sample Tasks:

    - name: Get object info
      amazon.aws.s3_object_info:
        bucket_name: "{{ bucket_name }}"
        object_key: "{{ object_key }}"

    - name: Get object info details
      amazon.aws.s3_object_info:
        bucket_name: "{{ bucket_name }}"
        object_key: "{{ object_key }}"
        object_details:
          object_acl: true
          object_tagging: true
          object_attributes: true
          object_retention: true
        object_attributes:
          - ETag
          - ObjectSize
          - StorageClass

@github-actions

This comment was marked as outdated.

@ansibullbot
Copy link

@ansibullbot ansibullbot added WIP Work in progress integration tests/integration module module new_plugin New plugin plugins plugin (any type) tests tests labels Aug 24, 2022
@softwarefactory-project-zuul

This comment was marked as outdated.

@softwarefactory-project-zuul

This comment was marked as outdated.

@softwarefactory-project-zuul

This comment was marked as outdated.

plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
tests/integration/targets/s3_object/meta/main.yml Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
@softwarefactory-project-zuul

This comment was marked as outdated.

@softwarefactory-project-zuul

This comment was marked as outdated.

@mandar242 mandar242 changed the title [WIP] new module: s3_object_info new module: s3_object_info Aug 29, 2022
@ansibullbot ansibullbot added community_review and removed WIP Work in progress labels Aug 29, 2022
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Show resolved Hide resolved
plugins/modules/s3_object_info.py Outdated Show resolved Hide resolved
plugins/modules/s3_object_info.py Show resolved Hide resolved
@ansibullbot ansibullbot added community_review and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR labels Sep 23, 2022
@softwarefactory-project-zuul
Copy link
Contributor

Build failed.

✔️ ansible-galaxy-importer SUCCESS in 5m 14s
✔️ build-ansible-collection SUCCESS in 5m 29s
ansible-test-sanity-aws-ansible-python38 FAILURE in 9m 52s
ansible-test-sanity-aws-ansible-2.12-python38 FAILURE in 10m 34s
ansible-test-sanity-aws-ansible-2.13-python38 FAILURE in 9m 19s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 6m 14s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 6m 58s
✔️ ansible-test-splitter SUCCESS in 2m 35s
integration-amazon.aws-1 FAILURE in 6m 42s
⚠️ integration-amazon.aws-2 SKIPPED
⚠️ integration-amazon.aws-3 SKIPPED
⚠️ integration-amazon.aws-4 SKIPPED
⚠️ integration-amazon.aws-5 SKIPPED
⚠️ integration-amazon.aws-6 SKIPPED
⚠️ integration-amazon.aws-7 SKIPPED
⚠️ integration-amazon.aws-8 SKIPPED
⚠️ integration-amazon.aws-9 SKIPPED
⚠️ integration-amazon.aws-10 SKIPPED
⚠️ integration-amazon.aws-11 SKIPPED
⚠️ integration-amazon.aws-12 SKIPPED
⚠️ integration-amazon.aws-13 SKIPPED
⚠️ integration-amazon.aws-14 SKIPPED
⚠️ integration-amazon.aws-15 SKIPPED
⚠️ integration-amazon.aws-16 SKIPPED
⚠️ integration-amazon.aws-17 SKIPPED
⚠️ integration-amazon.aws-18 SKIPPED
⚠️ integration-community.aws-1 SKIPPED
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED
⚠️ integration-community.aws-14 SKIPPED
⚠️ integration-community.aws-15 SKIPPED
⚠️ integration-community.aws-16 SKIPPED
⚠️ integration-community.aws-17 SKIPPED
⚠️ integration-community.aws-18 SKIPPED
✔️ ansible-test-changelog SUCCESS in 2m 18s

return False


def get_s3_connection(module, aws_connect_kwargs, location, ceph, endpoint_url, sig_4=False):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can keep it as it is for the moment. For the future, we should move this in module_utils instead of introducing redundancy and refactor the other s3 modules as needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Makes sense.
Once this (and other s3 PRs if any) are merged, I can work on refactoring this.

@alinabuzachis
Copy link
Collaborator

recheck

Copy link
Collaborator

@alinabuzachis alinabuzachis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.

✔️ ansible-galaxy-importer SUCCESS in 4m 45s
✔️ build-ansible-collection SUCCESS in 5m 57s
✔️ ansible-test-sanity-aws-ansible-python38 SUCCESS in 9m 47s
✔️ ansible-test-sanity-aws-ansible-2.12-python38 SUCCESS in 11m 13s
✔️ ansible-test-sanity-aws-ansible-2.13-python38 SUCCESS in 11m 58s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 6m 45s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 7m 38s
✔️ ansible-test-splitter SUCCESS in 3m 04s
✔️ integration-amazon.aws-1 SUCCESS in 12m 16s
⚠️ integration-amazon.aws-2 SKIPPED
⚠️ integration-amazon.aws-3 SKIPPED
⚠️ integration-amazon.aws-4 SKIPPED
⚠️ integration-amazon.aws-5 SKIPPED
⚠️ integration-amazon.aws-6 SKIPPED
⚠️ integration-amazon.aws-7 SKIPPED
⚠️ integration-amazon.aws-8 SKIPPED
⚠️ integration-amazon.aws-9 SKIPPED
⚠️ integration-amazon.aws-10 SKIPPED
⚠️ integration-amazon.aws-11 SKIPPED
⚠️ integration-amazon.aws-12 SKIPPED
⚠️ integration-amazon.aws-13 SKIPPED
⚠️ integration-amazon.aws-14 SKIPPED
⚠️ integration-amazon.aws-15 SKIPPED
⚠️ integration-amazon.aws-16 SKIPPED
⚠️ integration-amazon.aws-17 SKIPPED
⚠️ integration-amazon.aws-18 SKIPPED
⚠️ integration-community.aws-1 SKIPPED
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED
⚠️ integration-community.aws-14 SKIPPED
⚠️ integration-community.aws-15 SKIPPED
⚠️ integration-community.aws-16 SKIPPED
⚠️ integration-community.aws-17 SKIPPED
⚠️ integration-community.aws-18 SKIPPED
✔️ ansible-test-changelog SUCCESS in 2m 49s

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.

✔️ ansible-galaxy-importer SUCCESS in 3m 39s
✔️ build-ansible-collection SUCCESS in 5m 33s
✔️ ansible-test-sanity-aws-ansible-python38 SUCCESS in 10m 43s
✔️ ansible-test-sanity-aws-ansible-2.12-python38 SUCCESS in 9m 01s
✔️ ansible-test-sanity-aws-ansible-2.13-python38 SUCCESS in 9m 08s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 6m 31s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 6m 41s
✔️ ansible-test-splitter SUCCESS in 2m 50s
✔️ integration-amazon.aws-1 SUCCESS in 9m 40s
⚠️ integration-amazon.aws-2 SKIPPED
⚠️ integration-amazon.aws-3 SKIPPED
⚠️ integration-amazon.aws-4 SKIPPED
⚠️ integration-amazon.aws-5 SKIPPED
⚠️ integration-amazon.aws-6 SKIPPED
⚠️ integration-amazon.aws-7 SKIPPED
⚠️ integration-amazon.aws-8 SKIPPED
⚠️ integration-amazon.aws-9 SKIPPED
⚠️ integration-amazon.aws-10 SKIPPED
⚠️ integration-amazon.aws-11 SKIPPED
⚠️ integration-amazon.aws-12 SKIPPED
⚠️ integration-amazon.aws-13 SKIPPED
⚠️ integration-amazon.aws-14 SKIPPED
⚠️ integration-amazon.aws-15 SKIPPED
⚠️ integration-amazon.aws-16 SKIPPED
⚠️ integration-amazon.aws-17 SKIPPED
⚠️ integration-amazon.aws-18 SKIPPED
⚠️ integration-community.aws-1 SKIPPED
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED
⚠️ integration-community.aws-14 SKIPPED
⚠️ integration-community.aws-15 SKIPPED
⚠️ integration-community.aws-16 SKIPPED
⚠️ integration-community.aws-17 SKIPPED
⚠️ integration-community.aws-18 SKIPPED
✔️ ansible-test-changelog SUCCESS in 2m 15s

@mandar242 mandar242 added the mergeit Merge the PR (SoftwareFactory) label Sep 27, 2022
@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.

✔️ ansible-galaxy-importer SUCCESS in 9m 48s
✔️ build-ansible-collection SUCCESS in 8m 49s
✔️ ansible-test-sanity-aws-ansible-python38 SUCCESS in 18m 05s
✔️ ansible-test-sanity-aws-ansible-2.12-python38 SUCCESS in 27m 24s
✔️ ansible-test-sanity-aws-ansible-2.13-python38 SUCCESS in 18m 06s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 10m 58s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 9m 37s
✔️ ansible-test-splitter SUCCESS in 5m 47s
✔️ integration-amazon.aws-1 SUCCESS in 25m 50s
⚠️ integration-amazon.aws-2 SKIPPED
⚠️ integration-amazon.aws-3 SKIPPED
⚠️ integration-amazon.aws-4 SKIPPED
⚠️ integration-amazon.aws-5 SKIPPED
⚠️ integration-amazon.aws-6 SKIPPED
⚠️ integration-amazon.aws-7 SKIPPED
⚠️ integration-amazon.aws-8 SKIPPED
⚠️ integration-amazon.aws-9 SKIPPED
⚠️ integration-amazon.aws-10 SKIPPED
⚠️ integration-amazon.aws-11 SKIPPED
⚠️ integration-amazon.aws-12 SKIPPED
⚠️ integration-amazon.aws-13 SKIPPED
⚠️ integration-amazon.aws-14 SKIPPED
⚠️ integration-amazon.aws-15 SKIPPED
⚠️ integration-amazon.aws-16 SKIPPED
⚠️ integration-amazon.aws-17 SKIPPED
⚠️ integration-amazon.aws-18 SKIPPED
⚠️ integration-community.aws-1 SKIPPED
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED
⚠️ integration-community.aws-14 SKIPPED
⚠️ integration-community.aws-15 SKIPPED
⚠️ integration-community.aws-16 SKIPPED
⚠️ integration-community.aws-17 SKIPPED
⚠️ integration-community.aws-18 SKIPPED
✔️ ansible-test-changelog SUCCESS in 4m 23s

@mandar242 mandar242 added mergeit Merge the PR (SoftwareFactory) and removed mergeit Merge the PR (SoftwareFactory) labels Sep 28, 2022
@mandar242
Copy link
Contributor Author

regate

@alinabuzachis alinabuzachis dismissed stale reviews from goneri and tremble September 28, 2022 09:09

Requested changes have been addressed. It hangs on gating.

@alinabuzachis
Copy link
Collaborator

regate

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded (gate pipeline).

✔️ ansible-galaxy-importer SUCCESS in 4m 49s
✔️ build-ansible-collection SUCCESS in 5m 17s
✔️ ansible-test-sanity-aws-ansible-python38 SUCCESS in 10m 09s
✔️ ansible-test-sanity-aws-ansible-2.12-python38 SUCCESS in 9m 31s
✔️ ansible-test-sanity-aws-ansible-2.13-python38 SUCCESS in 9m 31s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 6m 08s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 5m 32s
✔️ ansible-test-splitter SUCCESS in 2m 48s
✔️ integration-amazon.aws-1 SUCCESS in 9m 30s
⚠️ integration-amazon.aws-2 SKIPPED
⚠️ integration-amazon.aws-3 SKIPPED
⚠️ integration-amazon.aws-4 SKIPPED
⚠️ integration-amazon.aws-5 SKIPPED
⚠️ integration-amazon.aws-6 SKIPPED
⚠️ integration-amazon.aws-7 SKIPPED
⚠️ integration-amazon.aws-8 SKIPPED
⚠️ integration-amazon.aws-9 SKIPPED
⚠️ integration-amazon.aws-10 SKIPPED
⚠️ integration-amazon.aws-11 SKIPPED
⚠️ integration-amazon.aws-12 SKIPPED
⚠️ integration-amazon.aws-13 SKIPPED
⚠️ integration-amazon.aws-14 SKIPPED
⚠️ integration-amazon.aws-15 SKIPPED
⚠️ integration-amazon.aws-16 SKIPPED
⚠️ integration-amazon.aws-17 SKIPPED
⚠️ integration-amazon.aws-18 SKIPPED
⚠️ integration-community.aws-1 SKIPPED
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED
⚠️ integration-community.aws-14 SKIPPED
⚠️ integration-community.aws-15 SKIPPED
⚠️ integration-community.aws-16 SKIPPED
⚠️ integration-community.aws-17 SKIPPED
⚠️ integration-community.aws-18 SKIPPED
✔️ ansible-test-changelog SUCCESS in 2m 29s

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit 4d0ac2f into ansible-collections:main Sep 28, 2022
@mandar242 mandar242 deleted the s3_object_info branch May 9, 2023 02:46
abikouo pushed a commit to abikouo/amazon.aws that referenced this pull request Sep 18, 2023
Add backoff logic to elb_application_lb_info

SUMMARY
From time to time rate limiting failures occur on the usage of this module, this PR adds backoff logic to the module to improve its stability.
fatal: [127.0.0.1 -> 127.0.0.1]: FAILED! => changed=false 
boto3_version: 1.20.34
botocore_version: 1.23.34
error:
code: Throttling
message: Rate exceeded
type: Sender
msg: 'Failed to list load balancers: An error occurred (Throttling) when calling the DescribeLoadBalancers operation (reached max retries: 4): Rate exceeded'
response_metadata:
http_headers:
content-length: '271'
content-type: text/xml
date: Thu, 10 Mar 2022 10:34:23 GMT
x-amzn-requestid: xxxxx
http_status_code: 400
max_attempts_reached: true
request_id: xxxxx
retry_attempts: 4

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
elb_application_lb_info
ADDITIONAL INFORMATION

Reviewed-by: Markus Bergholz <git@osuv.de>
Reviewed-by: Alina Buzachis <None>
abikouo pushed a commit to abikouo/amazon.aws that referenced this pull request Sep 18, 2023
Add backoff logic to elb_application_lb_info

SUMMARY
From time to time rate limiting failures occur on the usage of this module, this PR adds backoff logic to the module to improve its stability.
fatal: [127.0.0.1 -> 127.0.0.1]: FAILED! => changed=false 
boto3_version: 1.20.34
botocore_version: 1.23.34
error:
code: Throttling
message: Rate exceeded
type: Sender
msg: 'Failed to list load balancers: An error occurred (Throttling) when calling the DescribeLoadBalancers operation (reached max retries: 4): Rate exceeded'
response_metadata:
http_headers:
content-length: '271'
content-type: text/xml
date: Thu, 10 Mar 2022 10:34:23 GMT
x-amzn-requestid: xxxxx
http_status_code: 400
max_attempts_reached: true
request_id: xxxxx
retry_attempts: 4

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
elb_application_lb_info
ADDITIONAL INFORMATION

Reviewed-by: Markus Bergholz <git@osuv.de>
Reviewed-by: Alina Buzachis <None>
abikouo pushed a commit to abikouo/amazon.aws that referenced this pull request Oct 24, 2023
Add backoff logic to elb_application_lb_info

SUMMARY
From time to time rate limiting failures occur on the usage of this module, this PR adds backoff logic to the module to improve its stability.
fatal: [127.0.0.1 -> 127.0.0.1]: FAILED! => changed=false 
boto3_version: 1.20.34
botocore_version: 1.23.34
error:
code: Throttling
message: Rate exceeded
type: Sender
msg: 'Failed to list load balancers: An error occurred (Throttling) when calling the DescribeLoadBalancers operation (reached max retries: 4): Rate exceeded'
response_metadata:
http_headers:
content-length: '271'
content-type: text/xml
date: Thu, 10 Mar 2022 10:34:23 GMT
x-amzn-requestid: xxxxx
http_status_code: 400
max_attempts_reached: true
request_id: xxxxx
retry_attempts: 4

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
elb_application_lb_info
ADDITIONAL INFORMATION

Reviewed-by: Markus Bergholz <git@osuv.de>
Reviewed-by: Alina Buzachis <None>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community_review integration tests/integration mergeit Merge the PR (SoftwareFactory) module module new_module New module new_plugin New plugin plugins plugin (any type) tests tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants