s3_bucket fails if bucket does not exist yet #707

tkintscher · 2022-03-11T22:27:33Z

Summary

I am trying to create an S3 bucket on AWS, if it does not exist yet.

This is my task:

- name: Setup S3 bucket
  amazon.aws.s3_bucket:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECRET_KEY }}"
    region: "{{ aws_region }}"
    name: "{{ s3_bucket_name }}"
    versioning: no
    state: present

It fails if the bucket does not exist yet (according to the docs, a bucket should be created in this case).
If the bucket already exists, the task completes successfully.

I suspect that it is due to this change:
#357

For the case of a non-existent bucket, the ListBucket command returns an empty list.
The HeadBucket command returns 403 in this case, but the code checks for 404.
(Note: I gave my IAM user full admin rights, which include the list-bucket permissions necessary to run HeadBucket).

Would someone be able to confirm that the new code indeed works to create buckets that did not exist before?

Issue Type

Bug Report

Component Name

s3_bucket

Ansible Version

$ ansible --version
ansible [core 2.12.2]
  config file = /Users/thomas/test/playbooks/host1/ansible.cfg
  configured module search path = ['/Users/thomas/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/thomas/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.2 (main, Feb  2 2022, 06:19:27) [Clang 13.0.0 (clang-1300.0.29.3)]
  jinja version = 3.0.3
  libyaml = True

Collection Versions

$ ansible-galaxy collection list


# /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0
ansible.netcommon             2.5.0
ansible.posix                 1.3.0
ansible.utils                 2.4.3
ansible.windows               1.9.0
arista.eos                    3.1.0
awx.awx                       19.4.0
azure.azcollection            1.11.0
check_point.mgmt              2.2.2
chocolatey.chocolatey         1.1.0
cisco.aci                     2.1.0
cisco.asa                     2.1.0
cisco.intersight              1.0.18
cisco.ios                     2.6.0
cisco.iosxr                   2.6.0
cisco.ise                     1.2.1
cisco.meraki                  2.6.0
cisco.mso                     1.3.0
cisco.nso                     1.0.3
cisco.nxos                    2.8.2
cisco.ucs                     1.6.0
cloud.common                  2.1.0
cloudscale_ch.cloud           2.2.0
community.aws                 2.2.0
community.azure               1.1.0
community.ciscosmb            1.0.4
community.crypto              2.2.0
community.digitalocean        1.15.0
community.dns                 2.0.6
community.docker              2.1.1
community.fortios             1.0.0
community.general             4.4.0
community.google              1.0.0
community.grafana             1.3.0
community.hashi_vault         2.2.0
community.hrobot              1.2.2
community.kubernetes          2.0.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.2
community.mysql               2.3.3
community.network             3.0.0
community.okd                 2.1.0
community.postgresql          1.6.1
community.proxysql            1.3.1
community.rabbitmq            1.1.0
community.routeros            2.0.0
community.skydive             1.0.0
community.sops                1.2.0
community.vmware              1.17.1
community.windows             1.9.0
community.zabbix              1.5.1
containers.podman             1.9.1
cyberark.conjur               1.1.0
cyberark.pas                  1.0.13
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            4.4.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.14.0
fortinet.fortimanager         2.1.4
fortinet.fortios              2.1.3
frr.frr                       1.0.3
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.6.0
hpe.nimble                    1.1.4
ibm.qradar                    1.0.3
infinidat.infinibox           1.3.3
infoblox.nios_modules         1.2.1
inspur.sm                     1.3.0
junipernetworks.junos         2.8.0
kubernetes.core               2.2.3
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.13.0
netapp.elementsw              21.7.0
netapp.ontap                  21.15.1
netapp.storagegrid            21.9.0
netapp.um_info                21.8.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.5.1
ngine_io.cloudstack           2.2.2
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.6.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   1.6.6
purestorage.flasharray        1.12.1
purestorage.flashblade        1.9.0
sensu.sensu_go                1.13.0
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.27.0
theforeman.foreman            2.2.0
vyos.vyos                     2.6.0
wti.remote                    1.0.3

# /Users/thomas/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
amazon.aws       3.1.1
community.aws    3.1.0
community.docker 2.2.0

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.20.46
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.23.46
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
HOST_KEY_CHECKING(/Users/thomas/test/playbooks/host1/ansible.cfg) = False

OS / Environment

MacOS 12.2.1, ansible installed using brew

Steps to Reproduce

---
# Let's list all buckets with the ListBucket command
- name: List S3 buckets
  community.aws.aws_s3_bucket_info:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECRET_KEY }}"
    name: "some-bucket-name"
  register: s3_buckets

# Should be an empty list if the bucket does not exist
- name: Print info
  debug:
    msg: "{{ s3_buckets.buckets }}"

# This uses the HeadBucket command and fails
- name: Setup S3 bucket
  amazon.aws.s3_bucket:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECRET_KEY }}"
    region: "{{ aws_region }}"
    name: "some-bucket-name"
    state: present

Expected Results

I expected the task to complete ok.

It works, if the bucket already exists.
If fails, if the bucket does not exist yet, although the documentation states that a new bucket should be created.

Actual Results

The full traceback is:
Traceback (most recent call last):
  File "/var/folders/_b/h67l22w15_93p1fj81z8fxfc0000gn/T/ansible_amazon.aws.s3_bucket_payload_mzgkhq4c/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 372, in create_or_update_bucket
  File "/var/folders/_b/h67l22w15_93p1fj81z8fxfc0000gn/T/ansible_amazon.aws.s3_bucket_payload_mzgkhq4c/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 625, in bucket_exists
  File "/usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden
fatal: [localhost]: FAILED! => {
    "boto3_version": "1.20.46",
    "botocore_version": "1.23.46",
    "changed": false,
    "error": {
        "code": "403",
        "message": "Forbidden"
    },
    "invocation": {
        "module_args": {
            "acl": null,
            "aws_access_key": "---CENSORED---",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "ceph": false,
            "debug_botocore_endpoint_logs": true,
            "delete_object_ownership": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "some-bucket-name",
            "object_ownership": null,
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": "eu-west-1",
            "requester_pays": null,
            "s3_url": null,
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "versioning": false
        }
    },
    "msg": "Failed to check bucket presence: An error occurred (403) when calling the HeadBucket operation: Forbidden",
    "resource_actions": [
        "s3:HeadBucket"
    ],
    "response_metadata": {
        "host_id": "cgw0AZfbYormenP0FnQHV947/AnHAvlFsiW8aai4sSCkGjoE66qZVJlc9gg7pfGb3aZwlAL0iKk=",
        "http_headers": {
            "content-type": "application/xml",
            "date": "Fri, 11 Mar 2022 22:10:19 GMT",
            "server": "AmazonS3",
            "x-amz-bucket-region": "eu-west-1",
            "x-amz-id-2": "---CENSORED---",
            "x-amz-request-id": "---CENSORED---"
        },
        "http_status_code": 403,
        "request_id": "ZAWD12VZRHQAGAYH",
        "retry_attempts": 0
    }
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

tkintscher · 2022-03-11T22:35:48Z

As always, one realizes after hitting the Submit button...

The bucket name must be globally unique.
If HeadBucket returns '403', it's either due to insufficient permissions for the IAM user, or because the bucket name is already taken by someone else.

…-version cloudfront_distribution: add new security policy version TLSv1.2_2021 SUMMARY Add new TLSv1.2_2021 security policy for CloudFront viewer connections. ISSUE TYPE Feature Pull Request COMPONENT NAME cloudfront_distribution ADDITIONAL INFORMATION https://aws.amazon.com/about-aws/whats-new/2021/06/amazon-cloudfront-announces-new-tlsv12_2021-security-policy-for-viewer-connections/ Reviewed-by: Alina Buzachis <None> Reviewed-by: Markus Bergholz <git@osuv.de> Reviewed-by: None <None>

tkintscher closed this as completed Mar 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3_bucket fails if bucket does not exist yet #707

s3_bucket fails if bucket does not exist yet #707

tkintscher commented Mar 11, 2022

tkintscher commented Mar 11, 2022

s3_bucket fails if bucket does not exist yet #707

s3_bucket fails if bucket does not exist yet #707

Comments

tkintscher commented Mar 11, 2022

Summary

Issue Type

Component Name

Ansible Version

Collection Versions

AWS SDK versions

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

tkintscher commented Mar 11, 2022