s3_object get operations fail with "object not found"

[dsultanaabc]

Summary

On version 6.0.0, using the s3_object module to get objects from S3 buckets fails with "object not found". This was previously working using version 5.4.0.

Breaking configuration (NOTE: This configuration WORKS on 5.4.0):

BREAKING

  - name: 'Test retrieve'
    amazon.aws.s3_object:
      bucket: 'abc-sultana-bucket-test-1'
      object: '/test'
      dest: '/root/test'
      mode: 'get'

When removing the prefixed "/" on the object parameter, the get operation is successful.

SUCCESSFUL

  - name: 'Test retrieve'
    amazon.aws.s3_object:
      bucket: 'abc-sultana-bucket-test-1'
      object: 'test'
      dest: '/root/test'
      mode: 'get'

The documentation here - (https://ansible-collections.github.io/amazon.aws/branch/stable-6/collections/amazon/aws/s3_object_module.html#ansible-collections-amazon-aws-s3-object-module) - doesn't mention anything about removing the prefixed forward slash.

Error output:

fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "bucket": "abc-sultana-bucket-test-1",
            "ceph": false,
            "content": null,
            "content_base64": null,
            "copy_src": null,
            "debug_botocore_endpoint_logs": false,
            "dest": "/root/test",
            "dualstack": false,
            "encrypt": true,
            "encryption_kms_key_id": null,
            "encryption_mode": "AES256",
            "endpoint_url": null,
            "expiry": 600,
            "headers": null,
            "ignore_nonexistent_bucket": false,
            "marker": "",
            "max_keys": 1000,
            "metadata": null,
            "mode": "get",
            "object": "/test",
            "overwrite": "different",
            "permission": [
                "private"
            ],
            "prefix": "",
            "profile": null,
            "purge_tags": true,
            "region": null,
            "retries": 0,
            "secret_key": null,
            "session_token": null,
            "sig_v4": true,
            "src": null,
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "version": null
        }
    },
    "msg": "Key /test does not exist."
}

Issue Type

Bug Report

Component Name

s3_object

Ansible Version

ansible [core 2.12.10]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/splunk_venv/lib/python3.10/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /root/splunk_venv/bin/ansible
  python version = 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
  jinja version = 3.1.2
  libyaml = True

Collection Version
---------- -------
amazon.aws 6.0.0  

Collection Versions

$ ansible-galaxy collection list

Collection Version
---------- -------
amazon.aws 6.0.0  

AWS SDK versions

WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.133
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /root/splunk_venv/lib/python3.10/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.29.133
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /root/splunk_venv/lib/python3.10/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed

OS / Environment

PRETTY_NAME="Ubuntu 22.04.2 LTS"

Steps to Reproduce

Example breaking retrieve task:

---
  - name: 'Test retrieve'
    amazon.aws.s3_object:
      bucket: 'abc-sultana-bucket-test-1'
      object: '/test'
      dest: '/root/test'
      mode: 'get'

Expected Results

When I run the above task, I expect the get operation to succeed.

This configurations WORKS on 5.4.0. BREAKING on 6.0.0. Root cause seems to be leading "/" in object parameter. Documentation states it is required.

Actual Results

Broken results on 6.0.0 WITH leading "/" in object parameter:

TASK [bb_agent : Test retrieve] ************************************************************************************************************
task path: /home/ubuntu/ansible/roles/bb_agent/tasks/main.yml:26
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1684201268.117918-9926-38526686359550 `" && echo ansible-tmp-1684201268.117918-9926-38526686359550="` echo /root/.ansible/tmp/ansible-tmp-1684201268.117918-9926-38526686359550 `" ) && sleep 0'
Using module file /root/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_object.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-9822onw0be9o/tmpcwme31o0 TO /root/.ansible/tmp/ansible-tmp-1684201268.117918-9926-38526686359550/AnsiballZ_s3_object.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1684201268.117918-9926-38526686359550/ /root/.ansible/tmp/ansible-tmp-1684201268.117918-9926-38526686359550/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/root/splunk_venv/bin/python3.10 /root/.ansible/tmp/ansible-tmp-1684201268.117918-9926-38526686359550/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1684201268.117918-9926-38526686359550/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "bucket": "abc-sultana-bucket-test-1",
            "ceph": false,
            "content": null,
            "content_base64": null,
            "copy_src": null,
            "debug_botocore_endpoint_logs": false,
            "dest": "/root/test",
            "dualstack": false,
            "encrypt": true,
            "encryption_kms_key_id": null,
            "encryption_mode": "AES256",
            "endpoint_url": null,
            "expiry": 600,
            "headers": null,
            "ignore_nonexistent_bucket": false,
            "marker": "",
            "max_keys": 1000,
            "metadata": null,
            "mode": "get",
            "object": "/test",
            "overwrite": "different",
            "permission": [
                "private"
            ],
            "prefix": "",
            "profile": null,
            "purge_tags": true,
            "region": null,
            "retries": 0,
            "secret_key": null,
            "session_token": null,
            "sig_v4": true,
            "src": null,
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "version": null
        }
    },
    "msg": "Key /test does not exist."
}

PLAY RECAP *********************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

WORKING results on 6.0.0 WITHOUT leading "/" in object parameter:

TASK [bb_agent : Test retrieve] ************************************************************************************************************
task path: /home/ubuntu/ansible/roles/bb_agent/tasks/main.yml:26
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1684201338.0680258-10089-163409229139833 `" && echo ansible-tmp-1684201338.0680258-10089-163409229139833="` echo /root/.ansible/tmp/ansible-tmp-1684201338.0680258-10089-163409229139833 `" ) && sleep 0'
Using module file /root/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_object.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-99552vc421gc/tmpt9mp06kk TO /root/.ansible/tmp/ansible-tmp-1684201338.0680258-10089-163409229139833/AnsiballZ_s3_object.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1684201338.0680258-10089-163409229139833/ /root/.ansible/tmp/ansible-tmp-1684201338.0680258-10089-163409229139833/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/root/splunk_venv/bin/python3.10 /root/.ansible/tmp/ansible-tmp-1684201338.0680258-10089-163409229139833/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1684201338.0680258-10089-163409229139833/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "bucket": "abc-sultana-bucket-test-1",
            "ceph": false,
            "content": null,
            "content_base64": null,
            "copy_src": null,
            "debug_botocore_endpoint_logs": false,
            "dest": "/root/test",
            "dualstack": false,
            "encrypt": true,
            "encryption_kms_key_id": null,
            "encryption_mode": "AES256",
            "endpoint_url": null,
            "expiry": 600,
            "headers": null,
            "ignore_nonexistent_bucket": false,
            "marker": "",
            "max_keys": 1000,
            "metadata": null,
            "mode": "get",
            "object": "test",
            "overwrite": "different",
            "permission": [
                "private"
            ],
            "prefix": "",
            "profile": null,
            "purge_tags": true,
            "region": null,
            "retries": 0,
            "secret_key": null,
            "session_token": null,
            "sig_v4": true,
            "src": null,
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "version": null
        }
    },
    "msg": "GET operation complete"
}
META: role_complete for localhost
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[tremble]

@dsultanaabc,

Thanks for taking the time to open this issue. I've been able to reproduce the issue and I've opened #1549 which should fix the issue.

That said, looking specifically at the Amazon AWS documentation, object names should not include the leading /. https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html . While historically we have pruned out this leading /, from experience tweaking inputs to do things like removing the / tends to result in painful edge cases in the long run (especially since we support other platforms too).

[dsultanaabc]

Thanks for much for taking a look. Agree with removing the leading /. I think when we originally put these tasks together, we heavily relied on the documentation which says to included it. Your PR attentions a controlled, phased and informed approach of moving away from this though. Appreciate you taking a look so quickly.

Cheers.