etcd snapshots configured with incorrect s3 permissions results in no snapshots

[ryanelliottsmith]

RKE version:
v1.0.4
RKE Tools verison:
rancher/rke-tools:v0.1.52

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
ec2(no cloudprovider)

cluster.yml file:

cluster_name: test_s3
nodes:
  - address: x.x.x.x
    internal_address: x.x.x.x
    user: ubuntu
    role: [controlplane,worker,etcd]
services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h
    backup_config:
        interval_hours: 1
        retention: 48
        s3backupconfig:
          access_key: badkey
          secret_key: badsecret
          bucket_name: xxxx
          endpoint: s3.amazonaws.com
          folder: xxxx
          region: ap-southeast-2
        safe_timestamp: false

Steps to Reproduce:
Create(or modify existing) cluster with incorrect credentials defined for s3backupconfig.

Results:
No local snapshot created in /opt/rke/etcd-snapshots/

etcd-rolling-snapshots logs the following:

time="2020-08-20T01:45:48Z" level=error msg="failed to set s3 server: failed to check s3 bucket:xxxx, err:Access Denied." s3-accessKey=xxxx s3-bucketName=xxxx s3-endpoint=s3.amazonaws.com s3-endpoint-ca= s3-folder=xxxx s3-region=ap-southeast-2
time="2020-08-20T01:45:48Z" level=fatal msg="failed to set s3 server: failed to check s3 bucket:xxxx, err:Access Denied."

More info:
The same symptoms are seen if accesskey/secretkey are excluded and an IAM policy is not attached to the nodes, or an IAM policy is attached that doesn't have enough permission to list/write to the specified bucket.

gz#11851