Download from S3 (still failing) with spaces in path

[afischer-opentext-com]

Description of Issue/Question

When attempting to download files from S3 using the file.managed states this only works when files have no space in their name. Their were various issues in the past which are currently closed but this still does not work. This breaks automation efforts as it requires manual intervention/renaming when using installers/MSI files on Windows with have spaces in their path. I can confirm this issue on 2017.7.x and 2018.3-x. This ticket shows reproduction details using 2018.3.2.

Setup

Configure your pillar s3.configuration

s3:
  path_style: True
  https_enable: True
  verify_ssl: False
  location: us-east-1
  keyid: removed
  key: removed

Configure your state s3.tested

download_without_space:
  file.managed:
    - name: '/tmp/foo_bar_baz.txt'
    - makedirs: True
    - source: 's3://some-bucket/somefolder/foo_bar_baz.txt'
    - skip_verify: True

clear_without_space:
  file.absent:
    - name: '/tmp/foo_bar_baz.txt'
    - require:
      - download_without_space

download_with_space:
  file.managed:
    - name: '/tmp/foo bar baz.txt'
    - makedirs: True
    - source: 's3://some-bucket/somefolder/foo bar baz.txt'
    - skip_verify: True

clear_with_space:
  file.absent:
    - name: '/tmp/foo bar baz.txt'
    - require:
      - download_with_space

Steps to Reproduce Issue

When now calling

salt '*' saltutil.clear_cache
salt '*' saltutil.state.apply s3.tested

this leads to the output

root@vagranttest:~# salt '*' state.apply s3.tested
vagranttest:
----------
          ID: download_with_space
    Function: file.managed
        Name: /tmp/foo bar baz.txt
      Result: False
     Comment: Failed to cache s3://some-bucket/somefolder/foo bar baz.txt: Could not fetch from s3://some-bucket/somefolder/foo bar baz.txt. Exception: Failed to get file. SignatureDoesNotMatch: The request signature we calculated does not
match the signature you provided. Check your key and signing method.
     Started: 11:55:36.878496
    Duration: 414.042 ms
     Changes:
----------
          ID: clear_with_space
    Function: file.absent
        Name: /tmp/foo bar baz.txt
      Result: False
     Comment: One or more requisite failed: s3.tested.download_with_space
     Started: 11:55:37.292982
    Duration: 0.007 ms
     Changes:

Summary for vagranttest
------------
Succeeded: 2 (changed=2)
Failed:    2
------------
Total states run:     4
Total run time:   1.195 s
ERROR: Minions returned with non-zero exit code

Your minion log states (debug log level enabled)

2018-08-22 11:55:35,923 [salt.minion      :1428][INFO    ][8279] User sudo_vagrant Executing command state.apply with jid 20180822115535915865
2018-08-22 11:55:35,925 [salt.minion      :1435][DEBUG   ][8279] Command details {u'tgt_type': u'glob', u'jid': u'20180822115535915865', u'tgt': u'*', u'ret': u'', u'user': u'sudo_vagrant', u'arg': [u's3.tested'], u'fun': u'state.apply'}
2018-08-22 11:55:35,928 [salt.minion      :1563][INFO    ][8279] Starting a new job with PID 8279
2018-08-22 11:55:35,934 [salt.transport.zeromq:128 ][DEBUG   ][8279] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506', u'aes')
2018-08-22 11:55:35,934 [salt.crypt       :463 ][DEBUG   ][8279] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506')
2018-08-22 11:55:35,935 [salt.transport.zeromq:194 ][DEBUG   ][8279] Connecting the Minion to the Master URI (for the return server): tcp://127.0.1.1:4506
2018-08-22 11:55:35,935 [salt.transport.zeromq:1029][DEBUG   ][8279] Trying to connect to: tcp://127.0.1.1:4506
2018-08-22 11:55:35,959 [salt.pillar      :58  ][DEBUG   ][8279] Determining pillar cache
2018-08-22 11:55:35,959 [salt.transport.zeromq:128 ][DEBUG   ][8279] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506', u'aes')
2018-08-22 11:55:35,960 [salt.crypt       :463 ][DEBUG   ][8279] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506')
2018-08-22 11:55:35,960 [salt.transport.zeromq:194 ][DEBUG   ][8279] Connecting the Minion to the Master URI (for the return server): tcp://127.0.1.1:4506
2018-08-22 11:55:35,961 [salt.transport.zeromq:1029][DEBUG   ][8279] Trying to connect to: tcp://127.0.1.1:4506
2018-08-22 11:55:36,023 [salt.crypt       :199 ][DEBUG   ][8279] salt.crypt.get_rsa_key: Loading private key
2018-08-22 11:55:36,024 [salt.crypt       :776 ][DEBUG   ][8279] Loaded minion key: /etc/salt/pki/minion/minion.pem
2018-08-22 11:55:36,028 [salt.state       :923 ][INFO    ][8279] Loading fresh modules for state activity
2018-08-22 11:55:36,052 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded jinja.render
2018-08-22 11:55:36,052 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded yaml.render
2018-08-22 11:55:36,066 [salt.fileclient  :1124][DEBUG   ][8279] In saltenv 'base', looking at rel_path 's3/tested.sls' to resolve 'salt://s3/tested.sls'
2018-08-22 11:55:36,067 [salt.fileclient  :1132][DEBUG   ][8279] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/s3/tested.sls' to resolve 'salt://s3/tested.sls'
2018-08-22 11:55:36,067 [salt.fileclient  :1152][DEBUG   ][8279] Fetching file from saltenv 'base', ** attempting ** 'salt://s3/tested.sls'
2018-08-22 11:55:36,067 [salt.fileclient  :1176][DEBUG   ][8279] No dest file found
2018-08-22 11:55:36,081 [salt.fileclient  :1257][INFO    ][8279] Fetching file from saltenv 'base', ** done ** 's3/tested.sls'
2018-08-22 11:55:36,081 [salt.template    :51  ][DEBUG   ][8279] compile template: /var/cache/salt/minion/files/base/s3/tested.sls
2018-08-22 11:55:36,082 [salt.utils.jinja :69  ][DEBUG   ][8279] Jinja search path: [u'/var/cache/salt/minion/files/base']
2018-08-22 11:55:36,086 [salt.transport.zeromq:128 ][DEBUG   ][8279] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506', u'aes')
2018-08-22 11:55:36,086 [salt.crypt       :463 ][DEBUG   ][8279] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506')
2018-08-22 11:55:36,087 [salt.transport.zeromq:194 ][DEBUG   ][8279] Connecting the Minion to the Master URI (for the return server): tcp://127.0.1.1:4506
2018-08-22 11:55:36,087 [salt.transport.zeromq:1029][DEBUG   ][8279] Trying to connect to: tcp://127.0.1.1:4506
2018-08-22 11:55:36,089 [salt.template    :26  ][PROFILE ][8279] Time (in seconds) to render '/var/cache/salt/minion/files/base/s3/tested.sls' using 'jinja' renderer: 0.00683808326721
2018-08-22 11:55:36,089 [salt.template    :113 ][DEBUG   ][8279] Rendered data from file: /var/cache/salt/minion/files/base/s3/tested.sls:
download_without_space:
  file.managed:
    - name: '/tmp/foo_bar_baz.txt'
    - makedirs: True
    - source: 's3://some-bucket/somefolder/foo_bar_baz.txt'
    - skip_verify: True

clear_without_space:
  file.absent:
    - name: '/tmp/foo_bar_baz.txt'
    - require:
      - download_without_space

download_with_space:
  file.managed:
    - name: '/tmp/foo bar baz.txt'
    - makedirs: True
    - source: 's3://some-bucket/somefolder/foo bar baz.txt'
    - skip_verify: True

clear_with_space:
  file.absent:
    - name: '/tmp/foo bar baz.txt'
    - require:
      - download_with_space

2018-08-22 11:55:36,093 [salt.loaded.int.render.yaml:68  ][DEBUG   ][8279] Results of YAML rendering: 
OrderedDict([(u'download_without_space', OrderedDict([(u'file.managed', [OrderedDict([(u'name', u'/tmp/foo_bar_baz.txt')]), OrderedDict([(u'makedirs', True)]), OrderedDict([(u'source', u's3://some-bucket/somefolder/foo_bar_baz.txt')]), OrderedDict([(u'skip_verify', True)])])])), (u'clear_without_space', OrderedDict([(u'file.absent', [OrderedDict([(u'name', u'/tmp/foo_bar_baz.txt')]), OrderedDict([(u'require', [u'download_without_space'])])])])), (u'download_with_space', OrderedDict([(u'file.managed', [OrderedDict([(u'name', u'/tmp/foo bar baz.txt')]), OrderedDict([(u'makedirs', True)]), OrderedDict([(u'source', u's3://some-bucket/somefolder/foo bar baz.txt')]), OrderedDict([(u'skip_verify', True)])])])), (u'clear_with_space', OrderedDict([(u'file.absent', [OrderedDict([(u'name', u'/tmp/foo bar baz.txt')]), OrderedDict([(u'require', [u'download_with_space'])])])]))])
2018-08-22 11:55:36,094 [salt.template    :26  ][PROFILE ][8279] Time (in seconds) to render '/var/cache/salt/minion/files/base/s3/tested.sls' using 'yaml' renderer: 0.00396609306335
2018-08-22 11:55:36,095 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded config.option
2018-08-22 11:55:36,096 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded file.managed
2018-08-22 11:55:36,096 [salt.state       :1789][INFO    ][8279] Running state [/tmp/foo_bar_baz.txt] at time 11:55:36.096818
2018-08-22 11:55:36,097 [salt.state       :1822][INFO    ][8279] Executing state file.managed for [/tmp/foo_bar_baz.txt]
2018-08-22 11:55:36,099 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded file.source_list
2018-08-22 11:55:36,100 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded cp.is_cached
2018-08-22 11:55:36,104 [salt.transport.zeromq:128 ][DEBUG   ][8279] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506', u'aes')
2018-08-22 11:55:36,104 [salt.crypt       :463 ][DEBUG   ][8279] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506')
2018-08-22 11:55:36,105 [salt.transport.zeromq:194 ][DEBUG   ][8279] Connecting the Minion to the Master URI (for the return server): tcp://127.0.1.1:4506
2018-08-22 11:55:36,106 [salt.transport.zeromq:1029][DEBUG   ][8279] Trying to connect to: tcp://127.0.1.1:4506
2018-08-22 11:55:36,107 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded s3.query
2018-08-22 11:55:36,108 [salt.loaded.int.utils.s3:145 ][DEBUG   ][8279] S3 Request: https://s3.amazonaws.com/some-bucket/somefolder/foo_bar_baz.txt?
2018-08-22 11:55:36,108 [salt.loaded.int.utils.s3:146 ][DEBUG   ][8279] S3 Headers::
2018-08-22 11:55:36,108 [salt.loaded.int.utils.s3:147 ][DEBUG   ][8279]     Authorization: AWS4-HMAC-SHA256 Credential=removed/20180822/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2e570ce480004f929b21ae1394aff51561f5176693a7aad449e169e3dfd1cebf
2018-08-22 11:55:36,109 [urllib3.connectionpool:826 ][DEBUG   ][8279] Starting new HTTPS connection (1): s3.amazonaws.com
2018-08-22 11:55:36,678 [py.warnings      :860 ][WARNING ][8279] /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:860: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

2018-08-22 11:55:36,863 [urllib3.connectionpool:396 ][DEBUG   ][8279] https://s3.amazonaws.com:443 "GET /some-bucket/somefolder/foo_bar_baz.txt HTTP/1.1" 200 11
2018-08-22 11:55:36,864 [salt.loaded.int.utils.s3:199 ][DEBUG   ][8279] S3 Response Status Code: 200
2018-08-22 11:55:36,865 [salt.loaded.int.utils.s3:239 ][DEBUG   ][8279] Saving to local file: /var/cache/salt/minion/extrn_files/base/some-bucket/somefolder/foo_bar_baz.txt
2018-08-22 11:55:36,867 [salt.utils.lazy  :100 ][DEBUG   ][8279] LazyLoaded cmd.run
2018-08-22 11:55:36,868 [salt.loaded.int.module.cmdmod:392 ][INFO    ][8279] Executing command [u'lsattr', u'/tmp/foo_bar_baz.txt'] in directory '/root'
2018-08-22 11:55:36,874 [salt.loaded.int.module.cmdmod:755 ][DEBUG   ][8279] stdout: --------------e--- /tmp/foo_bar_baz.txt
2018-08-22 11:55:36,875 [salt.loaded.int.module.cmdmod:1080][DEBUG   ][8279] output: --------------e--- /tmp/foo_bar_baz.txt
2018-08-22 11:55:36,876 [salt.state       :308 ][INFO    ][8279] File changed:
New file
2018-08-22 11:55:36,876 [salt.state       :1968][INFO    ][8279] Completed state [/tmp/foo_bar_baz.txt] at time 11:55:36.876501 (duration_in_ms=779.683)
2018-08-22 11:55:36,877 [salt.state       :1789][INFO    ][8279] Running state [/tmp/foo_bar_baz.txt] at time 11:55:36.877044
2018-08-22 11:55:36,877 [salt.state       :1822][INFO    ][8279] Executing state file.absent for [/tmp/foo_bar_baz.txt]
2018-08-22 11:55:36,877 [salt.state       :308 ][INFO    ][8279] {u'removed': u'/tmp/foo_bar_baz.txt'}
2018-08-22 11:55:36,878 [salt.state       :1968][INFO    ][8279] Completed state [/tmp/foo_bar_baz.txt] at time 11:55:36.878165 (duration_in_ms=1.12)
2018-08-22 11:55:36,878 [salt.state       :1789][INFO    ][8279] Running state [/tmp/foo bar baz.txt] at time 11:55:36.878496
2018-08-22 11:55:36,878 [salt.state       :1822][INFO    ][8279] Executing state file.managed for [/tmp/foo bar baz.txt]
2018-08-22 11:55:36,880 [salt.loaded.int.utils.s3:145 ][DEBUG   ][8279] S3 Request: https://s3.amazonaws.com/some-bucket/somefolder/foo bar baz.txt?
2018-08-22 11:55:36,880 [salt.loaded.int.utils.s3:146 ][DEBUG   ][8279] S3 Headers::
2018-08-22 11:55:36,880 [salt.loaded.int.utils.s3:147 ][DEBUG   ][8279]     Authorization: AWS4-HMAC-SHA256 Credential=removed/20180822/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=073c857be5856a046831cd3bb19a8cb0aee7f1cedd0c0c45f083766fb3eb346a
2018-08-22 11:55:36,881 [urllib3.connectionpool:826 ][DEBUG   ][8279] Starting new HTTPS connection (1): s3.amazonaws.com
2018-08-22 11:55:37,184 [py.warnings      :860 ][WARNING ][8279] /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:860: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

2018-08-22 11:55:37,288 [urllib3.connectionpool:396 ][DEBUG   ][8279] https://s3.amazonaws.com:443 "GET /some-bucket/somefolder/foo%20bar%20baz.txt HTTP/1.1" 403 None
2018-08-22 11:55:37,290 [salt.loaded.int.utils.s3:184 ][DEBUG   ][8279]     Response content: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>removed</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20180822T115536Z
20180822/us-east-1/s3/aws4_request
dc113aba39863909c62295797a070e16c98e88086e953754e8be0427c1944d63</StringToSign><SignatureProvided>073c857be5856a046831cd3bb19a8cb0aee7f1cedd0c0c45f083766fb3eb346a</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 38 30 38 32 32 54 31 31 35 35 33 36 5a 0a 32 30 31 38 30 38 32 32 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 63 31 31 33 61 62 61 33 39 38 36 33 39 30 39 63 36 32 32 39 35 37 39 37 61 30 37 30 65 31 36 63 39 38 65 38 38 30 38 36 65 39 35 33 37 35 34 65 38 62 65 30 34 32 37 63 31 39 34 34 64 36 33</StringToSignBytes><CanonicalRequest>GET
/some-bucket/somefolder/foo%20bar%20baz.txt

host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180822T115536Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 72 65 63 2d 74 65 73 74 62 75 63 6b 65 74 2f 56 61 67 72 61 6e 74 2f 66 6f 6f 25 32 30 62 61 72 25 32 30 62 61 7a 2e 74 78 74 0a 0a 68 6f 73 74 3a 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 38 30 38 32 32 54 31 31 35 35 33 36 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>02DC3DDA3C898A6F</RequestId><HostId>8aXm9mw42U3DcIDKp2kiJbsEjRvthRO5/pAkQ9hKgQqM5oUv29zDjCi1V1K68elbiN/Ykutxun8=</HostId></Error>
2018-08-22 11:55:37,291 [salt.loaded.int.utils.s3:199 ][DEBUG   ][8279] S3 Response Status Code: 403
2018-08-22 11:55:37,291 [salt.state       :310 ][ERROR   ][8279] Failed to cache s3://some-bucket/somefolder/foo bar baz.txt: Could not fetch from s3://some-bucket/somefolder/foo bar baz.txt. Exception: Failed to get file. SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
2018-08-22 11:55:37,292 [salt.state       :1968][INFO    ][8279] Completed state [/tmp/foo bar baz.txt] at time 11:55:37.292538 (duration_in_ms=414.042)
2018-08-22 11:55:37,293 [salt.state       :2768][DEBUG   ][8279] File /var/cache/salt/minion/accumulator/139734403140752 does not exist, no need to cleanup
2018-08-22 11:55:37,294 [salt.minion      :755 ][DEBUG   ][8279] Minion return retry timer set to 8 seconds (randomized)
2018-08-22 11:55:37,294 [salt.minion      :1860][INFO    ][8279] Returning information for job: 20180822115535915865
2018-08-22 11:55:37,295 [salt.transport.zeromq:128 ][DEBUG   ][8279] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506', u'aes')
2018-08-22 11:55:37,295 [salt.crypt       :463 ][DEBUG   ][8279] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'saltminion', u'tcp://127.0.1.1:4506')
2018-08-22 11:55:37,296 [salt.transport.zeromq:194 ][DEBUG   ][8279] Connecting the Minion to the Master URI (for the return server): tcp://127.0.1.1:4506
2018-08-22 11:55:37,297 [salt.transport.zeromq:1029][DEBUG   ][8279] Trying to connect to: tcp://127.0.1.1:4506
2018-08-22 11:55:37,312 [salt.minion      :1708][DEBUG   ][8279] minion return: {u'fun_args': [u's3.tested'], u'jid': u'20180822115535915865', u'return': {u'file_|-download_with_space_|-/tmp/foo bar baz.txt_|-managed': {u'comment': u'Failed to cache s3://some-bucket/somefolder/foo bar baz.txt: Could not fetch from s3://some-bucket/somefolder/foo bar baz.txt. Exception: Failed to get file. SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.', u'pchanges': {}, u'name': u'/tmp/foo bar baz.txt', u'start_time': '11:55:36.878496', u'result': False, u'duration': 414.042, u'__run_num__': 2, u'__sls__': u's3.tested', u'changes': {}, u'__id__': u'download_with_space'}, u'file_|-clear_with_space_|-/tmp/foo bar baz.txt_|-absent': {u'comment': u'One or more requisite failed: s3.tested.download_with_space', u'start_time': '11:55:37.292982', u'result': False, u'duration': 0.007, u'__run_num__': 3, u'__sls__': u's3.tested', u'changes': {}}, u'file_|-clear_without_space_|-/tmp/foo_bar_baz.txt_|-absent': {u'comment': u'Removed file /tmp/foo_bar_baz.txt', u'pchanges': {u'removed': u'/tmp/foo_bar_baz.txt'}, u'name': u'/tmp/foo_bar_baz.txt', u'start_time': '11:55:36.877045', u'result': True, u'duration': 1.12, u'__run_num__': 1, u'__sls__': u's3.tested', u'changes': {u'removed': u'/tmp/foo_bar_baz.txt'}, u'__id__': u'clear_without_space'}, u'file_|-download_without_space_|-/tmp/foo_bar_baz.txt_|-managed': {u'comment': u'File /tmp/foo_bar_baz.txt updated', u'pchanges': {}, u'name': u'/tmp/foo_bar_baz.txt', u'start_time': '11:55:36.096818', u'result': True, u'duration': 779.683, u'__run_num__': 0, u'__sls__': u's3.tested', u'changes': {u'diff': u'New file', u'mode': u'0644'}, u'__id__': u'download_without_space'}}, u'retcode': 2, u'success': True, u'fun': u'state.apply'}

I believe that the issue may relate to this log line

2018-08-22 11:55:37,288 [urllib3.connectionpool:396 ][DEBUG ][8279] https://s3.amazonaws.com:443 "GET /some-bucket/somefolder/foo%20bar%20baz.txt HTTP/1.1" 403 None
I argue that spaces would not need to be %20 but + instead.

Versions Report

Master and Minion running on the same machine.

user@machine:~# salt --versions-report
Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-20-generic
         system: Linux
        version: Ubuntu 18.04 bionic

[damon-atkins]

What happens if the spaces are turned into %20

[afischer-opentext-com]

This indeed seems to work!

Is there any option to have this converted automatically in the S3 module? We are calculating the base location of the file dependent on pillar information (it may be S3 or a file share or the salt file server as the deployments may be air gapped), so it would be some effort to workaround all occurrences of it dependent on the source of the files. Also I am not aware of an Jinja urlencode filter unfortunately.

[damon-atkins]

As a workaround something like this

{% set with_spaces = "s3://some-bucket/somefolder/foo bar baz.txt" %}
{% set no_spaces = with_spaces | replace(" ","%20") %}

download_with_space:
  file.managed:
    - name: '/tmp/foo bar baz.txt'
    - makedirs: True
    - source: '{{ no_spaces }}'
    - skip_verify: True

[garethgreenaway]

This is fixed in #48328, which will be available in 2018.3.3 and 2017.7.8.

[afischer-opentext-com]

That's awesome! Thank you very much!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.