unarchive: unable to unpack tar.gz from URI with python3 interpreter

[pvdputte]

ISSUE TYPE

• Bug Report

COMPONENT NAME

unarchive

ANSIBLE VERSION

$ ansible --version
ansible 2.4.0 (devel 313591f8b8) last updated 2017/04/13 21:20:44 (GMT +200)
  config file = 
  configured module search path = [u'/home/pvdputte/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/pvdputte/VC/vendor/ansible-master/lib/ansible
  executable location = /home/pvdputte/VC/vendor/ansible-master/bin/ansible
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]

CONFIGURATION

Default

OS / ENVIRONMENT

Running from ubuntu 14.04LTS
Managing ubuntu 17.04 (only python 3 after base install)

SUMMARY

Attempting to run the unarchive task to download and extract an http URL yields an error "Source 'http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz' failed to transfer"

STEPS TO REPRODUCE

$ ansible-d targethost -e 'ansible_python_interpreter=/usr/bin/python3' -m unarchive -a "src=http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz dest=/tmp remote_src=true"

EXPECTED RESULTS

tar.gz file is fetched and extracted in the expected location. E.g. with python 2:

$ ansible-d targethost -m unarchive -a "src=http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz dest=/tmp remote_src=true"
targethost | SUCCESS => {
    "changed": true, 
    "dest": "/tmp", 
    "extract_results": {
        "cmd": [
            "/bin/tar", 
            "--extract", 
            "-C", 
            "/tmp", 
            "-z", 
            "-f", 
            "/tmp/ansible_3W9Kei/SoapUI-5.3.0-linux-bin.tar.gz"
        ], 
        "err": "", 
        "out": "", 
        "rc": 0
    }, 
    "gid": 0, 
    "group": "root", 
    "handler": "TgzArchive", 
    "mode": "01777", 
    "owner": "root", 
    "size": 4096, 
    "src": "/tmp/ansible_3W9Kei/SoapUI-5.3.0-linux-bin.tar.gz", 
    "state": "directory", 
    "uid": 0
}

ACTUAL RESULTS

ansible 2.3:

targethost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "copy": true, 
            "creates": null, 
            "delimiter": null, 
            "dest": "/tmp", 
            "directory_mode": null, 
            "exclude": [], 
            "extra_opts": [], 
            "follow": false, 
            "force": null, 
            "group": null, 
            "keep_newer": false, 
            "list_files": false, 
            "mode": null, 
            "original_basename": "SoapUI-5.3.0-linux-bin.tar.gz", 
            "owner": null, 
            "regexp": null, 
            "remote_src": true, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz", 
            "unsafe_writes": null, 
            "validate_certs": true
        }
    }, 
    "msg": "Failure downloading http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz, write() argument must be str, not bytes"
}

ansible 2.4:

targethost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "copy": true, 
            "creates": null, 
            "delimiter": null, 
            "dest": "/usr/local", 
            "directory_mode": null, 
            "exclude": [], 
            "extra_opts": [], 
            "follow": false, 
            "force": null, 
            "group": null, 
            "keep_newer": false, 
            "list_files": false, 
            "mode": null, 
            "original_basename": "SoapUI-5.3.0-linux-bin.tar.gz", 
            "owner": null, 
            "regexp": null, 
            "remote_src": false, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz", 
            "unsafe_writes": null, 
            "validate_certs": true
        }
    }, 
    "msg": "Source 'http://cdn01.downloads.smartbear.com/soapui/5.3.0/SoapUI-5.3.0-linux-bin.tar.gz' failed to transfer"
}

Using ansible + python 2 to download on the target system works fine.

At first I thought it would be similar to #19710, but apparently something related has already changed in master, as the 2.4 output looks different.

[ansibot]

cc @dagwieers @pileofrogs
click here for bot help

[maximede]

I'm having the same issue ( with python 2.7 though) I think it's coming fromn the fact that remote_src is somehow set incorrectly.
If you look at the ansible 2.4 log, it says "remote_src": false,

[sivel]

This appears to be an inconsistency in the way that remote_src works in comparison to copy.

Specifically due to these lines:

    # did tar file arrive?
    if not os.path.exists(src):
        if not remote_src and copy:
            module.fail_json(msg="Source '%s' failed to transfer" % src)

The change that likely caused this is 6aeea1f

The work around currently is to also supply copy=false. The documentation lists that copy is deprecated in favor of remote_src but this code path will not allow you to use a URL, without setting copy=false

[maximede]

git bisect tells me that it hasn't been working since 7b197d8

probably because of this :

# remove action plugin only key
for key in ('remote_src', 'decrypt'):
    if key in new_module_args:
        del new_module_args[key]

in ansible/lib/ansible/plugins/action/unarchive.py

Using copy: false instead of remote_src: true worked for me

[sivel]

I've submitted a PR which should resolve the issue:

#24732

If possible, please test.

[jctanner]

resolved_by_pr #24732

[pvdputte]

I confirm that the issue is resolved. Great!

[maximede]

@pvdputte not sure what the Ansible policy is, but, the PR is still not merged. I wouldn't consider this ticket closed until it's merged as it might confuse people or lead to the creation of dupes of this ticket

[pvdputte]

Need sleep, sorry.