win_get_url doesn't use the proxy even if proxy_url is specified

[kai2nenobu]

SUMMARY

win_get_url doesn't use the proxy even if proxy_url is specified. Therefore, if I execute a win_get_url task under a proxy restricted network, it fails with 407 error (Proxy Authentication Required).

ISSUE TYPE

• Bug Report

COMPONENT NAME

win_get_url

ANSIBLE VERSION

$ ansible --version
ansible 2.8.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.15+ (default, Nov 27 2018, 23:36:35) [GCC 7.3.0]

CONFIGURATION

$ ansible-config dump --only-changed
ANSIBLE_NOCOWS(/mnt/c/Users/user/tmp/ansible-bug/ansible.cfg) = True
INJECT_FACTS_AS_VARS(/mnt/c/Users/user/tmp/ansible-bug/ansible.cfg) = False
RETRY_FILES_ENABLED(/mnt/c/Users/user/tmp/ansible-bug/ansible.cfg) = False

OS / ENVIRONMENT

Control machine is Ubuntu 18.04 on WSL.

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Installed pip libraries

$ pip list --format=columns
Package       Version
------------- ---------
ansible       2.8.1
asn1crypto    0.24.0
certifi       2019.6.16
chardet       3.0.4
cryptography  2.1.4
enum34        1.1.6
gyp           0.1
httplib2      0.9.2
idna          2.6
ipaddress     1.0.17
Jinja2        2.10
keyring       10.6.0
keyrings.alt  3.0
MarkupSafe    1.0
ntlm-auth     1.3.0
paramiko      2.0.0
pip           9.0.1
pyasn1        0.4.2
pycrypto      2.6.1
pygobject     3.26.1
python-apt    1.6.4
pywinrm       0.3.0
pyxdg         0.25
PyYAML        3.12
requests      2.22.0
requests-ntlm 1.1.0
SecretStorage 2.3.1
setuptools    39.0.1
six           1.11.0
urllib3       1.25.3
wheel         0.30.0
xmltodict     0.12.0

Target machine is Windows 10.

• Edition: Windows 10 Pro
• Version: 1809
• Installation Date: 2019/02/07
• OS Build: 17763.379

STEPS TO REPRODUCE

Inventory file (hosts) is as below.

[windows10]
localhost

[windows10:vars]
ansible_port=5986
ansible_connection=winrm
ansible_winrm_server_cert_validation=ignore
ansible_user=domain\user
ansible_password=********
ansible_winrm_transport=ntlm

Playbook (test.yml) is as below.

- hosts: windows10
  gather_facts: no
  tasks:
    - name: Download file via proxy
      win_get_url:
        url: https://github.com/ansible/ansible/archive/v2.8.1.zip
        dest: "%TEMP%"
        proxy_url: http://proxy.example.com:8080
        proxy_username: proxyuser
        proxy_password: ********

I applied a playbook like this, it fails with 407 error.

$ ansible-playbook -vvv -i hosts test.yml
ansible-playbook 2.8.1
  config file = /mnt/c/Users/user/tmp/ansible-bug/ansible.cfg
  configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.15+ (default, Nov 27 2018, 23:36:35) [GCC 7.3.0]
Using /mnt/c/Users/user/tmp/ansible-bug/ansible.cfg as config file
host_list declined parsing /mnt/c/Users/user/tmp/ansible-bug/hosts as it did not pass it's verify_file() method
script declined parsing /mnt/c/Users/user/tmp/ansible-bug/hosts as it did not pass it's verify_file() method
auto declined parsing /mnt/c/Users/user/tmp/ansible-bug/hosts as it did not pass it's verify_file() method
Parsed /mnt/c/Users/user/tmp/ansible-bug/hosts inventory source with ini plugin

PLAYBOOK: test.yml ************************************************************************************************************************************************************************************************
1 plays in test.yml

PLAY [windows10] **************************************************************************************************************************************************************************************************
META: ran handlers

TASK [Download file via proxy] ************************************************************************************************************************************************************************************
task path: /mnt/c/Users/user/tmp/ansible-bug/test.yml:4
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/windows/win_get_url.ps1
Pipelining is enabled.
<localhost> ESTABLISH WINRM CONNECTION FOR USER: domain\user on PORT 5986 TO localhost
EXEC (via pipeline wrapper)
The full traceback is:
The remote server returned an error: (407) Proxy Authentication Required.
At line:107 char:9
+         $web_response = $web_request.GetResponse()
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], WebException
    + FullyQualifiedErrorId : WebException

ScriptStackTrace:
at Invoke-AnsibleWebRequest, <No file>: line 107
at Invoke-DownloadFile, <No file>: line 311
at <ScriptBlock>, <No file>: line 406
fatal: [localhost]: FAILED! => {
    "changed": false,
    "dest": "C:\\Users\\user\\AppData\\Local\\Temp\\v2.8.1.zip",
    "elapsed": 0,
    "invocation": {
        "module_args": {
            "checksum": null,
            "checksum_algorithm": "sha1",
            "checksum_url": null,
            "dest": "C:\\Users\\user\\AppData\\Local\\Temp",
            "force": true,
            "force_basic_auth": false,
            "headers": {},
            "proxy_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "proxy_url": "http://proxy.example.com:8080",
            "proxy_username": "proxyuser",
            "timeout": 10,
            "url": "https://github.com/ansible/ansible/archive/v2.8.1.zip",
            "url_password": null,
            "url_username": null,
            "use_proxy": true,
            "validate_certs": true
        }
    },
    "msg": "Error requesting 'https://github.com/ansible/ansible/archive/v2.8.1.zip'. The remote server returned an error: (407) Proxy Authentication Required.",
    "status_code": 407,
    "url": "https://github.com/ansible/ansible/archive/v2.8.1.zip"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

EXPECTED RESULTS

win_get_url uses a proxy configuration and download a file successfully.

ACTUAL RESULTS

win_get_url doesn't use a proxy configuration and fails with 407 error.

[kai2nenobu]

After some debug, I noticed win_get_url.ps1 refers a non-existent variable. I modified win_get_url.ps1 as below, and win_get_url works successfully as I expected.

--- /usr/lib/python2.7/dist-packages/ansible/modules/windows/win_get_url.ps1.orig       2019-07-03 15:25:43.579725700 +0900
+++ /usr/lib/python2.7/dist-packages/ansible/modules/windows/win_get_url.ps1    2019-07-03 15:35:17.172836200 +0900
@@ -90,7 +90,7 @@

     if (-not $UseProxy) {
         $web_request.Proxy = $null
-    } elseif ($ProxyUri) {
+    } elseif ($Proxy) {
         $web_request.Proxy = $Proxy
     }

Execution result is like this.

$ ansible-playbook -vvv -i hosts test.yml
ansible-playbook 2.8.1
  config file = /mnt/c/Users/user/tmp/ansible-bug/ansible.cfg
  configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.15+ (default, Nov 27 2018, 23:36:35) [GCC 7.3.0]
Using /mnt/c/Users/user/tmp/ansible-bug/ansible.cfg as config file
host_list declined parsing /mnt/c/Users/user/tmp/ansible-bug/hosts as it did not pass it's verify_file() method
script declined parsing /mnt/c/Users/user/tmp/ansible-bug/hosts as it did not pass it's verify_file() method
auto declined parsing /mnt/c/Users/user/tmp/ansible-bug/hosts as it did not pass it's verify_file() method
Parsed /mnt/c/Users/user/tmp/ansible-bug/hosts inventory source with ini plugin

PLAYBOOK: test.yml ************************************************************************************************************************************************************************************************
1 plays in test.yml

PLAY [windows10] **************************************************************************************************************************************************************************************************
META: ran handlers

TASK [Download file via proxy] ************************************************************************************************************************************************************************************
task path: /mnt/c/Users/user/tmp/ansible-bug/test.yml:4
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/windows/win_get_url.ps1
Pipelining is enabled.
<localhost> ESTABLISH WINRM CONNECTION FOR USER: domain\user on PORT 5986 TO localhost
EXEC (via pipeline wrapper)
changed: [localhost] => {
    "changed": true,
    "checksum_dest": "fbc9df670fe85f5e7464a618114bc5fd4f77f277",
    "checksum_src": "fbc9df670fe85f5e7464a618114bc5fd4f77f277",
    "dest": "C:\\Users\\user\\AppData\\Local\\Temp\\v2.8.1.zip",
    "elapsed": 45.864948299999995,
    "invocation": {
        "module_args": {
            "checksum": null,
            "checksum_algorithm": "sha1",
            "checksum_url": null,
            "dest": "C:\\Users\\user\\AppData\\Local\\Temp",
            "force": true,
            "force_basic_auth": false,
            "headers": {},
            "proxy_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "proxy_url": "http://proxy.example.com:8080",
            "proxy_username": "proxyuser",
            "timeout": 10,
            "url": "https://github.com/ansible/ansible/archive/v2.8.1.zip",
            "url_password": null,
            "url_username": null,
            "use_proxy": true,
            "validate_certs": true
        }
    },
    "msg": "OK",
    "size": 25088008,
    "status_code": 200,
    "url": "https://github.com/ansible/ansible/archive/v2.8.1.zip"
}
META: ran handlers
META: ran handlers

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

[ansibot]

cc @ShachafGoldstein @if-meaton @jborean93 @jhawkesworth @nitzmahone @tksarah
click here for bot help

[jborean93]

@kai2nenobu can you create a PR with your changes, seems like an easy fix.