module_utils/urls.py: TypeError: a bytes-like object is required, not 'str'

[gangefors]

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible/module_utils/urls.py

ANSIBLE VERSION

ansible 2.4.0.0
  config file = None
  configured module search path = ['/home/stefanga/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/stefanga/.pyenv/versions/3.5.4/envs/ansible354/lib/python3.5/site-packages/ansible
  executable location = /home/stefanga/.pyenv/versions/ansible354/bin/ansible
  python version = 3.5.4 (default, Sep 29 2017, 15:48:08) [GCC 6.3.0 20170516]

CONFIGURATION

No configuration

OS / ENVIRONMENT

Debian stretch
Python 3.5.4

SUMMARY

When running a playbook with a uri task and having https_proxy environment variable set, the task fails with:

TypeError: a bytes-like object is required, not 'str'

Running the same task but without https_proxy succeeds.

STEPS TO REPRODUCE

---
- hosts: localhost
  tasks:
  - uri:
      url: "https://www.google.com"
      return_content: yes
    environment:
      https_proxy: "http://wwwproxy:3128"

EXPECTED RESULTS

The task successfully retrieves the URL.

ACTUAL RESULTS

ansible-playbook 2.4.0.0
  config file = None
  configured module search path = ['/home/stefanga/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/stefanga/.pyenv/versions/3.5.4/envs/ansible354/lib/python3.5/site-packages/ansible
  executable location = /home/stefanga/.pyenv/versions/ansible354/bin/ansible-playbook
  python version = 3.5.4 (default, Sep 29 2017, 15:48:08) [GCC 6.3.0 20170516]
No config file found; using defaults
setting up inventory plugins
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: all

 [WARNING]: provided hosts list is empty, only localhost is available

Loading callback plugin default of type stdout, v2.0 from /home/stefanga/.pyenv/versions/3.5.4/envs/ansible354/lib/python3.5/site-packages/ansible/plugins/callback/__init__.py

PLAYBOOK: play.yml *************************************************************************************************************************************************************************************************************************************************************
1 plays in play.yml

PLAY [localhost] ***************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************************************************************
Using module file /home/stefanga/.pyenv/versions/3.5.4/envs/ansible354/lib/python3.5/site-packages/ansible/modules/system/setup.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: stefanga
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/stefanga/.ansible/tmp/ansible-tmp-1506704178.3238819-26445008214164 `" && echo ansible-tmp-1506704178.3238819-26445008214164="` echo /home/stefanga/.ansible/tmp/ansible-tmp-1506704178.3238819-26445008214164 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpbl0l6tlo TO /home/stefanga/.ansible/tmp/ansible-tmp-1506704178.3238819-26445008214164/setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/stefanga/.ansible/tmp/ansible-tmp-1506704178.3238819-26445008214164/ /home/stefanga/.ansible/tmp/ansible-tmp-1506704178.3238819-26445008214164/setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/stefanga/.pyenv/versions/3.5.4/envs/ansible354/bin/python3.5 /home/stefanga/.ansible/tmp/ansible-tmp-1506704178.3238819-26445008214164/setup.py; rm -rf "/home/stefanga/.ansible/tmp/ansible-tmp-1506704178.3238819-26445008214164/" > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [uri] *********************************************************************************************************************************************************************************************************************************************************************
task path: /home/stefanga/projects/reactor/backend/ansible/play.yml:4
Using module file /home/stefanga/.pyenv/versions/3.5.4/envs/ansible354/lib/python3.5/site-packages/ansible/modules/net_tools/basics/uri.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: stefanga
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/stefanga/.ansible/tmp/ansible-tmp-1506704179.197424-244113412509946 `" && echo ansible-tmp-1506704179.197424-244113412509946="` echo /home/stefanga/.ansible/tmp/ansible-tmp-1506704179.197424-244113412509946 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpanyvor9p TO /home/stefanga/.ansible/tmp/ansible-tmp-1506704179.197424-244113412509946/uri.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/stefanga/.ansible/tmp/ansible-tmp-1506704179.197424-244113412509946/ /home/stefanga/.ansible/tmp/ansible-tmp-1506704179.197424-244113412509946/uri.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'https_proxy=http://wwwproxy:3128 /home/stefanga/.pyenv/versions/3.5.4/envs/ansible354/bin/python3.5 /home/stefanga/.ansible/tmp/ansible-tmp-1506704179.197424-244113412509946/uri.py; rm -rf "/home/stefanga/.ansible/tmp/ansible-tmp-1506704179.197424-244113412509946/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_o8tq6kso/ansible_modlib.zip/ansible/module_utils/urls.py", line 1044, in fetch_url
    client_key=client_key, cookies=cookies)
  File "/tmp/ansible_o8tq6kso/ansible_modlib.zip/ansible/module_utils/urls.py", line 951, in open_url
    r = urllib_request.urlopen(*urlopen_args)
  File "/home/stefanga/.pyenv/versions/3.5.4/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/home/stefanga/.pyenv/versions/3.5.4/lib/python3.5/urllib/request.py", line 464, in open
    req = meth(req)
  File "/tmp/ansible_o8tq6kso/ansible_modlib.zip/ansible/module_utils/urls.py", line 729, in http_request
    s.sendall(self.CONNECT_COMMAND % (self.hostname, self.port))
TypeError: a bytes-like object is required, not 'str'

fatal: [localhost]: FAILED! => {
    "changed": false,
    "content": "",
    "failed": true,
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": null,
            "body": null,
            "body_format": "raw",
            "client_cert": null,
            "client_key": null,
            "content": null,
            "creates": null,
            "delimiter": null,
            "dest": null,
            "directory_mode": null,
            "follow": false,
            "follow_redirects": "safe",
            "force": false,
            "force_basic_auth": false,
            "group": null,
            "headers": {},
            "http_agent": "ansible-httpget",
            "method": "GET",
            "mode": null,
            "owner": null,
            "regexp": null,
            "remote_src": null,
            "removes": null,
            "return_content": true,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": null,
            "status_code": [
                200
            ],
            "timeout": 30,
            "unsafe_writes": null,
            "url": "https://www.google.com",
            "url_password": null,
            "url_username": null,
            "use_proxy": true,
            "validate_certs": true
        }
    },
    "msg": "Status code was not [200]: An unknown error occurred: a bytes-like object is required, not 'str'",
    "redirected": false,
    "status": -1,
    "url": "https://www.google.com"
}
	to retry, use: --limit @/home/stefanga/projects/reactor/backend/ansible/play.retry

PLAY RECAP *********************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1   

[sivel]

resolved_by_pr 30811

[gangefors]

Ah, sorry that I never searched PRs as well. Nice to see that #30811 is on its way to get merged.

[abadger]

Closing as # 30811 has been merged.