[get_url] fails with DNS errors when http_proxy and https_proxy are undefined and omitted #82433

yaroslavhlushko · 2023-12-18T15:46:04Z

Summary

ansible.builtin.get_url does not ignore proxy environment variables set with ... | default(None) | default(omit) filters and fails with irrelevant "Temporary failure in name resolution" error message.
It should ignore proxy settings if not set as ansible.builtin.apt and other modules do.

Issue Type

Bug Report

Component Name

get_url

Ansible Version

$ ansible --version
ansible [core 2.16.2]
  config file = /mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg
  configured module search path = ['/home/yaroslav/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/yaroslav/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/yaroslav/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/yaroslav/.local/bin/ansible
  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

Configuration

# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
CONFIG_FILE() = /mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg
DEFAULT_JINJA2_NATIVE(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = True
HOST_KEY_CHECKING(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = False
PERSISTENT_COMMAND_TIMEOUT(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = 60
PERSISTENT_CONNECT_TIMEOUT(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = 60
SHOW_CUSTOM_STATS(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = True

CALLBACK:
========

default:
_______
show_custom_stats(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = True

CONNECTION:
==========

paramiko_ssh:
____________
host_key_checking(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = False

ssh:
___
host_key_checking(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = False
scp_if_ssh(/mnt/d/OneDrive/Code/Infrastructure-ansible/ansible.cfg) = True

OS / Environment

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

Steps to Reproduce

Implement the following playbook (initially tried by using tasks from community.zabbix collection to install zabbix agents):
https://github.com/ansible-collections/community.zabbix/blob/main/roles/zabbix_agent/tasks/Debian.yml
Filename: url_get_test.yml

---
- hosts: localhost

  tasks:
  - name: "Debian | Download gpg key"
    ansible.builtin.get_url:
      url: http://repo.zabbix.com/zabbix-official-repo.key
      dest: "~/zabbix-official-repo.key"
      mode: "0644"
      force: true
    environment:
      http_proxy: "{{ zabbix_http_proxy | default(None) | default(omit) }}"
      https_proxy: "{{ zabbix_https_proxy | default(None) | default(omit) }}"
    become: false
    tags:
      - install

ansible-playbook -i inventory/inventory.ini -k -u <your_username> -vvv url_get_test.yml
Get the error: "msg": "Request failed: <urlopen error [Errno -3] Temporary failure in name resolution>"

Expected Results

Achieved when the proxy vars are commented out

TASK [Debian | Download gpg key] *******************************************************************************************************************************************************************************************************************************************************
task path: /mnt/d/OneDrive/Code/Infrastructure-ansible/url_get_test.yml:5
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: yaroslav
<127.0.0.1> EXEC /bin/sh -c 'echo ~yaroslav && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/yaroslav/.ansible/tmp `"&& mkdir "` echo /home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981 `" && echo ansible-tmp-1702912074.8073697-28185-127121582242981="` echo /home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981 `" ) && sleep 0'
Using module file /home/yaroslav/.local/lib/python3.10/site-packages/ansible/modules/get_url.py
<127.0.0.1> PUT /home/yaroslav/.ansible/tmp/ansible-local-28076tfwahq35/tmp49w51j6t TO /home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981/AnsiballZ_get_url.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981/ /home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981/AnsiballZ_get_url.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981/AnsiballZ_get_url.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "checksum_dest": null,
    "checksum_src": "2e9c2a85f0ef5f81e21734833f61e7774441bd48",
    "dest": "/home/yaroslav/zabbix-official-repo.key",
    "elapsed": 0,
    "gid": 1000,
    "group": "yaroslav",
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": false,
            "checksum": "",
            "ciphers": null,
            "client_cert": null,
            "client_key": null,
            "decompress": true,
            "dest": "/home/yaroslav/zabbix-official-repo.key",
            "force": true,
            "force_basic_auth": false,
            "group": null,
            "headers": null,
            "http_agent": "ansible-httpget",
            "mode": "0644",
            "owner": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "timeout": 10,
            "tmp_dest": null,
            "unredirected_headers": [],
            "unsafe_writes": false,
            "url": "http://repo.zabbix.com/zabbix-official-repo.key",
            "url_password": null,
            "url_username": null,
            "use_gssapi": false,
            "use_netrc": true,
            "use_proxy": true,
            "validate_certs": true
        }
    },
    "md5sum": "29dd6f8234591edc38ac53272b1666df",
    "mode": "0644",
    "msg": "OK (2938 bytes)",
    "owner": "yaroslav",
    "size": 2938,
    "src": "/home/yaroslav/.ansible/tmp/ansible-tmp-1702912074.8073697-28185-127121582242981/tmpe7g9g8p0",
    "state": "file",
    "status_code": 200,
    "uid": 1000,
    "url": "http://repo.zabbix.com/zabbix-official-repo.key"
}

Actual Results

TASK [Debian | Download gpg key] *******************************************************************************************************************************************************************************************************************************************************
task path: /mnt/d/OneDrive/Code/Infrastructure-ansible/url_get_test.yml:5
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: yaroslav
<127.0.0.1> EXEC /bin/sh -c 'echo ~yaroslav && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/yaroslav/.ansible/tmp `"&& mkdir "` echo /home/yaroslav/.ansible/tmp/ansible-tmp-1702912042.7635632-28051-129180776043943 `" && echo ansible-tmp-1702912042.7635632-28051-129180776043943="` echo /home/yaroslav/.ansible/tmp/ansible-tmp-1702912042.7635632-28051-129180776043943 `" ) && sleep 0'
Using module file /home/yaroslav/.local/lib/python3.10/site-packages/ansible/modules/get_url.py
<127.0.0.1> PUT /home/yaroslav/.ansible/tmp/ansible-local-27942f8v4ws0s/tmpsor0a4lg TO /home/yaroslav/.ansible/tmp/ansible-tmp-1702912042.7635632-28051-129180776043943/AnsiballZ_get_url.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/yaroslav/.ansible/tmp/ansible-tmp-1702912042.7635632-28051-129180776043943/ /home/yaroslav/.ansible/tmp/ansible-tmp-1702912042.7635632-28051-129180776043943/AnsiballZ_get_url.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'http_proxy=None https_proxy=None /usr/bin/python3 /home/yaroslav/.ansible/tmp/ansible-tmp-1702912042.7635632-28051-129180776043943/AnsiballZ_get_url.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/yaroslav/.ansible/tmp/ansible-tmp-1702912042.7635632-28051-129180776043943/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "dest": "/home/yaroslav/zabbix-official-repo.key",
    "elapsed": 0,
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": false,
            "checksum": "",
            "ciphers": null,
            "client_cert": null,
            "client_key": null,
            "decompress": true,
            "dest": "/home/yaroslav/zabbix-official-repo.key",
            "force": true,
            "force_basic_auth": false,
            "group": null,
            "headers": null,
            "http_agent": "ansible-httpget",
            "mode": "0644",
            "owner": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "timeout": 10,
            "tmp_dest": null,
            "unredirected_headers": [],
            "unsafe_writes": false,
            "url": "http://repo.zabbix.com/zabbix-official-repo.key",
            "url_password": null,
            "url_username": null,
            "use_gssapi": false,
            "use_netrc": true,
            "use_proxy": true,
            "validate_certs": true
        }
    },
    "msg": "Request failed: <urlopen error [Errno -3] Temporary failure in name resolution>",
    "url": "http://repo.zabbix.com/zabbix-official-repo.key"
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibot · 2023-12-18T15:50:35Z

Files identified in the description:

lib/ansible/modules/get_url.py

If these files are incorrect, please update the component name section of the description or use the component bot command.

nitzmahone · 2024-01-04T19:35:54Z

The chained defaults you're using would likely never work the way you want anyway, but the underlying behavior should be fixed with some expansion of omit support we've done in the data tagging project.

Meantime, you might also want to have a look at:
https://gist.github.com/sivel/fc917eb275489d1bfb4492ff25fe9c26

Finally, it might make sense if we made empty string on the proxy envvars be treated as "don't use the proxy".

yaroslavhlushko · 2024-01-09T09:06:39Z

@nitzmahone Thank you for your reply.
For my own playbooks I might use the suggested ways to overcome the issue.
But I stumbled upon it during the execution of zabbix community collection tasks, which ran fine for ansible.builtin.apt but failed for ansible.builtin.get_url
I found it strange what one module can handle proxy vars omit fine and the other fails with DNS resolution issue (which is far from the real reason).
The sample task, which runs correctly (from the community package):

- name: "Debian | Installing gnupg"
  ansible.builtin.apt:
    pkg: gnupg
    update_cache: true
    cache_valid_time: 3600
    force: true
    state: present
  environment:
    http_proxy: "{{ zabbix_http_proxy | default(None) | default(omit) }}"
    https_proxy: "{{ zabbix_https_proxy | default(None) | default(omit) }}"
  register: gnupg_installed
  until: gnupg_installed is succeeded
  become: true
  tags:
    - install

ansibot added bug This issue/PR relates to a bug. needs_triage Needs a first human triage before being processed. affects_2.16 module This issue/PR relates to a module. labels Dec 18, 2023

nitzmahone added data_tagging and removed needs_triage Needs a first human triage before being processed. labels Jan 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[get_url] fails with DNS errors when http_proxy and https_proxy are undefined and omitted #82433

[get_url] fails with DNS errors when http_proxy and https_proxy are undefined and omitted #82433

yaroslavhlushko commented Dec 18, 2023 •

edited

ansibot commented Dec 18, 2023

nitzmahone commented Jan 4, 2024

yaroslavhlushko commented Jan 9, 2024 •

edited

[get_url] fails with DNS errors when http_proxy and https_proxy are undefined and omitted #82433

[get_url] fails with DNS errors when http_proxy and https_proxy are undefined and omitted #82433

Comments

yaroslavhlushko commented Dec 18, 2023 • edited

Summary

Issue Type

Component Name

Ansible Version

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Achieved when the proxy vars are commented out

Actual Results

Code of Conduct

ansibot commented Dec 18, 2023

nitzmahone commented Jan 4, 2024

yaroslavhlushko commented Jan 9, 2024 • edited

yaroslavhlushko commented Dec 18, 2023 •

edited

yaroslavhlushko commented Jan 9, 2024 •

edited