Ansible reads proxy from yum.conf but doesn't handle _none_

[rrauenza]

SUMMARY

This appears to be a regression after upgrading to 2.7.10 from 2.5.x

I'm getting this error:

The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_yum_payload_Yszxv7/ansible_yum_payload.zip/ansible/module_utils/urls.py", line 1259, in fetch_url
    client_key=client_key, cookies=cookies)
  File "/tmp/ansible_yum_payload_Yszxv7/ansible_yum_payload.zip/ansible/module_utils/urls.py", line 1162, in open_url
    client_cert=client_cert, client_key=client_key, cookies=cookies)
  File "/tmp/ansible_yum_payload_Yszxv7/ansible_yum_payload.zip/ansible/module_utils/urls.py", line 1070, in open
    r = urllib_request.urlopen(*urlopen_args)
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 429, in open
    req = meth(req)
  File "/tmp/ansible_yum_payload_Yszxv7/ansible_yum_payload.zip/ansible/module_utils/urls.py", line 741, in http_request
    " Please make sure you export https proxy as 'https_proxy=<SCHEME>://<IP_ADDRESS>:<PORT>'" % https_proxy)

I modified ansible source to show the value of https proxy to the error message:

"msg": "Failed to parse https_proxy environment variable (_none_). Please make sure you export https proxy as 'https_proxy=<SCHEME>://<IP_ADDRESS>:<PORT>'"

_none_ comes from yum.conf and is explicitly allowed by yum for yum to disable proxy:

https://superuser.com/questions/393099/how-to-specify-that-yum-should-not-use-proxy-for-specific-domain

From yum.conf manual:

proxy URL to the proxy server for this repository. Set to 'none' to disable the global proxy setting for this repository. If this is unset it inherits it from the global setting

My suggestion is when you are reading yum.conf for proxy, treat _none_ as ''

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible/module_utils/urls.py

ANSIBLE VERSION

  config file = /mts/home5/rich/src/perfauto-ansible/ansible.cfg
  configured module search path = [u'/mts/home5/rich/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Apr  9 2019, 14:30:50) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

CONFIGURATION

$ /usr/bin/ansible-config dump | cat
ACTION_WARNINGS(default) = True
AGNOSTIC_BECOME_PROMPT(default) = False
ALLOW_WORLD_READABLE_TMPFILES(default) = False
ANSIBLE_COW_PATH(default) = None
ANSIBLE_COW_SELECTION(default) = default
ANSIBLE_COW_WHITELIST(default) = ['bud-frogs', 'bunny', 'cheese', 'daemon', 'default', 'dragon', 'elephant-in-snake', 'elephant', 'eyes', 'hellokitty', 'kitty', 'luke-koala', 'meow', 'milk', 'moofasa', 'moose', 'ren', 'sheep', 'small', 'stegosaurus', 'stimpy', 'supermilker', 'three-eyes', 'turkey', 'turtle', 'tux', 'udder', 'vader-koala', 'vader', 'www']
ANSIBLE_FORCE_COLOR(default) = False
ANSIBLE_NOCOLOR(default) = False
ANSIBLE_NOCOWS(default) = False
ANSIBLE_PIPELINING(/mts/home5/rich/src/perfauto-ansible/ansible.cfg) = True
ANSIBLE_SSH_ARGS(default) = -C -o ControlMaster=auto -o ControlPersist=60s
ANSIBLE_SSH_CONTROL_PATH(default) = None
ANSIBLE_SSH_CONTROL_PATH_DIR(/mts/home5/rich/src/perfauto-ansible/ansible.cfg) = /tmp/.ansible-${USER}/control_path
ANSIBLE_SSH_EXECUTABLE(default) = ssh
ANSIBLE_SSH_RETRIES(default) = 0
ANY_ERRORS_FATAL(default) = False
BECOME_ALLOW_SAME_USER(default) = False
CACHE_PLUGIN(default) = memory
CACHE_PLUGIN_CONNECTION(default) = None
CACHE_PLUGIN_PREFIX(default) = ansible_facts
CACHE_PLUGIN_TIMEOUT(default) = 86400
COLOR_CHANGED(default) = yellow
COLOR_CONSOLE_PROMPT(default) = white
COLOR_DEBUG(default) = dark gray
COLOR_DEPRECATE(default) = purple
COLOR_DIFF_ADD(default) = green
COLOR_DIFF_LINES(default) = cyan
COLOR_DIFF_REMOVE(default) = red
COLOR_ERROR(default) = red
COLOR_HIGHLIGHT(default) = white
COLOR_OK(default) = green
COLOR_SKIP(default) = cyan
COLOR_UNREACHABLE(default) = bright red
COLOR_VERBOSE(default) = blue
COLOR_WARN(default) = bright purple
COMMAND_WARNINGS(default) = True
DEFAULT_ACTION_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']
DEFAULT_ALLOW_UNSAFE_LOOKUPS(default) = False
DEFAULT_ASK_PASS(default) = False
DEFAULT_ASK_SUDO_PASS(default) = False
DEFAULT_ASK_SU_PASS(default) = False
DEFAULT_ASK_VAULT_PASS(default) = False
DEFAULT_BECOME(default) = False
DEFAULT_BECOME_ASK_PASS(default) = False
DEFAULT_BECOME_EXE(default) = None
DEFAULT_BECOME_FLAGS(default) = 
DEFAULT_BECOME_METHOD(default) = sudo
DEFAULT_BECOME_USER(default) = root
DEFAULT_CACHE_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/cache', u'/usr/share/ansible/plugins/cache']
DEFAULT_CALLABLE_WHITELIST(default) = []
DEFAULT_CALLBACK_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/callback', u'/usr/share/ansible/plugins/callback']
DEFAULT_CALLBACK_WHITELIST(default) = []
DEFAULT_CLICONF_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/cliconf', u'/usr/share/ansible/plugins/cliconf']
DEFAULT_CONNECTION_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/connection', u'/usr/share/ansible/plugins/connection']
DEFAULT_DEBUG(default) = False
DEFAULT_EXECUTABLE(default) = /bin/sh
DEFAULT_FACT_PATH(default) = None
DEFAULT_FILTER_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/filter', u'/usr/share/ansible/plugins/filter']
DEFAULT_FORCE_HANDLERS(default) = False
DEFAULT_FORKS(/mts/home5/rich/src/perfauto-ansible/ansible.cfg) = 100
DEFAULT_GATHERING(default) = implicit
DEFAULT_GATHER_SUBSET(default) = ['all']
DEFAULT_GATHER_TIMEOUT(default) = 10
DEFAULT_HANDLER_INCLUDES_STATIC(default) = False
DEFAULT_HASH_BEHAVIOUR(default) = replace
DEFAULT_HOST_LIST(default) = [u'/etc/ansible/hosts']
DEFAULT_HTTPAPI_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/httpapi', u'/usr/share/ansible/plugins/httpapi']
DEFAULT_INTERNAL_POLL_INTERVAL(default) = 0.001
DEFAULT_INVENTORY_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/inventory', u'/usr/share/ansible/plugins/inventory']
DEFAULT_JINJA2_EXTENSIONS(default) = []
DEFAULT_JINJA2_NATIVE(default) = False
DEFAULT_KEEP_REMOTE_FILES(default) = False
DEFAULT_LIBVIRT_LXC_NOSECLABEL(default) = False
DEFAULT_LOAD_CALLBACK_PLUGINS(default) = False
DEFAULT_LOCAL_TMP(/mts/home5/rich/src/perfauto-ansible/ansible.cfg) = /tmp/.ansible-rich/local_tmp/ansible-local-592ibphbc
DEFAULT_LOG_FILTER(default) = []
DEFAULT_LOG_PATH(default) = None
DEFAULT_LOOKUP_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/lookup', u'/usr/share/ansible/plugins/lookup']
DEFAULT_MANAGED_STR(default) = Ansible managed
DEFAULT_MODULE_ARGS(default) = 
DEFAULT_MODULE_COMPRESSION(default) = ZIP_DEFLATED
DEFAULT_MODULE_LANG(default) = {{ CONTROLLER_LANG }}
DEFAULT_MODULE_NAME(default) = command
DEFAULT_MODULE_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
DEFAULT_MODULE_SET_LOCALE(default) = False
DEFAULT_MODULE_UTILS_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/module_utils', u'/usr/share/ansible/plugins/module_utils']
DEFAULT_NETCONF_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/netconf', u'/usr/share/ansible/plugins/netconf']
DEFAULT_NO_LOG(default) = False
DEFAULT_NO_TARGET_SYSLOG(default) = False
DEFAULT_NULL_REPRESENTATION(default) = None
DEFAULT_POLL_INTERVAL(default) = 15
DEFAULT_PRIVATE_KEY_FILE(default) = None
DEFAULT_PRIVATE_ROLE_VARS(default) = False
DEFAULT_REMOTE_PORT(default) = None
DEFAULT_REMOTE_USER(default) = None
DEFAULT_ROLES_PATH(default) = [u'/mts/home5/rich/.ansible/roles', u'/usr/share/ansible/roles', u'/etc/ansible/roles']
DEFAULT_SCP_IF_SSH(default) = smart
DEFAULT_SELINUX_SPECIAL_FS(default) = ['fuse', 'nfs', 'vboxsf', 'ramfs', '9p']
DEFAULT_SFTP_BATCH_MODE(default) = True
DEFAULT_SQUASH_ACTIONS(default) = ['apk', 'apt', 'dnf', 'homebrew', 'openbsd_pkg', 'pacman', 'pip', 'pkgng', 'yum', 'zypper']
DEFAULT_SSH_TRANSFER_METHOD(default) = None
DEFAULT_STDOUT_CALLBACK(default) = default
DEFAULT_STRATEGY(default) = linear
DEFAULT_STRATEGY_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/strategy', u'/usr/share/ansible/plugins/strategy']
DEFAULT_SU(default) = False
DEFAULT_SUDO(default) = False
DEFAULT_SUDO_EXE(default) = sudo
DEFAULT_SUDO_FLAGS(default) = -H -S -n
DEFAULT_SUDO_USER(default) = None
DEFAULT_SU_EXE(default) = su
DEFAULT_SU_FLAGS(default) = 
DEFAULT_SU_USER(default) = None
DEFAULT_SYSLOG_FACILITY(default) = LOG_USER
DEFAULT_TASK_INCLUDES_STATIC(default) = False
DEFAULT_TERMINAL_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/terminal', u'/usr/share/ansible/plugins/terminal']
DEFAULT_TEST_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/test', u'/usr/share/ansible/plugins/test']
DEFAULT_TIMEOUT(/mts/home5/rich/src/perfauto-ansible/ansible.cfg) = 3600
DEFAULT_TRANSPORT(default) = smart
DEFAULT_UNDEFINED_VAR_BEHAVIOR(default) = True
DEFAULT_VARS_PLUGIN_PATH(default) = [u'/mts/home5/rich/.ansible/plugins/vars', u'/usr/share/ansible/plugins/vars']
DEFAULT_VAULT_ENCRYPT_IDENTITY(default) = None
DEFAULT_VAULT_IDENTITY(default) = default
DEFAULT_VAULT_IDENTITY_LIST(default) = []
DEFAULT_VAULT_ID_MATCH(default) = False
DEFAULT_VAULT_PASSWORD_FILE(default) = None
DEFAULT_VERBOSITY(default) = 0
DEPRECATION_WARNINGS(default) = True
DIFF_ALWAYS(default) = False
DIFF_CONTEXT(default) = 3
DISPLAY_ARGS_TO_STDOUT(default) = False
DISPLAY_SKIPPED_HOSTS(default) = True
ENABLE_TASK_DEBUGGER(default) = False
ERROR_ON_MISSING_HANDLER(default) = True
GALAXY_IGNORE_CERTS(default) = False
GALAXY_ROLE_SKELETON(default) = None
GALAXY_ROLE_SKELETON_IGNORE(default) = ['^.git$', '^.*/.git_keep$']
GALAXY_SERVER(default) = https://galaxy.ansible.com
GALAXY_TOKEN(default) = None
HOST_KEY_CHECKING(default) = True
INJECT_FACTS_AS_VARS(default) = True
INVALID_TASK_ATTRIBUTE_FAILED(default) = True
INVENTORY_ANY_UNPARSED_IS_FAILED(default) = False
INVENTORY_ENABLED(default) = ['host_list', 'script', 'yaml', 'ini', 'auto']
INVENTORY_EXPORT(default) = False
INVENTORY_IGNORE_EXTS(default) = {{(BLACKLIST_EXTS + ( '~', '.orig', '.ini', '.cfg', '.retry'))}}
INVENTORY_IGNORE_PATTERNS(default) = []
INVENTORY_UNPARSED_IS_FAILED(default) = False
LOCALHOST_WARNING(default) = True
MAX_FILE_SIZE_FOR_DIFF(default) = 104448
NETCONF_SSH_CONFIG(default) = None
NETWORK_GROUP_MODULES(default) = ['eos', 'nxos', 'ios', 'iosxr', 'junos', 'enos', 'ce', 'vyos', 'sros', 'dellos9', 'dellos10', 'dellos6', 'asa', 'aruba', 'aireos', 'bigip', 'ironware', 'onyx', 'netconf']
PARAMIKO_HOST_KEY_AUTO_ADD(default) = False
PARAMIKO_LOOK_FOR_KEYS(default) = True
PERSISTENT_COMMAND_TIMEOUT(default) = 10
PERSISTENT_CONNECT_RETRY_TIMEOUT(default) = 15
PERSISTENT_CONNECT_TIMEOUT(default) = 30
PERSISTENT_CONTROL_PATH_DIR(default) = /mts/home5/rich/.ansible/pc
PLAYBOOK_VARS_ROOT(default) = top
PLUGIN_FILTERS_CFG(default) = None
RETRY_FILES_ENABLED(default) = True
RETRY_FILES_SAVE_PATH(default) = None
SHOW_CUSTOM_STATS(default) = False
STRING_TYPE_FILTERS(default) = ['string', 'to_json', 'to_nice_json', 'to_yaml', 'ppretty', 'json']
SYSTEM_WARNINGS(default) = True
TAGS_RUN(default) = []
TAGS_SKIP(default) = []
TASK_DEBUGGER_IGNORE_ERRORS(default) = True
USE_PERSISTENT_CONNECTIONS(default) = False
VARIABLE_PRECEDENCE(default) = ['all_inventory', 'groups_inventory', 'all_plugins_inventory', 'all_plugins_play', 'groups_plugins_inventory', 'groups_plugins_play']
YAML_FILENAME_EXTENSIONS(default) = ['.yml', '.yaml', '.json']

OS / ENVIRONMENT

CentOS 7

STEPS TO REPRODUCE

I am trying to install a package via

yum:
pkg: http://...

EXPECTED RESULTS

That it should work

ACTUAL RESULTS

See stack trace above.

[ansibot]

Files identified in the description:

• lib/ansible/module_utils/urls.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[mkrizek]

From your error:

"msg": "Failed to parse https_proxy environment variable (_none_). Please make sure you export https proxy as 'https_proxy=<SCHEME>://<IP_ADDRESS>:<PORT>'"

it is obvious that you set https_proxy environment variable to _none_ and NOT proxy=_none_ in /etc/yum.repos.d/your_repo_name.repo configuration file which man yum.conf talks about. The latter works as expected with the yum module.

I am not sure setting the environment variable to _none_ value is supposed to work, even the link you provided does not suggest so.

needs_info

[rrauenza]

In my environment, I was doing:

- name: Configure YUM not to use proxy (our proxies have gotten weird.)
  ini_file: dest=/etc/yum.conf section=main option=proxy value=_none_

That triggers this defect. But have changed it to the following and ansible now works again:

- block:
    - name: Remove proxy config from yum.conf
      ini_file: dest=/etc/yum.conf section=main option=proxy state=absent

  tags:
    - yum-proxy

So, no, https_proxy was not set in /etc/yum.repos.d/...

That error message was changed by me into my local source. That was not an ansible error message. The problem was vague until I added extra logging.

It was not an environment variable -- _none_ is a documented and supported value in yum.conf. A recent change to ansible apparently takes whatever is in the yum.conf and pokes it into the environment (or passes to urlllib) because the yum ansible module now appears to predownload http:// packages on behalf of yum.

In yum, this _none_ option forces yum to use no proxy even if the traditional environment variables environment were set. It is explicitly documented in the yum.conf man page:

I quote:

proxy URL to the proxy server for this repository. Set to 'none' to disable the global proxy setting for this repository. If this is unset it inherits it from the global setting

See also other issues brought up in other projects due to how _none_ is handled:
tmatilai/vagrant-proxyconf#146 (comment)
seporaitis/yum-s3-iam#59
https://bugzilla.redhat.com/show_bug.cgi?id=191864

I previously set this in my environment's yum.conf because the corporate proxy was set in our shell environments, but our proxies were flakey with yum -- so I wanted to force yum to never look at the environment on all of the systems I manage. proxy = _none_ is the yum.conf documented workaround. Ansible assuming that the value in yum.conf is always acceptable to put into the environment is a problem. _none_ is explicitly interpreted differently by yum than the libraries that ansible uses to handle http_proxy.

I propose that ansible's yum module never poke the value _none_ into it's proxy settings if it finds specifically that string while parsing the yum.conf. (Or possibly remove http_proxy from env or set it to the empty string if _none_ is found.) If ansible is going to act on yum's behalf and use yum's configuration file to configure the proxy, then it seems like it ought to adopt yum's semantics for the values yum allows in there.