New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Python interpreter discovery #50163

Open
wants to merge 1 commit into
base: devel
from

Conversation

Projects
None yet
4 participants
@nitzmahone
Copy link
Member

nitzmahone commented Dec 19, 2018

SUMMARY
  • No longer blindly default to only /usr/bin/python
  • ansible_python_interpreter defaults to auto_legacy, which will discover the platform Python interpreter on some platforms (but still favor /usr/bin/python if present for backward compatibility). Use auto to always use the discovered interpreter.
ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

BaseAction

ADDITIONAL INFORMATION

Python interpreter discovery
* No longer blindly default to only `/usr/bin/python`
* `ansible_python_interpreter` defaults to `auto_legacy`, which will discover the platform Python interpreter on some platforms (but still favor `/usr/bin/python` if present for backward compatibility). Use `auto` to always use the discovered interpreter.
return found_interpreters[0]

return platform_interpreter
except NotImplementedError as ex:

This comment has been minimized.

@nitzmahone

nitzmahone Dec 19, 2018

Member

Should probably also just catch Exception in another except, so fallback will always occur (but we log the unexpected ones more aggressively and have a different warning).

@ansibot

This comment has been minimized.

Copy link
Contributor

ansibot commented Dec 19, 2018

lines = fd.readlines()

# we only care about a couple of basic values, so this is not a fully-compliant os-release parser
linere = re.compile(r'^([\w_]+)="?([\w_\-]+)"?$')

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

Two bugs:

  • This would need to be a byte string
  • This doesn't handle single quoted values.
Suggested change Beta
linere = re.compile(r'^([\w_]+)="?([\w_\-]+)"?$')
linere = re.compile(br'^([\w_]+)=['"]?([\w_\-]+)['"]?$')
# try to fall back to parsing /etc/os-release if present
osr = parse_os_release()
distro = osr.get('ID')
version = osr.get('VERSION_ID')

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

This won't work on Python3 because parse_os_release() is using byte strings for the key and value but this code will be looking for text versions of the keys.

def main():
info = get_platform_info()

print(json.dumps(info))

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

This will fail on Python3 because bytes objects aren't json serializable on python3.

@abadger

This comment has been minimized.

Copy link
Member

abadger commented Dec 19, 2018

Suggestion: Since all of the bugs I found in the script that's shipped over to the remote host involve what happens to os-release, it might be better to simply read and return os-release to the controller. We can then use libraries on the controller to parse it and decide what to do.

If we always return os-release and the results of platform.dist if available, that could also give us the opportunity to override platform.dist if we need to.

run on a target.

``ansible_python_interpreter`` now defaults to ``auto_legacy``, which consults a lookup table and selects the default
Python if the target platform/version is listed in the table. It will also favor using ``/usr/bin/ansible`` if present,

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

Should this be /usr/bin/python rather than /usr/bin/ansible?

@ansibot ansibot removed the needs_triage label Dec 19, 2018

# FIXME: use one invocation of `command -v` with multiple values and a multiline regex instead?
exec_tmpl = "echo FOUND `command -v '{0}'` ENDFOUND"

python_locator = self._connection._shell._SHELL_AND.join(["uname", ";".join([exec_tmpl.format(t) for t in bootstrap_python_list])])

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

This should be done in the shell plugin

@ansibot

This comment was marked as outdated.

Copy link
Contributor

ansibot commented Dec 19, 2018

The test ansible-test sanity --test ansible-doc --python 3.6 [explain] failed with the error:

Command "ansible-doc -t cache jsonfile memcached memory mongodb pickle redis yaml" returned exit status 1.
>>> Standard Error
Unhandled error:
 Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType


Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/ansible/bin/ansible-doc", line 59, in <module>
    import ansible.constants as C
  File "/root/ansible/lib/ansible/constants.py", line 184, in <module>
    config = ConfigManager()
  File "/root/ansible/lib/ansible/config/manager.py", line 262, in __init__
    self.update_config_data()
  File "/root/ansible/lib/ansible/config/manager.py", line 516, in update_config_data
    raise AnsibleError("Invalid settings supplied for %s: %s\n" % (config, to_native(e)), orig_exc=e)
ansible.errors.AnsibleError: Invalid settings supplied for DEFAULT_INTERPRETER_PYTHON: str expected, not NoneType

The test ansible-test sanity --test ansible-doc --python 3.5 [explain] failed with the error:

Command "ansible-doc -t cache jsonfile memcached memory mongodb pickle redis yaml" returned exit status 1.
>>> Standard Error
Unhandled error:
 Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.5/_collections_abc.py", line 595, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.5/os.py", line 722, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.5/os.py", line 798, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType


Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.5/_collections_abc.py", line 595, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.5/os.py", line 722, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.5/os.py", line 798, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/ansible/bin/ansible-doc", line 59, in <module>
    import ansible.constants as C
  File "/root/ansible/lib/ansible/constants.py", line 184, in <module>
    config = ConfigManager()
  File "/root/ansible/lib/ansible/config/manager.py", line 262, in __init__
    self.update_config_data()
  File "/root/ansible/lib/ansible/config/manager.py", line 516, in update_config_data
    raise AnsibleError("Invalid settings supplied for %s: %s\n" % (config, to_native(e)), orig_exc=e)
ansible.errors.AnsibleError: Invalid settings supplied for DEFAULT_INTERPRETER_PYTHON: str expected, not NoneType

The test ansible-test sanity --test ansible-doc --python 3.7 [explain] failed with the error:

Command "ansible-doc -t cache jsonfile memcached memory mongodb pickle redis yaml" returned exit status 1.
>>> Standard Error
Unhandled error:
 Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.7/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.7/os.py", line 675, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.7/os.py", line 753, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType


Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.7/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.7/os.py", line 675, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.7/os.py", line 753, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/ansible/bin/ansible-doc", line 59, in <module>
    import ansible.constants as C
  File "/root/ansible/lib/ansible/constants.py", line 184, in <module>
    config = ConfigManager()
  File "/root/ansible/lib/ansible/config/manager.py", line 262, in __init__
    self.update_config_data()
  File "/root/ansible/lib/ansible/config/manager.py", line 516, in update_config_data
    raise AnsibleError("Invalid settings supplied for %s: %s\n" % (config, to_native(e)), orig_exc=e)
ansible.errors.AnsibleError: Invalid settings supplied for DEFAULT_INTERPRETER_PYTHON: str expected, not NoneType

The test ansible-test sanity --test docs-build [explain] failed with the error:

Command "/usr/bin/python test/sanity/code-smell/docs-build.py" returned exit status 1.
>>> Standard Error
Command 'make singlehtmldocs' failed with status code: 2
--> Standard Output
cat _themes/srtd/static/css/theme.css | sed -e 's/^[ 	]*//g; s/[ 	]*$//g; s/\([:{;,]\) /\1/g; s/ {/{/g; s/\/\*.*\*\///g; /^$/d' | sed -e :a -e '$!N; s/\n\(.\)/\1/; ta' > _themes/srtd/static/css/theme.min.css
PYTHONPATH=../../lib ../bin/dump_config.py --template-file=../templates/config.rst.j2 --output-dir=rst/reference_appendices/ -d ../../lib/ansible/config/base.yml
mkdir -p rst/cli
PYTHONPATH=../../lib ../bin/generate_man.py --template-file=../templates/cli_rst.j2 --output-dir=rst/cli/ --output-format rst ../../lib/ansible/cli/*.py
Makefile:83: recipe for target 'cli' failed
--> Standard Error
Unhandled error:
 Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType


Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../bin/generate_man.py", line 253, in <module>
    allvars[cli_name] = opts_docs(cli_class_name, cli_name)
  File "../bin/generate_man.py", line 104, in opts_docs
    fromlist=[cli_class_name]), cli_class_name)
  File "/root/ansible/lib/ansible/cli/__init__.py", line 36, in <module>
    from ansible import constants as C
  File "/root/ansible/lib/ansible/constants.py", line 184, in <module>
    config = ConfigManager()
  File "/root/ansible/lib/ansible/config/manager.py", line 262, in __init__
    self.update_config_data()
  File "/root/ansible/lib/ansible/config/manager.py", line 516, in update_config_data
    raise AnsibleError("Invalid settings supplied for %s: %s\n" % (config, to_native(e)), orig_exc=e)
ansible.errors.AnsibleError: Invalid settings supplied for DEFAULT_INTERPRETER_PYTHON: str expected, not NoneType

make: *** [cli] Error 1

The test ansible-test sanity --test changelog [explain] failed with the error:

Command "/usr/bin/python test/sanity/code-smell/changelog.py" returned exit status 1.
>>> Standard Error
Unhandled error:
 Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType


Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "packaging/release/changelogs/changelog.py", line 27, in <module>
    from ansible import constants as C
  File "/root/ansible/lib/ansible/constants.py", line 184, in <module>
    config = ConfigManager()
  File "/root/ansible/lib/ansible/config/manager.py", line 262, in __init__
    self.update_config_data()
  File "/root/ansible/lib/ansible/config/manager.py", line 516, in update_config_data
    raise AnsibleError("Invalid settings supplied for %s: %s\n" % (config, to_native(e)), orig_exc=e)
ansible.errors.AnsibleError: Invalid settings supplied for DEFAULT_INTERPRETER_PYTHON: str expected, not NoneType

Traceback (most recent call last):
  File "test/sanity/code-smell/changelog.py", line 14, in <module>
    main()
  File "test/sanity/code-smell/changelog.py", line 10, in main
    subprocess.check_call(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['packaging/release/changelogs/changelog.py', 'lint', 'changelogs/fragments/11349-add-ansible_play_name-var.yaml', 'changelogs/fragments/11935-slack-add_hex_color_values.yaml', 'changelogs/fragments/2.8-core-deprecations.yaml', 'changelogs/fragments/2.8-removed-modules.yaml', 'changelogs/fragments/33754-docker_image_fix_changed_in_force_mode.yaml', 'changelogs/fragments/35370-add_support_for_docker_network_internal_flag.yaml', 'changelogs/fragments/38833-docker_volume-option-minimum-versions', 'changelogs/fragments/40174-junit-test-case-prefix-filter.yaml', 'changelogs/fragments/40176-junit-hide-task-arguments.yaml', 'changelogs/fragments/42866-galaxy-search-unicode.yaml', 'changelogs/fragments/43123-add_support_for_per_host_no_stats.yaml', 'changelogs/fragments/43874-docker_container-stop_timeout.yaml', 'changelogs/fragments/44278-pamd_valid_simple_controls.yaml', 'changelogs/fragments/44428-inventory-plugin-list.yml', 'changelogs/fragments/44789-docker_container-comparisons.yaml', 'changelogs/fragments/44988-acme-post-as-get.yaml', 'changelogs/fragments/45155-vmware_host_service_facts-update_docs.yaml', 'changelogs/fragments/45628-fetch_url-error-headers.yaml', 'changelogs/fragments/46322-docker_container-image-not-given.yaml', 'changelogs/fragments/46483-role_names-change.yaml', 'changelogs/fragments/46594-docker_container-publish-all-ports.yml', 'changelogs/fragments/46595-docker_container-expected_ports.yml', 'changelogs/fragments/46596-docker_container-published_ports.yml', 'changelogs/fragments/46598-docker_container-volume-modes.yml', 'changelogs/fragments/46608_azure_rm_inv_py3.yaml', 'changelogs/fragments/46658-plugin_filter-improve_error_handling.yaml', 'changelogs/fragments/46739-gcp-compute-instance-metadata.yaml', 'changelogs/fragments/46740-gcp-utils-credentials-scoping.yaml', 'changelogs/fragments/46743-fix-native-jinja-newlines.yaml', 'changelogs/fragments/46772-docker_container-healthcheck.yaml', 'changelogs/fragments/46961_fix_aws_ec2_cache.yaml', 'changelogs/fragments/47019-mail-fix-py27-regression.yaml', 'changelogs/fragments/47134-elasticsearch_plugin-fix_param_type.yml', 'changelogs/fragments/47193-fix-gce-and-scaleway-vm-detection.yml', 'changelogs/fragments/47213-onepassword_facts_fix_password_lookup.yaml', 'changelogs/fragments/47247-docker_container-add-runtime-option.yaml', 'changelogs/fragments/47281-pamd-dont-delete-named_temporary_file_on_close.yaml', 'changelogs/fragments/47300-ios-check_rc.yaml', 'changelogs/fragments/47307-handler-include-task.yml', 'changelogs/fragments/47313-vmware-fix_module_error.yaml', 'changelogs/fragments/47393-docker_image-id.yaml', 'changelogs/fragments/47395-docker_container-ipvX_address.yaml', 'changelogs/fragments/47396-docker_container-detach-auto-remove.yaml', 'changelogs/fragments/47459_grafana_dashboard_consistency_fix.yaml', 'changelogs/fragments/47492-docker_network-add-ipv6-support.yaml', 'changelogs/fragments/47500-rds_instance.yaml', 'changelogs/fragments/47539-fix-netaddr-network.yaml', 'changelogs/fragments/47668-aci_switch_leaf_selector-support_empty_policy_group.yaml', 'changelogs/fragments/47689-yum-fix-version-syntax.yaml', 'changelogs/fragments/47695-pamd-fix-idempotence-and-parsing-issues.yml', 'changelogs/fragments/47704-apt-warn-auto-intall.yml', 'changelogs/fragments/47711-docker_container-minimal-version-checks.yml', 'changelogs/fragments/47712-docker_container-detach-auto-remove.yml', 'changelogs/fragments/47722-vmware_guest_powerstate-restore_timeout.yaml', 'changelogs/fragments/47814-docker_container-device-io-limit-parameters.yaml', 'changelogs/fragments/47846-cs_ip_address-fix-vpc-vs-network.yaml', 'changelogs/fragments/47859-vmware_guest-convert_vm_disk_clone.yaml', 'changelogs/fragments/47900-docker_container-paused.yml', 'changelogs/fragments/47916-grafana_dashboard-fix-logic-behind-overwrite-param.yaml', 'changelogs/fragments/47920-vmware_guest-handle_no_root_snapshot.yaml', 'changelogs/fragments/47938-docker_swarm_service-requirements.yaml', 'changelogs/fragments/47997-docker_container-ipc-pid-mode.yml', 'changelogs/fragments/48036-vultr-fix-empty-list-handling.yaml', 'changelogs/fragments/48061-docker_container-auto_removal.yml', 'changelogs/fragments/48342-vultr_server_facts-fix-firewall-group.yml', 'changelogs/fragments/48471-win_xml-xml-parser.yaml', 'changelogs/fragments/48536-docker_volume-labels.yml', 'changelogs/fragments/48546-docker-diff.yml', 'changelogs/fragments/48551-docker_container-idempotency.yml', 'changelogs/fragments/48599-rabbitmq_binding-state-absent.yaml', 'changelogs/fragments/48673-fix-omit-on-play-keywords.yaml', 'changelogs/fragments/48675-cs_template-fix-keyerror-state-extracted.yml', 'changelogs/fragments/48728-win_nssm-credential-quoting.yml', 'changelogs/fragments/48730-zabbix_hostmacro-fixes.yaml', 'changelogs/fragments/48936-import-handlers.yaml', 'changelogs/fragments/48950-vault-encrypted-data-native-jinja.yaml', 'changelogs/fragments/49078-docker_container-min-version-fix.yml', 'changelogs/fragments/49084-influxdb_user-default-password-fix.yaml', 'changelogs/fragments/49158-detect-kvm-on-freebsd.yaml', 'changelogs/fragments/49188-zabbix_template-fix-idempotency.yaml', 'changelogs/fragments/49212-require-git-ansible-galaxy.yaml', 'changelogs/fragments/49235-docker_swarm_service-user-default.yaml', 'changelogs/fragments/49266-acme-error-messages.yml', 'changelogs/fragments/49319-docker_container-pids_limit.yaml', 'changelogs/fragments/49409-lineinfile_must_not_insert_lines_multiples_times_with_insertbefore_insertafter.yml', 'changelogs/fragments/49410-acme-diff.yml', 'changelogs/fragments/49473-multiple-ipv6-addresses-per-device.yaml', 'changelogs/fragments/49545-ansible-doc_version_help.yaml', 'changelogs/fragments/49553-aci_rest-fix-ignoring-custom-port.yaml', 'changelogs/fragments/49676-s3bucket-requester_pays_change_if_condition.yml', 'changelogs/fragments/49787-docker_image-cache_from.yaml', 'changelogs/fragments/49794-docker_container-network-mode.yml', 'changelogs/fragments/49843-docker_container-wrap-env.yaml', 'changelogs/fragments/49884-tower-project-scm-cred-org-fallback.yaml', 'changelogs/fragments/aci_access_port_to_interface_policy_leaf_profile-missing_policy_group.yaml', 'changelogs/fragments/aci_interface_policy_leaf_policy_group-missing_aep.yaml', 'changelogs/fragments/add-elapsed-return-value-to-select-modules.yaml', 'changelogs/fragments/agnostic-become-prompt.yaml', 'changelogs/fragments/ajson-nested-decode.yaml', 'changelogs/fragments/ansible-doc-fixes.yml', 'changelogs/fragments/async-dir.yaml', 'changelogs/fragments/async_statys_pyx_compat_fix.yml', 'changelogs/fragments/avoid_spurious_unique_warnings.yml', 'changelogs/fragments/avoid_ssh_retry_discolsures.yml', 'changelogs/fragments/azure_rm_appgateway-probe.yaml', 'changelogs/fragments/azure_rm_deployment_fix_45941.yaml', 'changelogs/fragments/better-kv-error-reporting.yml', 'changelogs/fragments/better_cfgmgr_errors.yml', 'changelogs/fragments/blockinfile-bytes-fix.yaml', 'changelogs/fragments/callback-keep-more-debug-keys.yml', 'changelogs/fragments/cfg_mgr_fix.yml', 'changelogs/fragments/clear_system_variablse.yml', 'changelogs/fragments/code-cleanup-no-get-exception.yaml', 'changelogs/fragments/command-stdin-no-newline.yaml', 'changelogs/fragments/contains-test.yaml', 'changelogs/fragments/copy-diff-text.yaml', 'changelogs/fragments/copy-recursive-remote-src.yml', 'changelogs/fragments/dd-put-empty-files.yaml', 'changelogs/fragments/deal_with_bad_config_types.yml', 'changelogs/fragments/delegate_to_loop_hostvars.yaml', 'changelogs/fragments/detect_interpreter_stdout.yml', 'changelogs/fragments/diff_yaml.yml', 'changelogs/fragments/display-singleton.yaml', 'changelogs/fragments/dnf-fix-plugin-loading.yaml', 'changelogs/fragments/dnf-group-removal.yaml', 'changelogs/fragments/dnf-localgpgcheck.yaml', 'changelogs/fragments/dnf-modularity.yaml', 'changelogs/fragments/dnfyum-disable-excludes.yaml', 'changelogs/fragments/doc_debug_var.yml', 'changelogs/fragments/docfixes.yml', 'changelogs/fragments/docker-image-ids.yaml', 'changelogs/fragments/docker-swarm-service-defaults.yml', 'changelogs/fragments/docker_container-idempotency.yaml', 'changelogs/fragments/docker_network-adding-scope-and-attachable-flags.yaml', 'changelogs/fragments/docker_network-driver_options.yaml', 'changelogs/fragments/docker_network-requirements.yaml', 'changelogs/fragments/docker_volume-force-change-detection.yaml', 'changelogs/fragments/drop-pkg_resources.yaml', 'changelogs/fragments/ec2_asg-launch-template-support.yml', 'changelogs/fragments/ec2_asg_retry_deletion_when_busy.yaml', 'changelogs/fragments/ec2_group_fix_target_containing_list_within_list.yaml', 'changelogs/fragments/ec2_vpc_peer_describe_peer_with_exception_handling.yaml', 'changelogs/fragments/elb_target_group_fix_KeyError.yaml', 'changelogs/fragments/end-host-meta-task.yaml', 'changelogs/fragments/explain_bare.yml', 'changelogs/fragments/fix-callbacks-mixed-keys.yaml', 'changelogs/fragments/fix-password-lookup-on-fips.yaml', 'changelogs/fragments/fix_adhoc_includes.yml', 'changelogs/fragments/fix_ec2_group_target_vpc_precedence.yaml', 'changelogs/fragments/fix_ec2_group_vpc_precedence_classic.yaml', 'changelogs/fragments/fix_taggged_gather.yml', 'changelogs/fragments/free-strategy-include-var-tags.yaml', 'changelogs/fragments/from_handlers.yml', 'changelogs/fragments/get-url-fix-idempotency.yaml', 'changelogs/fragments/get_url.yaml', 'changelogs/fragments/host-start-callback.yaml', 'changelogs/fragments/ibm-storag_add_domain_keywords_to_module.yml', 'changelogs/fragments/include-run-once.yaml', 'changelogs/fragments/influxdb_user-admin-role-update.yaml', 'changelogs/fragments/interpreter_discovery.yaml', 'changelogs/fragments/inv_fixes.yml', 'changelogs/fragments/inventory-docker-service-stack-groups.yaml', 'changelogs/fragments/iscsi_facts_hp-ux_aix.yaml', 'changelogs/fragments/jinja-now.yml', 'changelogs/fragments/jinja2_native-fallback-warning.yaml', 'changelogs/fragments/jira_fix_description_field.yaml', 'changelogs/fragments/k8s_append_hash.yml', 'changelogs/fragments/k8s_facts_fix.yaml', 'changelogs/fragments/k8s_validate.yml', 'changelogs/fragments/k8s_wait.yml', 'changelogs/fragments/last-loaded-handler-same-name-wins.yaml', 'changelogs/fragments/lineinfile-insertbefore-index-out-of-range.yaml', 'changelogs/fragments/loop-cache-include-apply.yml', 'changelogs/fragments/loop-control-label-template-error.yaml', 'changelogs/fragments/loop-empty-literal-list.yaml', 'changelogs/fragments/loop-info.yaml', 'changelogs/fragments/loop_undefined_delegate_to.yaml', 'changelogs/fragments/macports-upgrade-selfupdate.yml', 'changelogs/fragments/mathstuff-filter-py3-scope.yaml', 'changelogs/fragments/mysql-migrate_to_pymysql.yaml', 'changelogs/fragments/nicer_role_list.yml', 'changelogs/fragments/no-mutable-fieldattribute-defaults.yaml', 'changelogs/fragments/no-overwrite-roles.yaml', 'changelogs/fragments/no_empty_groups.yml', 'changelogs/fragments/omit-list-of-dicts.yaml', 'changelogs/fragments/openssl-python3.yaml', 'changelogs/fragments/openstack_inventory_fix.yml', 'changelogs/fragments/os-server-facts-all-projects.yaml', 'changelogs/fragments/ovirt_host_network_fix_type_conversion.yaml', 'changelogs/fragments/pear_better_error.yml', 'changelogs/fragments/pip-fix-idempotence-in-check-mode.yaml', 'changelogs/fragments/piped-transfer-empty-files.yaml', 'changelogs/fragments/platform-dist-to-nir0s-distro.yaml', 'changelogs/fragments/playbook-order-py3.yaml', 'changelogs/fragments/plugin-docs-list-fix.yaml', 'changelogs/fragments/plugin-filters-cfg.yaml', 'changelogs/fragments/postgresql_user-not-sup-error.yaml', 'changelogs/fragments/powershell_basic_util.yaml', 'changelogs/fragments/ps_sb_logging.yaml', 'changelogs/fragments/psexec-handle-socket-errors.yaml', 'changelogs/fragments/psexec-imp-error.yaml', 'changelogs/fragments/psrp-utf8-stdio.yaml', 'changelogs/fragments/psrp-utf8.yaml', 'changelogs/fragments/reboot-change-default-boot-command.yaml', 'changelogs/fragments/reboot-fix-exception-type.yaml', 'changelogs/fragments/reboot-path-alpine-solaris.yaml', 'changelogs/fragments/reboot-show-timeout.yaml', 'changelogs/fragments/reboot-unicode-string.yaml', 'changelogs/fragments/reboot-vmware-esxi.yaml', 'changelogs/fragments/reboot_missing_parameter.yaml', 'changelogs/fragments/reboot_openbsd_support.yaml', 'changelogs/fragments/redis-3-compat.yaml', 'changelogs/fragments/remove_redundant_md5.yml', 'changelogs/fragments/restore_sigpipe_dfl.yml', 'changelogs/fragments/rhn_regiter-user-pass-unregister.yaml', 'changelogs/fragments/run-command-expand-shell.yaml', 'changelogs/fragments/s3_bucket_delete_nonexistent_bucket.yml', 'changelogs/fragments/s3_bucket_fix_non_str_tags.yaml', 'changelogs/fragments/s3_bucket_requester_pays_default_value.yaml', 'changelogs/fragments/s3_bucket_walrus_endpoint.yaml', 'changelogs/fragments/scaleway-getheaders.yaml', 'changelogs/fragments/script-module-no-file-path.yaml', 'changelogs/fragments/sns-boto3.yaml', 'changelogs/fragments/solaris-prtdiag-path.yaml', 'changelogs/fragments/squash-deprecation-message.yml', 'changelogs/fragments/synchronize-warning.yaml', 'changelogs/fragments/tag_gathering.yml', 'changelogs/fragments/tags-var.yaml', 'changelogs/fragments/toml-inventory.yaml', 'changelogs/fragments/tower_credential_ssh_key_data.yaml', 'changelogs/fragments/tweek_msg.yml', 'changelogs/fragments/unsafe-set-wrap.yaml', 'changelogs/fragments/unsafe_cleanup.yml', 'changelogs/fragments/uri-supports-async.yaml', 'changelogs/fragments/user-do-not-pass-ssh_key_passphrase-on-cmdline.yaml', 'changelogs/fragments/user-docs-underlying-tools.yaml', 'changelogs/fragments/user-fix-zero-negative-expiration.yaml', 'changelogs/fragments/user-password_lock-change-fix.yaml', 'changelogs/fragments/user-restore-disabled-account.yaml', 'changelogs/fragments/v2.8.0-initial-commit.yaml', 'changelogs/fragments/vault-read-error.yml', 'changelogs/fragments/vbox_fix.yml', 'changelogs/fragments/vm_fix.yml', 'changelogs/fragments/vm_guest_facts.yml', 'changelogs/fragments/win-say-ansible-basic.yaml', 'changelogs/fragments/win_become-passwordless.yaml', 'changelogs/fragments/win_chocolatey-allow-multiple.yaml', 'changelogs/fragments/win_copy-dest-quote.yaml', 'changelogs/fragments/win_copy-empty-dir.yaml', 'changelogs/fragments/win_group_membership-com-marshal.yaml', 'changelogs/fragments/win_lineinfile-output.yaml', 'changelogs/fragments/win_mapped_drive-fixes.yaml', 'changelogs/fragments/win_package_chdir.yaml', 'changelogs/fragments/win_privileges_util.yaml', 'changelogs/fragments/win_route.yaml', 'changelogs/fragments/win_say-fix.yaml', 'changelogs/fragments/win_scheduled_task-repetition.yaml', 'changelogs/fragments/win_script-become.yaml', 'changelogs/fragments/win_security_policy-rights.yaml', 'changelogs/fragments/win_shortcut.yaml', 'changelogs/fragments/win_updates-post-categories.yaml', 'changelogs/fragments/win_uri-junk-data.yaml', 'changelogs/fragments/win_uri-oneitem-list.yaml', 'changelogs/fragments/windows-deprecated-functionality.yaml', 'changelogs/fragments/windows-exec-changes.yaml', 'changelogs/fragments/windows-psrp-unreachable.yaml', 'changelogs/fragments/winrm_pexpect.yaml', 'changelogs/fragments/yaml_inventory_more_tolerant.yml', 'changelogs/fragments/yum-handle-obsoletes-check-update.yaml', 'changelogs/fragments/yumdnf-autoremove.yaml', 'changelogs/fragments/yumdnf-better-uri-handling.yaml', 'changelogs/fragments/yumdnf-update-cache.yaml', 'changelogs/fragments/zabbix_inventory_vars.yaml']' returned non-zero exit status 1.

The test ansible-test sanity --test integration-aliases [explain] failed with 1 error:

test/integration/targets/interpreter_discovery_python/aliases:0:0: missing alias `shippable/posix/group[1-3]` or `unsupported`

The test ansible-test sanity --test no-tests-as-filters [explain] failed with the error:

Command "/usr/bin/python test/sanity/code-smell/no-tests-as-filters.py" returned exit status 1.
>>> Standard Error
Unhandled error:
 Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType


Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 504, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/root/ansible/lib/ansible/config/manager.py", line 407, in get_config_value_and_origin
    value, origin = self._loop_entries(py3compat.environ, defs[config]['env'])
  File "/root/ansible/lib/ansible/config/manager.py", line 353, in _loop_entries
    temp_value = container.get(name, None)
  File "/usr/lib/python3.6/_collections_abc.py", line 660, in get
    return self[key]
  File "/root/ansible/lib/ansible/utils/py3compat.py", line 43, in __getitem__
    value = self._raw_environ[key]
  File "/usr/lib/python3.6/os.py", line 666, in __getitem__
    value = self._data[self.encodekey(key)]
  File "/usr/lib/python3.6/os.py", line 744, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test/sanity/code-smell/no-tests-as-filters.py", line 25, in <module>
    from ansible.plugins.test import core, files, mathstuff
  File "/root/ansible/lib/ansible/plugins/__init__.py", line 26, in <module>
    from ansible import constants as C
  File "/root/ansible/lib/ansible/constants.py", line 184, in <module>
    config = ConfigManager()
  File "/root/ansible/lib/ansible/config/manager.py", line 262, in __init__
    self.update_config_data()
  File "/root/ansible/lib/ansible/config/manager.py", line 516, in update_config_data
    raise AnsibleError("Invalid settings supplied for %s: %s\n" % (config, to_native(e)), orig_exc=e)
ansible.errors.AnsibleError: Invalid settings supplied for DEFAULT_INTERPRETER_PYTHON: str expected, not NoneType

click here for bot help

rhel: *rhelish
ubuntu:
'14': /usr/bin/python
'16': /usr/bin/python3

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

Is there a way for users to specify dicts in their config files? Or is this a config setting that they won't be able to override because they can't match the data format?

This comment has been minimized.

@sivel

sivel Dec 19, 2018

Member

There is no ini or env here, so it wouldn't be modifiable via config. Also, until we support YAML configs, it wouldn't be settable by a user even if it was exposed.

@@ -890,3 +918,16 @@ def modify_module(module_name, module_path, module_args, templar, task_vars=None
b_module_data = b"\n".join(b_lines)

return (b_module_data, module_style, shebang)


class InterpreterDiscoveryRequiredError(Exception):

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

Style: Classes at the top of the file.

return self.message

def __repr__(self):
return self.message

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

repr should say how to recreate the class whenever possible:

>>> repr(err)
"ZeroDivisionError('integer division or modulo by zero',)"
>>> print(err)
integer division or modulo by zero
Suggested change Beta
return self.message
return 'InterpreterDiscoveryRequiredError({0}, {1}, {2})'.format(self.message, self.interpreter_name, self.discover_mode)

python_locator = self._connection._shell._SHELL_AND.join(["uname", ";".join([exec_tmpl.format(t) for t in bootstrap_python_list])])

# FUTURE: in most cases we probably don't want to use become, but maybe sometimes we do?

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

Depends on whether we want to detect ~/bin/python ~.local/bin/python and similar things.

# check for first-class interpreter config
interpreter_config_key = "DEFAULT_INTERPRETER_%s" % interpreter_name.upper()

if C.config.get_configuration_definitions().get(interpreter_config_key):

This comment has been minimized.

@abadger

abadger Dec 19, 2018

Member

It looks like this conditional is slightly off? If ansible_python_interpreter is set, then it needs to use that. But in this code, it looks like the value of DEFAULT_INTERPRETER_PYTHON causes that to be ignored.

This comment has been minimized.

@sivel

sivel Dec 19, 2018

Member

I believe based on what I understand, that the later get_config_value in which we pass task_vars will sort this out. I have not test that however.

I think I was misunderstanding. We should prefer ansible_python_interpreter over the config, so we might need to use get_config_value as is used below instead of this more generic get

This comment has been minimized.

@nitzmahone

nitzmahone Jan 9, 2019

Member

This is to determine if the interpreter has config definitions present for discovery at all, not what their values are. For Python, this should always be true (which delegates the handling of the ansible_python_interpreter var value to the config system- see the config definition for DEFAULT_INTERPRETER_PYTHON), for anything else, it drops into the else that uses the old behavior of just consulting the task vars.

@@ -685,6 +685,61 @@ DEFAULT_INTERNAL_POLL_INTERVAL:
Higher values are more suitable for Ansible usage in automation scenarios,
when UI responsiveness is not required but CPU usage might be a concern.
- "The default corresponds to the value hardcoded in Ansible <= 2.1"
DEFAULT_INTERPRETER_PYTHON:

This comment has been minimized.

@sivel

sivel Dec 19, 2018

Member

It's been discussed that we should not use the DEFAULT_ prefix on new config values. They were kept for older historical "constants", but shouldn't be used going forward.

This comment has been minimized.

@sivel

sivel Dec 19, 2018

Member

I'd probably just match the env name.

@@ -29,7 +31,7 @@
from ansible.utils.display import Display
from ansible.utils.unsafe_proxy import wrap_var
from ansible.vars.clean import remove_internal_keys

from distutils.version import LooseVersion

This comment has been minimized.

@sivel

sivel Dec 19, 2018

Member

LooseVersion or anything from distutils.version is problematic on Python3. I haven't fully understood how this is being used yet, but due to more strict type comparisons in Python3, comparing LooseVersion can result in a traceback if you compare a string and an integer. See #34427

I had an implementation of SafeLooseVersion that would cast all values to strings after LooseVersion did its parsing.

Something like:

    def parse(self, vstring):
        super(SafeLooseVersion, self).parse(vstring)
        self.version[:] = [str(v) for v in self.version]

This comment has been minimized.

@nitzmahone

nitzmahone Jan 9, 2019

Member

This shouldn't cause a problem with anything we know about today, though running discovery on (say) a beta version of a listed OS that includes text in the version string could cause it to blow up and go into a fallback mode on the bisect. The impl you have above will sort incorrectly on values requiring natural sort semantics (any component > 9, eg, 7.10 > 7.9). Switching to something like packaging.version.Version would likely cause problems as well (beyond the additional dep requirement), as IIRC it's very PEP-440-flavored, which the OS version numbers aren't guaranteed to look like.

I'm going to leave it as-is for now- if it ends up being a problem down the line, we'll probably just need to roll our own LooseVersion impl that behaves more like the py2 version. The version_compare test will suffer from the same issue, and I'm sure there are other places that will as well. I'm frankly a little amazed that this regression has existed in py3 for so long...

@ansibot ansibot added the stale_ci label Dec 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment