ansible-test integration seems to be ignoring specified python version and interpreter

[geerlingguy]

SUMMARY

For my php_roles collection build, I am using the following ansible-test command (see source):

ansible-test integration --docker geerlingguy/docker-centos7-ansible --python 3.6 --python-interpreter /usr/bin/python3

When the integration tests are run, there are a number of notices about python requirements and versions, then some warnings like:

WARNING: PyYAML will be slow due to installation without libyaml support for interpreter: /usr/bin/python3
WARNING: A Python interpreter was found at "/usr/bin/python", yet a matching pip was not found.
WARNING: A Python 2 interpreter was found at "/usr/bin/python2", yet a matching pip was not found.
WARNING: A Python 2.7 interpreter was found at "/usr/bin/python2.7", yet a matching pip was not found.
Running php_role_test integration test role

And when the integration playbook tasks are run, it seems they're trying to use the incorrect Python 2 environment in the Docker image, as I get errors like:

TASK [php : Ensure PHP packages are installed.] ********************************
355fatal: [testhost]: FAILED! => {"changed": false, "msg": "The Python 2 bindings for rpm are needed for this module. If you require Python 3 support use the `dnf` Ansible module instead.. The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."}

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible-test

ANSIBLE VERSION

ansible 2.9.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/Users/jgeerling/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.6 (default, Feb  9 2020, 13:28:08) [Clang 11.0.0 (clang-1100.0.33.17)]

CONFIGURATION

ANSIBLE_NOCOWS(/etc/ansible/ansible.cfg) = True
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
ANSIBLE_SSH_CONTROL_PATH(/etc/ansible/ansible.cfg) = /tmp/ansible-ssh-%%h-%%p-%%r
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 20
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/etc/ansible/hosts']
DEFAULT_ROLES_PATH(/etc/ansible/ansible.cfg) = ['/Users/jgeerling/Dropbox/VMs/roles']
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False

OS / ENVIRONMENT

macOS or Linux (Ubuntu 18.04)

STEPS TO REPRODUCE

1. (Go into an ansible collections directory structure.)
2. git clone https://github.com/geerlingguy/ansible-collection-php_roles.git
3. Move the cloned project into the path [collections path]/ansible_collections/geerlingguy/php_roles
4. Change directories into the php_roles collection
5. ansible-test integration --docker geerlingguy/docker-centos7-ansible --python 3.6 --python-interpreter /usr/bin/python3

EXPECTED RESULTS

Tests should pass, and Python 3 should be used as specified on the CLI.

ACTUAL RESULTS

TASK [php : Ensure PHP packages are installed.] ********************************
fatal: [testhost]: FAILED! => {"changed": false, "msg": "The Python 2 bindings for rpm are needed for this module. If you require Python 3 support use the `dnf` Ansible module instead.. The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."}

[ansibot]

Files identified in the description:

• bin/ansible-test

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[geerlingguy]

Apparently this is not a bug with ansible-test, it's a misconception I had about the package module and CentOS / Red Hat / Fedora:

The Python 2 bindings for rpm are needed for this module. If you require Python 3 support use the dnf Ansible module instead.. The Python 2 yum module is needed for this module. If you require Python 3 support use the dnf Ansible module instead.

This basically means you have to set ansible_python_interpreter to a Python 2 version (e.g. /usr/bin/python) if you're using CentOS 6 or 7. Even if you have Python 3 installed, there is no support for package management via yum with Python 3 (as I understand it).

Therefore the following task which works fine on CentOS 6/7/8, Fedora 30+, Red Hat, etc. if using the system default Python does not work on CentOS 6/7 if you have Python 3 installed:

- name: Ensure PHP packages are installed.
  package:
    name: "{{ php_packages }}"
    state: "{{ php_packages_state }}"
    enablerepo: "{{ php_enablerepo | default(omit, true) }}"
  notify: restart webserver

For CentOS 8, yum is actually dnf behind the curtains, so the dnf module is used and works with Python 3. For CentOS 7/6 yum doesn't work with Python 2.

[geerlingguy]

So a follow-up question—is there or will there ever be any way to use Python 3 on CentOS 6/7 or RHEL 6/7 and have Ansible work with it with package or yum, or will I have to modify any playbook I use to have a separate additional backwards-compatibility task that specifies system python for just CentOS 6/7 / RHEL 6/7?

[sivel]

So a follow-up question—is there or will there ever be any way to use Python 3 on CentOS 6/7 or RHEL 6/7 and have Ansible work with it with package or yum, or will I have to modify any playbook I use to have a separate additional backwards-compatibility task that specifies system python for just CentOS 6/7 / RHEL 6/7?

Likely, no. But this isn't our decision necessarily.

The OS package maintainers, while they have provided python3 for these older distros, have not packaged the yum or rpm python bindings for python3.

It's possible we could completely rewrite the modules to shell out, instead of using the python bindings, but trying to parse stdout for making structured data would be nearly impossible. I don't see us realistically doing so.

[geerlingguy]

A few more notes:

• The yum docs mention: "This module only works on Python 2. If you require Python 3 support see the dnf module."
• The package docs don't mention anything about Python compatibility, but does at least mention "This module actually calls the pertinent package modules for each system (apt, yum, etc)."
• @maxamillion explains the situation on Reddit here
• @sivel also provides similar details in this comment: Can't install yum package on Amazon Linux 2 when ansible is installed using python 3 #62722 (comment)
• @maxamillion gives some concrete examples of workarounds here: Can't install yum package on Amazon Linux 2 when ansible is installed using python 3 #62722 (comment)

So in the end, it looks like this is not an issue that Ansible itself can resolve, and the best option for me is to (sadly) continue using Python 2 on CentOS 7 and below, until support runs out (in ~2024 or so), or else rewrite any tasks where I'm using package to use yum plus set the python interpreter and use a when clause for centos < 7.