New install of 2.10.5 via Pip (3) results in no binaries #73390

hinchliff · 2021-01-27T16:54:44Z

SUMMARY

Started a new system, running RHEL 7.9, using Python 3.6.8 and pip 21.0. Used pip to install ansible 2.10.5, and while no errors were given, the resulting installation is missing the expected ansible "binaries" -- e.g. in /usr/bin/ansible

ISSUE TYPE

Bug Report

COMPONENT NAME

pip packaging

ANSIBLE VERSION

--

CONFIGURATION

--

OS / ENVIRONMENT

Red Hat Enterprise Linux Server release 7.9 (Maipo)
Python 3.6.8
pip 21.0 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
ansible 2.10.5 from a project requirements.txt file

# cat requirements.txt 
ansible==2.10.5
dnspython==2.1.0
requests==2.24.0
pywinrm==0.4.1
pykerberos==1.2.1
# for ansible-kafka-admin
kafka-python==2.0.2
kazoo==2.8.0
pure-sasl==0.6.2

STEPS TO REPRODUCE

Starting from a fresh install of RHEL 7, using the root user:

yum install -y   git   python3   python3-pip   python3-devel   krb5-devel   krb5-workstation   krb5-libs
yum group install -y "Development Tools"
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt # shown above, ansible 2.10.5

EXPECTED RESULTS

Have binaries installed

ACTUAL RESULTS

No ansible "binaries" are available. It is not possible to run ansible commands:

# ansible -v
-bash: ansible: command not found

# python3 -m pip install -r requirements3.txt
Collecting ansible==2.10.5
  Downloading ansible-2.10.5.tar.gz (29.1 MB)
     |████████████████████████████████| 29.1 MB 36.2 MB/s 
Collecting dnspython
  Downloading dnspython-2.1.0-py3-none-any.whl (241 kB)
     |████████████████████████████████| 241 kB 74.4 MB/s 
Requirement already satisfied: requests in /usr/local/lib/python3.6/site-packages (from -r requirements3.txt (line 3)) (2.24.0)
Collecting pywinrm
  Downloading pywinrm-0.4.1.tar.gz (36 kB)
Collecting pykerberos
  Downloading pykerberos-1.2.1.tar.gz (24 kB)
Collecting kafka-python
  Downloading kafka_python-2.0.2-py2.py3-none-any.whl (246 kB)
     |████████████████████████████████| 246 kB 69.5 MB/s 
Collecting kazoo
  Downloading kazoo-2.8.0-py2.py3-none-any.whl (142 kB)
     |████████████████████████████████| 142 kB 99.3 MB/s 
Collecting pure-sasl
  Downloading pure-sasl-0.6.2.tar.gz (11 kB)
Collecting ansible-base<2.11,>=2.10.4
  Downloading ansible-base-2.10.5.tar.gz (5.7 MB)
     |████████████████████████████████| 5.7 MB 37.7 MB/s 
Collecting jinja2
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 52.8 MB/s 
Collecting PyYAML
  Downloading PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
     |████████████████████████████████| 640 kB 65.9 MB/s 
Collecting cryptography
  Downloading cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
     |████████████████████████████████| 2.6 MB 43.6 MB/s 
Collecting packaging
  Downloading packaging-20.8-py2.py3-none-any.whl (39 kB)
Collecting six
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting xmltodict
  Downloading xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Collecting requests_ntlm>=0.3.0
  Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/site-packages (from requests->-r requirements3.txt (line 3)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/site-packages (from requests->-r requirements3.txt (line 3)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests->-r requirements3.txt (line 3)) (2020.12.5)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests->-r requirements3.txt (line 3)) (1.25.11)
Collecting ntlm-auth>=1.0.2
  Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
Collecting cffi>=1.12
  Downloading cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl (401 kB)
     |████████████████████████████████| 401 kB 73.3 MB/s 
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 92.3 MB/s 
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 17.5 MB/s 
Using legacy 'setup.py install' for ansible, since package 'wheel' is not installed.
Using legacy 'setup.py install' for ansible-base, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pure-sasl, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pykerberos, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pywinrm, since package 'wheel' is not installed.
Installing collected packages: pycparser, six, pyparsing, MarkupSafe, cffi, PyYAML, packaging, ntlm-auth, jinja2, cryptography, xmltodict, requests-ntlm, ansible-base, pywinrm, pykerberos, pure-sasl, kazoo, kafka-python, dnspython, ansible
    Running setup.py install for ansible-base ... done
    Running setup.py install for pywinrm ... done
    Running setup.py install for pykerberos ... done
    Running setup.py install for pure-sasl ... done
    Running setup.py install for ansible ... done
Successfully installed MarkupSafe-1.1.1 PyYAML-5.4.1 ansible-2.10.5 ansible-base-2.10.5 cffi-1.14.4 cryptography-3.3.1 dnspython-2.1.0 jinja2-2.11.2 kafka-python-2.0.2 kazoo-2.8.0 ntlm-auth-1.5.0 packaging-20.8 pure-sasl-0.6.2 pycparser-2.20 pykerberos-1.2.1 pyparsing-2.4.7 pywinrm-0.4.1 requests-ntlm-1.1.0 six-1.15.0 xmltodict-0.12.0


# pip3 show ansible
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Name: ansible
Version: 2.10.5
Summary: Radically simple IT automation
Home-page: https://ansible.com/
Author: Ansible, Inc.
Author-email: info@ansible.com
License: GPLv3+
Location: /usr/local/lib/python3.6/site-packages
Requires: ansible-base
Required-by: 

# pip3 show ansible-base
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Name: ansible-base
Version: 2.10.5
Summary: Radically simple IT automation
Home-page: https://ansible.com/
Author: Ansible, Inc.
Author-email: info@ansible.com
License: GPLv3+
Location: /usr/local/lib/python3.6/site-packages
Requires: jinja2, PyYAML, cryptography, packaging
Required-by: ansible

Is there a default installation of older ansible (using Python 2)? Does that need to be removed? Should this be captured in the ansible pip installation, like from #70348 ? But I don't think there is any previous version installed...

# locate ansible
/usr/lib/python2.7/site-packages/sos/plugins/ansible.py
/usr/lib/python2.7/site-packages/sos/plugins/ansible.pyc
/usr/lib/python2.7/site-packages/sos/plugins/ansible.pyo
/usr/lib/python2.7/site-packages/sos/plugins/ceph_ansible.py
/usr/lib/python2.7/site-packages/sos/plugins/ceph_ansible.pyc
/usr/lib/python2.7/site-packages/sos/plugins/ceph_ansible.pyo
/usr/lib/python2.7/site-packages/sos/plugins/openstack_ansible.py
/usr/lib/python2.7/site-packages/sos/plugins/openstack_ansible.pyc
/usr/lib/python2.7/site-packages/sos/plugins/openstack_ansible.pyo
/usr/share/doc/vdo/examples/ansible
/usr/share/doc/vdo/examples/ansible/README.txt
/usr/share/doc/vdo/examples/ansible/test_vdocreate.yml
/usr/share/doc/vdo/examples/ansible/test_vdocreate_alloptions.yml
/usr/share/doc/vdo/examples/ansible/test_vdoremove.yml
/var/cache/yum/x86_64/7Server/rhel-7-server-ansible-2-rpms
/var/cache/yum/x86_64/7Server/rhel-7-server-ansible-2-rpms/gen
/var/cache/yum/x86_64/7Server/rhel-7-server-ansible-2-rpms/packages
/var/lib/yum/repos/x86_64/7Server/rhel-7-server-ansible-2-rpms

# python2 -m pip show ansible
/bin/python2: No module named pip

The yum package is not installed:

# yum search ansible
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
=============================================================================== N/S matched: ansible ===============================================================================
ansible-doc.noarch : Documentation for Ansible
ansible.noarch : SSH-based configuration management, deployment, and task execution system

  Name and summary matches only, use "search all" for everything.

# rpm -q ansible
package ansible is not installed

The text was updated successfully, but these errors were encountered:

ansibot · 2021-01-27T16:55:30Z

Files identified in the description:

lib/ansible/modules/package.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

sivel · 2021-01-27T17:29:16Z

That output doesn't seem to indicate it found another Ansible install, but I cannot reproduce this on a clean EL7 (In my case I used a docker container).

Without the ability to reproduce this, you will likely need to repeat the steps and run the pip install with -v to provide verbose info.

webknjaz · 2021-01-27T17:45:27Z

@hinchliff I'd like to add that installing packages with pip into system-wide locations is a bad idea because conflicting with files that yum/dnf claims may damage the system.
Have you considered using a virtualenv?
What is the version of setuptools under the same interpreter/env as pip?
Also, try adding -vv --no-cache-dir to get more logs.

hinchliff · 2021-01-27T18:42:23Z

Ok, I uninstalled ansible, and reinstalled with -vv --no-cache-dir

python3 -m pip uninstall -r requirements.txt
python3 -m pip freeze # indicated ansible-base was still installed
python3 -m pip uninstall ansible-base
python3 -m pip install -vv --no-cache-dir ansible==2.10.5

Resulting in huge output file, captured in this gist: https://gist.github.com/hinchliff/cd6e288291ef35e2f0fd4144525bf4dd

And still no luck running ansible / no binaries available:

# ansible -v
-bash: ansible: command not found

Setuptools info

# python3 -m pip show setuptools
Name: setuptools
Version: 39.2.0
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: Python Packaging Authority
Author-email: distutils-sig@python.org
License: UNKNOWN
Location: /usr/lib/python3.6/site-packages
Requires: 
Required-by:

hinchliff · 2021-01-27T18:57:37Z

I see this in the verbose logs

    changing mode of /usr/local/bin/ansible to 755

And that binary does exist. Just not on my path, i guess?

# /usr/local/bin/ansible --version
ansible 2.10.5
[...]

hmmm...

webknjaz · 2021-01-27T18:59:52Z

And that binary does exist. Just not on my path, i guess?

Yes, depending on how the software is built (like CPython), the paths may be different in different OSs, or even if there are several different interpreters on the same system.

hinchliff · 2021-01-27T19:04:49Z

Yeah, searching around it looks like the behavior of /usr/local/bin being on the PATH or not is a little odd with RHEL, maybe others too. Part of the issue seems to be with using sudo / sudo -i, which will lose / remove that path from PATH.

This all used to work with ansible 2.9.x (2.9.6 specifically) that we have been using, where the ansible "binaries" were installed to /usr/bin, so it seems like the default installation path changed from 2.9 -> 2.10 ?

hinchliff · 2021-01-27T19:21:59Z

I suspect the easiest thing in my case might be to pass in extra options to the pip command, like --install-option="--install-scripts=/usr/bin"

webknjaz · 2021-01-27T19:24:06Z

I don't think our distribution package specifies where to install scripts. It should be up to pip/CPython.

jborean93 · 2021-01-28T19:11:05Z

Is This A Bug?

Hi!

Thanks very much for your submission to Ansible. It sincerely means a lot to us.

We're not sure this is a bug, and we don't mean for this to be confrontational. Let's explain what we're thinking:

This does not seem to be related to our packaging at all but to a custom setup

As such, we're going to close this ticket. However, we're open to being corrected, should you wish to discuss. You can stop by one of our two mailing lists
to talk about this and we might be persuaded otherwise.

https://groups.google.com/forum/#!forum/ansible-project - for user questions, tips, and tricks
https://groups.google.com/forum/#!forum/ansible-devel - for strategy, future planning, and questions about writing code

Comments on closed tickets aren't something we monitor, so if you do disagree with this, a mailing list thread is probably appropriate.

Thank you once again for this and your interest in Ansible!

ansibot added affects_2.11 bug This issue/PR relates to a bug. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. python3 support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Jan 27, 2021

jborean93 closed this as completed Jan 28, 2021

jborean93 removed the needs_triage Needs a first human triage before being processed. label Jan 28, 2021

ansible locked and limited conversation to collaborators Feb 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New install of 2.10.5 via Pip (3) results in no binaries #73390

New install of 2.10.5 via Pip (3) results in no binaries #73390

hinchliff commented Jan 27, 2021 •

edited

ansibot commented Jan 27, 2021

sivel commented Jan 27, 2021

webknjaz commented Jan 27, 2021

hinchliff commented Jan 27, 2021

hinchliff commented Jan 27, 2021

webknjaz commented Jan 27, 2021

hinchliff commented Jan 27, 2021

hinchliff commented Jan 27, 2021

webknjaz commented Jan 27, 2021

jborean93 commented Jan 28, 2021

New install of 2.10.5 via Pip (3) results in no binaries #73390

New install of 2.10.5 via Pip (3) results in no binaries #73390

Comments

hinchliff commented Jan 27, 2021 • edited

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ansibot commented Jan 27, 2021

sivel commented Jan 27, 2021

webknjaz commented Jan 27, 2021

hinchliff commented Jan 27, 2021

hinchliff commented Jan 27, 2021

webknjaz commented Jan 27, 2021

hinchliff commented Jan 27, 2021

hinchliff commented Jan 27, 2021

webknjaz commented Jan 27, 2021

jborean93 commented Jan 28, 2021

Is This A Bug?

hinchliff commented Jan 27, 2021 •

edited