pip install explicit requirements fails (but implicit succeeds)

[MPV]

• Pip version: 10.0.0
• Python version: 2.7.12
• Operating system: Ubuntu 16.04

Description:

What I'm trying to get done:
Let Ansible install some pip packages.

What has happened:

1. I run Ansible as part of PXE boot of a new machine.
2. Ansible pre-installs cryptography version 1.2.3 as part of its setup.
3. Ansible asks pip to install PyOpenSSL and cryptography.
4. Pip breaks itself (by allowing mismatching versions to be installed).

What went wrong:
PyOpenSSL requires cryptography version 2.2.1 (as per pyca/pyopenssl#559 (comment)) but pip fails to upgrade it from 1.2.3 to 2.2.1 when asked

What I had expected:
One of the following:

1. pip install exiting with non-zero return code when it's not able to solve the dependecies.
2. pip install succeeding in installing both PyOpenSSL and cryptography==2.2.1 (in this example).

What I've run:

1. Have an earlier version of (for example) cryptography installed (as in my case, setup of Ansible has installed):

$ pip2 install cryptography==1.2.3
Collecting cryptography==1.2.3
Requirement already satisfied: cffi>=1.4.1 in /usr/local/lib/python2.7/dist-packages (from cryptography==1.2.3) (1.11.5)
Collecting six>=1.4.1 (from cryptography==1.2.3)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python2.7/dist-packages (from cryptography==1.2.3) (2.6)
Requirement already satisfied: enum34 in /usr/lib/python2.7/dist-packages (from cryptography==1.2.3) (1.1.2)
Collecting pyasn1>=0.1.8 (from cryptography==1.2.3)
  Using cached https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=1.0 in /usr/lib/python2.7/dist-packages (from cryptography==1.2.3) (20.7.0)
Requirement already satisfied: ipaddress in /usr/lib/python2.7/dist-packages (from cryptography==1.2.3) (1.0.16)
Requirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.4.1->cryptography==1.2.3) (2.18)
Installing collected packages: six, pyasn1, cryptography
Successfully installed cryptography-1.2.3 pyasn1-0.4.2 six-1.11.0

2. Install some more packages:

...in this case PyOpenSSL and cryptography — where PyOpenSSL requires cryptography==2.2.1:

$ pip2 install PyOpenSSL cryptography
Collecting PyOpenSSL
  Using cached https://files.pythonhosted.org/packages/79/db/7c0cfe4aa8341a5fab4638952520d8db6ab85ff84505e12c00ea311c3516/pyOpenSSL-17.5.0-py2.py3-none-any.whl
Requirement already satisfied: cryptography in /usr/local/lib/python2.7/dist-packages (1.2.3)
Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python2.7/dist-packages (from PyOpenSSL) (1.11.0)
Requirement already satisfied: cffi>=1.4.1 in /usr/local/lib/python2.7/dist-packages (from cryptography) (1.11.5)
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python2.7/dist-packages (from cryptography) (2.6)
Requirement already satisfied: enum34 in /usr/lib/python2.7/dist-packages (from cryptography) (1.1.2)
Requirement already satisfied: pyasn1>=0.1.8 in /usr/local/lib/python2.7/dist-packages (from cryptography) (0.4.2)
Requirement already satisfied: setuptools>=1.0 in /usr/lib/python2.7/dist-packages (from cryptography) (20.7.0)
Requirement already satisfied: ipaddress in /usr/lib/python2.7/dist-packages (from cryptography) (1.0.16)
Requirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.4.1->cryptography) (2.18)
pyopenssl 17.5.0 has requirement cryptography>=2.1.4, but you'll have cryptography 1.2.3 which is incompatible.
Installing collected packages: PyOpenSSL
Successfully installed PyOpenSSL-17.5.0

Note the line:

pyopenssl 17.5.0 has requirement cryptography>=2.1.4, but you'll have cryptography 1.2.3 which is incompatible.

...but the return code isn't zero?

$ echo $?
0

Had I received a non-zero return code, my Ansible script would've stopped and I could've understood that something in my intended setup was broken. Now instead:

3. Suddenly pip2 is broken:

$ pip2
Traceback (most recent call last):
  File "/usr/local/bin/pip2", line 7, in <module>
    from pip._internal import main
  File "/usr/local/lib/python2.7/dist-packages/pip/_internal/__init__.py", line 42, in <module>
    from pip._internal import cmdoptions
  File "/usr/local/lib/python2.7/dist-packages/pip/_internal/cmdoptions.py", line 16, in <module>
    from pip._internal.index import (
  File "/usr/local/lib/python2.7/dist-packages/pip/_internal/index.py", line 15, in <module>
    from pip._vendor import html5lib, requests, six
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/__init__.py", line 86, in <module>
    from pip._vendor.urllib3.contrib import pyopenssl
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 204, in <module>
    if _lib.Cryptography_HAS_SSL_ST:
AttributeError: 'module' object has no attribute 'Cryptography_HAS_SSL_ST'

In comparison...

Now let's compare the above scenario with this:

I just install PyOpenSSL (which requires cryptography) but not cryptography explicitly:

1. Starting from scratch, as before, pre-install earlier version of cryptography:

$ pip2 install cryptography==1.2.3
Collecting cryptography==1.2.3
Requirement already satisfied: cffi>=1.4.1 in /usr/local/lib/python2.7/dist-packages (from cryptography==1.2.3) (1.11.5)
Collecting six>=1.4.1 (from cryptography==1.2.3)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python2.7/dist-packages (from cryptography==1.2.3) (2.6)
Requirement already satisfied: enum34 in /usr/lib/python2.7/dist-packages (from cryptography==1.2.3) (1.1.2)
Collecting pyasn1>=0.1.8 (from cryptography==1.2.3)
  Using cached https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=1.0 in /usr/lib/python2.7/dist-packages (from cryptography==1.2.3) (20.7.0)
Requirement already satisfied: ipaddress in /usr/lib/python2.7/dist-packages (from cryptography==1.2.3) (1.0.16)
Requirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.4.1->cryptography==1.2.3) (2.18)
Installing collected packages: six, pyasn1, cryptography
Successfully installed cryptography-1.2.3 pyasn1-0.4.2 six-1.11.0

2. Ask pip to install only PyOpenSSL this time:

$ pip2 install PyOpenSSL
Collecting PyOpenSSL
  Using cached https://files.pythonhosted.org/packages/79/db/7c0cfe4aa8341a5fab4638952520d8db6ab85ff84505e12c00ea311c3516/pyOpenSSL-17.5.0-py2.py3-none-any.whl
Collecting cryptography>=2.1.4 (from PyOpenSSL)
  Using cached https://files.pythonhosted.org/packages/dd/c2/3a5bfefb25690725824ade71e6b65449f0a9f4b29702cce10560f786ebf6/cryptography-2.2.2-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python2.7/dist-packages (from PyOpenSSL) (1.11.0)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != "PyPy" in /usr/local/lib/python2.7/dist-packages (from cryptography>=2.1.4->PyOpenSSL) (1.11.5)
Requirement already satisfied: enum34; python_version < "3" in /usr/lib/python2.7/dist-packages (from cryptography>=2.1.4->PyOpenSSL) (1.1.2)
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/local/lib/python2.7/dist-packages (from cryptography>=2.1.4->PyOpenSSL) (0.24.0)
Requirement already satisfied: idna>=2.1 in /usr/local/lib/python2.7/dist-packages (from cryptography>=2.1.4->PyOpenSSL) (2.6)
Requirement already satisfied: ipaddress; python_version < "3" in /usr/lib/python2.7/dist-packages (from cryptography>=2.1.4->PyOpenSSL) (1.0.16)
Requirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=2.1.4->PyOpenSSL) (2.18)
Installing collected packages: cryptography, PyOpenSSL
  Found existing installation: cryptography 1.2.3
    Uninstalling cryptography-1.2.3:
      Successfully uninstalled cryptography-1.2.3
Successfully installed PyOpenSSL-17.5.0 cryptography-2.2.2

Note how this time pip succeeds in upgrading cryptography from 1.2.3 to 2.2.2. 🎉

3. And how this time pip isn't broken:

$ pip2

Usage:
  pip2 <command> [options]
[...]

---

Note:

• I didn't have this problem when I ran the same Ansible setup and scripts 2 weeks ago.
• I've yet to test (and confirm) whether this is related to pip version 10.0.0.
• I do know that the machines that this has worked on before are running pip version 9.0.3.

---

I might just be using pip in the wrong way here.
Or could there be a bug in pip related to this? 😊

[MPV]

For reference, this is probably related to #988 ?

[pfmoore]

For reference, this is probably related to #988 ?

Essentially yes, I believe it is. I don't know why "Suddenly pip2 is broken" happened, but that's in relation to Unix stuff I have limited knowledge of. The other symptoms are basically as expected due to the fact that we don't have a resolver, and collect requirements of a "first come, first served" basis. So if you mention cryptography on the command line, that's what you're asking for and it's installed - the implicit specific version requirement is ignored. But if you don't mention cryptography on the command line, the first mention of it is the implicit cryptography>=2.1.4, so that's what you get.

[MPV]

In the scenario above, pip breaks since it starts using code from PyOpenSSL after it's been installed, which in turn uses code from cryptography. But since PyOpenSSL is built using cryptography==2.2.1, it fails when given cryptography==1.2.3 instead, as they're incompatible.

Looking at some other servers here that were provisioned some days/weeks ago (then with pip version 9.0.x), they are correctly running (= were upgraded to) cryptography==2.2.1.

Hence why I'm suspecting that pip version 10.0.0 might've introduced this loophole where it's now allowing inconsistent packages to be installed, compared to before where it might've been better at resolving this dependency challenge.

Also, it sounds great that #988 is getting some traction again. 😊

[pradyunsg]

Thanks for filing this @MPV!

You'd likely have some success using --upgrade-strategy=eager but really, this is #988 and I'll close this issue as a duplicate of that.

If you're feeling awesome, please file an issue over at https://github.com/pradyunsg/zazo for this. :)

[pradyunsg]

This is an interesting case of #988. Closing as a duplicate. (opening an issue on zazo to add a test for this scenario)

[pradyunsg]

Wait. Not closing... You should be run pip check to cause an exit code of 1 (it does the same checking as the one that printed that error).

Feel free to close this issue if that addresses your concern.

[chrahunt]

@MPV, I'll close this since It's been some time since the advice above has been given, but please let us know if you have any other concerns.