Pip chooses different version for setup_requires than what was specified in requirements.txt

[dwt]

Environment

• pip version: 19.1.1
• Python version: 2.7.5
• OS: CentOS Linux release 7.6.1810 (Core)

Description
I am trying to reproduce a package installation using a requirements.txt to get the same versions as before.
Doing this I noticed that the setup_requires of a project in my install switched to a newer version after the fact. This happened even though the correct version of the dependency was specified in the requirements.txt file.

To be more concrete, the requirements file contains both

Bottleneck==1.2.1
numpy==1.14.6

However, when building bottleneck a newer numpy version was used as the setup_requires dependency for bottleneck (which is bad, as it definitely needs to be compiled against the same version as it is run against later).

Expected behavior
Those same versions of packages should be installed. To be more concrete: setup_requires should respect the package versions specified in the requirements.txt

How to Reproduce

1. When trying to install this requirements.txt

Bottleneck==1.2.1
numpy==1.14.6

1.1. I was also able to trigger this bug with this command line pip install numpy==1.14.6 Bottleneck==1.2.1
2. In a virtualenv with python2
3. The installation fails because bottleneck has a setup_requires dependency on numpy and it seems that pip chooses to fetch the newest version of numpy (1.17rc1) as the build dependency. This is of course wrong, as I definitely want the version I specified to be used as the setup_requires dependency.

Output

# virtualenv creation skipped
$ pip install numpy==1.14.6 Bottleneck==1.2.1
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting numpy==1.14.6
  Downloading https://files.pythonhosted.org/packages/4e/5b/1077ec0ebfa06f42057e8315bc8e05f5978b6fd0f582879f35f4d62ff124/numpy-1.14.6-cp27-cp27mu-manylinux1_x86_64.whl (13.8MB)
     |████████████████████████████████| 13.8MB 5.4MB/s 
Collecting Bottleneck==1.2.1
  Using cached https://files.pythonhosted.org/packages/05/ae/cedf5323f398ab4e4ff92d6c431a3e1c6a186f9b41ab3e8258dff786a290/Bottleneck-1.2.1.tar.gz
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-tqQ5sA/Bottleneck/setup.py", line 110, in <module>
        setup(**metadata)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/dist.py", line 717, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/dist.py", line 784, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/home/someuser/virtualenv/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-xD3a2T/numpy-1.17.0rc1/setup.py", line 31, in <module>
        from bottleneck.src.template import make_c_files
    RuntimeError: Python version >= 3.5 required.
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-tqQ5sA/Bottleneck/

Workaround: I did workaround this by installing both packages individually after each other, thus seeding my caches with a built version of the wheel of bottleneck, after which the seems to have worked fine.

[chrahunt]

If I'm understanding correctly, this is what is failing:

1. Create a virtualenv with python2
2. Execute pip install numpy==1.14.6 Bottleneck==1.2.1
3. Notice that the installation operation fails

When the source distribution of Bottleneck is downloaded we attempt to build a wheel. This happens in an isolated environment independent of the other requested package installations (numpy==1.14.6). Because Bottleneck does not specify any version of numpy in its setup_requires, setuptools (via easy_install/distutils) attempts to download the latest numpy in order to satisfy the requirement.

Possible fixes:

1. Specify an explicit version of numpy in the setup_requires of Bottleneck's setup.py that is compatible with the Python versions being used
2. Install Bottleneck with --no-build-isolation, in which case the wheel build will take place in the same virtual environment the package is attempting to be installed into

I think pip is behaving as-expected here.

[dwt]

I would argue that pip may be behaving as it always did, but as a user, when I specify that I want to install numpy==1.14.6 Bottleneck==1.2.1 - wouldn't you kind of expect that Bottleneck is actually built against the version of numpy you are just installing? Keep in mind that that the binary portion of Bottleneck is (most probably) useless / incompatible if it isn't compiled against the same version of numpy that is later used.

From my point of view the problem seems to be that the setup_requires are not actually part of the dependency graph that pip builds and resolves -> thus leading to this problem.

Also Bottleneck cannot state a more specific version of numpy in its dependencies, as it is compatible with multiple versions of numpy just fine. The problem is that I need a specific version of numpy (which bottleneck is compatible with) but which therefore cannot be stated in it's dependencies.

The more general problem is that there is no way to encode this information or the workaround you mentioned in a requirements.txt file (as far as I know) in a way that allows me to replicate a virtualenv by doing pip freeze in it and then trying to replicate that install in another virtualenv.

[pradyunsg]

pip does not handle setup requires metadata directly -- it's handled by setuptools (hence easy_install).

This information is already not available to pip (you need to execute setup.py to know what's needed to execute setup.py in this scheme) and PEP 518 was written to move away from the specification of build requirements in setup.py to a dedicated file for the job.

[dwt]

@pradyunsg bottleneck does specify this at https://github.com/kwgoodman/bottleneck/blob/master/pyproject.toml so this information should be available to pip.

I just realised, that this change is not released yet. :-/

[pradyunsg]

Awesome! That'll mean the next release of bottleneck will fix this issue for users. :)

I'll go ahead and close this since I don't think there's any actionable next steps here for pip's developers here.