pip 6.0 broke "pip install --pre --upgrade --find-links" #2256

rgommers · 2014-12-22T21:55:22Z

TravisCI for numpy started failing after the pip 6.0 release: numpy/numpy#5386

The reason is that pip install --pre --upgrade --find-links dist numpy, where dist contains a wheel built from current master, starts to install numpy 1.9.1 from PyPi instead of that wheel. I have replicated the issue locally in a virtualenv:

$ pip install --pre --upgrade --find-links ../../numpy/dist numpy  # works with 1.5.6
$ pip uninstall numpy
$ pip install --upgrade pip   # installed pip 6.0
$ pip install --pre --upgrade --find-links ../../numpy/dist numpy  # starts downloading from PyPi

I can bisect the issue in pip if needed, but I'm hoping someone can tell me straight away what changed.

The text was updated successfully, but these errors were encountered:

dstufft · 2014-12-22T21:56:23Z

Can you post the final pip install with a -vvv added?

rgommers · 2014-12-22T21:59:20Z

Sure:

$ pip install -vvv --pre --upgrade --find-links ../../numpy/dist numpy
Collecting numpy
  Getting page https://pypi.python.org/simple/numpy/
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/numpy/ HTTP/1.1" 200 2596
  URLs to search for versions for numpy:
  * https://pypi.python.org/simple/numpy/
  Skipping link ../../numpy/dist (from -f); not a file
  Getting page https://pypi.python.org/simple/numpy/
  Analyzing links from page https://pypi.python.org/simple/numpy/

Didn't know about -vvv, looks useful. The "skipping link" is telling. I don't understand why the message is not a file, according to the docs it can be a path.

dstufft · 2014-12-22T22:00:21Z

Does it work if you use file:///full/path/to/numpy/dist instead?

rgommers · 2014-12-22T22:02:19Z

No, same message.

And just for completeness:

$ ls ../../numpy/dist
numpy-1.10.0.dev_243ab56-cp27-none-linux_i686.whl

dstufft · 2014-12-22T22:03:57Z

Hmm. That is very strange. I can't think of what would have changed between then and now. Let me see if I can reproduce locally.

dstufft · 2014-12-22T22:05:02Z

Oh!

I did just notice something. What happens if you change numpy-1.10.0.dev_243ab56-cp27-none-linux_i686.whl to numpy-1.10.0.dev+243ab56-cp27-none-linux_i686.whl.

rgommers · 2014-12-22T22:08:51Z

No change when changing _ to +. I hope a plus is not recommended in a versioning scheme now?

Edit: I see your suggestion also had no none. I copied the filename exactly now, still no change

dstufft · 2014-12-22T22:10:20Z

A + has a specific meaning in version schemes now (PEP 440 local versions).

Trying to repro this now.

dstufft · 2014-12-22T22:14:39Z

Hmm. I get this:

Collecting numpy
  URLs to search for versions for numpy:
  Skipping link ./dist (from -f); not a file
  Skipping file:///Users/dstufft/.virtualenvs/tmp-f87184b6fa6f379/dist/numpy-1.10.0.dev_243ab56-cp27-none-linux_i686.whl because it is not compatible with this Python

Wonder if this might be something specific to Linux.

rgommers · 2014-12-22T22:14:40Z

OK clear. I'm signing off for tonight. Thanks for looking into this.

argriffing · 2014-12-22T22:26:06Z

Looks superficially similar to #1029.

dstufft · 2014-12-22T22:28:11Z

I was able to reproduce the error, but changing the dev-<stuff> to dev+<stuff> fixed it. I believe this error is caused because the version isn't a valid PEP 440 version when you include the githash inside the version like that.

argriffing · 2014-12-22T23:45:42Z

With the new pip release, I think the latest development versions of numpy and scipy are considered older than any released version, whereas with the earlier pip version I think this was not the case. Numpy and scipy do not have valid PEP 440 version names.

"""
Check the relative ordering of package versions.

"""
from __future__ import print_function, division

import sys

import numpy
import scipy

import pip
import pkg_resources
import setuptools

def main():
    print('python version:', sys.version)
    print('numpy version:', numpy.__version__)
    print('scipy version:', scipy.__version__)
    print('pip version:', pip.__version__)
    print('setuptools version:', setuptools.__version__)
    print()

    version_names = ('1.14.2', '1.16.2', '1.17.0.dev-deadbeef')

    # I think this is the setuptools order.
    print('version order provided by setuptools pkg_resources:')
    version_objects = [
            pkg_resources.parse_version(x) for x in version_names]
    print(sorted(version_objects))
    print()

    # This should be the pip order.
    print('version order provided by the "vendored" setuptools pkg_resources '
            'provided by pip:')
    try:
        version_objects = [
                pip._vendor.packaging.version.parse(x) for x in version_names]
    except AttributeError:
        version_objects = [
                pip._vendor.pkg_resources.parse_version(x) for x in version_names]
    print(sorted(version_objects))
    print()

main()

1.5.6 output:

python version: 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2]
numpy version: 1.10.0.dev-243ab56
scipy version: 0.16.0.dev-619698b
pip version: 1.5.6
setuptools version: 8.2.1

version order provided by setuptools pkg_resources:
[<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>]

version order provided by the "vendored" setuptools pkg_resources provided by pip:
[('00000001', '00000014', '00000002', '*final'), ('00000001', '00000016', '00000002', '*final'), ('00000001', '00000017', '*@', '*deadbeef', '*final')]

6.0 output:

python version: 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2]
numpy version: 1.10.0.dev-243ab56
scipy version: 0.16.0.dev-619698b
pip version: 6.0
setuptools version: 8.2.1

version order provided by setuptools pkg_resources:
[<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>]

version order provided by the "vendored" setuptools pkg_resources provided by pip:
[<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>]

argriffing · 2014-12-22T23:53:12Z

... and when I change the version names to

version_names = (
            '1.14.2',
            '1.16.2',
            '1.17.0.dev-deadbeef',
            '1.17.0.dev+deadbeef',
            )

I get the order [<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>, <Version('1.17.0.dev0+deadbeef')>]. So I'm almost certain this is the problem.

rgommers · 2014-12-23T07:34:03Z

Still doesn't fix it for me. It looks like version ordering is part of the problem, but there's something else. @dstufft question: according to pip install help the --pre option does this:

Include pre-release and development versions. By default, pip only
finds stable versions.

that implies that it does not include local versions. Is that intended, or is the doc wrong? If intended, how do I force install of local versions from a given path?

rgommers · 2014-12-23T07:35:56Z

Note to self for easier debugging, also use --no-index (it will fail instead of picking up wrong version from PyPi):

$ pip install -vvv --pre --upgrade --find-links --no-index /home/rgommers/Code/numpy/dist/ numpy
Directory '/home/rgommers/Code/numpy/dist/' is not installable. File 'setup.py' not found.
Exception information:
...

Note that the error message is incomplete, doesn't mention wheels.

dstufft · 2014-12-23T09:00:18Z

PEP 440 local versions count as stable versions unless they are included as part of a pre-release version.

I wonder, does it work if you upgrade setuptools first? I wonder if the difference is I'm running a new setuptools.

dstufft · 2014-12-23T10:55:19Z

Well your latest command fails because it's doing --find-links=--no-index and then telling it to install /home/rgommers/Code/numpy/dist/. You want:

$ pip install -vvv --pre --upgrade --no-index --find-links /home/rgommers/Code/numpy/dist/ numpy

rgommers · 2014-12-23T22:33:57Z

@dstufft indeed after upgrading setuptools (version setuptools-8.2.1-py2.py3-none-any.whl) the old version number is not recognized but the PEP 440 one is.

rgommers · 2014-12-23T22:35:35Z

I see you already updated the release notes for this backwards compat break, so I guess nothing more to do here. Closing. Thanks for the help in figuring this out!

dstufft · 2014-12-23T22:43:54Z

Note: That we're considering PEP 440 as a sort of provisional thing, so we're not opposed to making changes if it makes sense in a broader ecosystem sense. But if you're able to get things working with how it is now that's good too.

rgommers · 2014-12-23T22:48:59Z

Good to know. As far as build/install toolchain issues go, this one is fairly benign. It's a one-character fix plus telling people who run into it to upgrade setuptools - should be no problem.

argriffing · 2015-02-10T17:17:01Z

It seems that something changed in pip or setuptools that has required another change in the numpy/scipy version names
numpy/numpy#5556
scipy/scipy#4512

This was referenced Dec 23, 2014

scipy and numpy version names do not follow pep 440 scipy/scipy#4301

Closed

BLD: PEP 440 numpy/numpy#5387

Merged

maniteja123 mentioned this issue Dec 23, 2014

BUG: Make RandomState.seed, RandomState.randint and RandomState.shuffle threadsafe numpy/numpy#5390

Closed

rgommers closed this as completed Dec 23, 2014

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 5, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip 6.0 broke "pip install --pre --upgrade --find-links" #2256

pip 6.0 broke "pip install --pre --upgrade --find-links" #2256

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

argriffing commented Dec 22, 2014

dstufft commented Dec 22, 2014

argriffing commented Dec 22, 2014

argriffing commented Dec 22, 2014

rgommers commented Dec 23, 2014

rgommers commented Dec 23, 2014

dstufft commented Dec 23, 2014

dstufft commented Dec 23, 2014

rgommers commented Dec 23, 2014

rgommers commented Dec 23, 2014

dstufft commented Dec 23, 2014

rgommers commented Dec 23, 2014

argriffing commented Feb 10, 2015

pip 6.0 broke "pip install --pre --upgrade --find-links" #2256

pip 6.0 broke "pip install --pre --upgrade --find-links" #2256

Comments

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

dstufft commented Dec 22, 2014

dstufft commented Dec 22, 2014

rgommers commented Dec 22, 2014

argriffing commented Dec 22, 2014

dstufft commented Dec 22, 2014

argriffing commented Dec 22, 2014

argriffing commented Dec 22, 2014

rgommers commented Dec 23, 2014

rgommers commented Dec 23, 2014

dstufft commented Dec 23, 2014

dstufft commented Dec 23, 2014

rgommers commented Dec 23, 2014

rgommers commented Dec 23, 2014

dstufft commented Dec 23, 2014

rgommers commented Dec 23, 2014

argriffing commented Feb 10, 2015