pip 6.0 installs in reverse order, as compared to 1.5 #2260

bukzor · 2014-12-23T03:57:23Z

I have a use case where two forked packages need to be installed in a particular order or else they throw an error. pip1.5 would install requirements in the order they were given on the commandline, or in a requirements file. pip6 seems to install in exactly the opposite order (last to first on commandline or requirements file).

Under pip1.5, note that coverage is installed before cov-core: (Sorry the example is a bit contrived, but this really did bite me.)

$ COV_CORE_SOURCE= COV_CORE_DATA_FILE=cov COV_CORE_CONFIG=.coveragerc pip install hg+https://bitbucket.org/bukzor/coverage.py@__main__-support#egg=coverage git+git://github.com/bukzor/cov-core.git@master#egg=cov-core

Downloading/unpacking coverage from hg+https://bitbucket.org/bukzor/coverage.py@__main__-support
  Cloning hg https://bitbucket.org/bukzor/coverage.py (to revision __main__-support) to /home/buck/venv/fresh/build/coverage
  Running setup.py (path:/home/buck/venv/fresh/build/coverage/setup.py) egg_info for package coverage

    warning: no files found matching 'coverage.egg-info/*.*'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Downloading/unpacking cov-core from git+git://github.com/bukzor/cov-core.git@master
  Cloning git://github.com/bukzor/cov-core.git (to master) to /home/buck/venv/fresh/build/cov-core
  Running setup.py (path:/home/buck/venv/fresh/build/cov-core/setup.py) egg_info for package cov-core

Installing collected packages: coverage, cov-core
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c coverage/tracer.c -o build/temp.linux-x86_64-2.7/coverage/tracer.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/coverage/tracer.o -o build/lib.linux-x86_64-2.7/coverage/tracer.so

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    Installing coverage2 script to /home/buck/venv/fresh/bin
    Installing coverage-2.7 script to /home/buck/venv/fresh/bin
    Installing coverage script to /home/buck/venv/fresh/bin
  Running setup.py install for cov-core

Successfully installed coverage cov-core
Cleaning up...

Same command under pip6 installs the two packages in opposite order, and crashes:

$ COV_CORE_SOURCE= COV_CORE_DATA_FILE=cov COV_CORE_CONFIG=.coveragerc pip install hg+https://bitbucket.org/bukzor/coverage.py@__main__-support#egg=coverage git+git://github.com/bukzor/cov-core.git@master#egg=cov-core

Collecting coverage from hg+https://bitbucket.org/bukzor/coverage.py@__main__-support#egg=coverage
  Cloning hg https://bitbucket.org/bukzor/coverage.py (to revision __main__-support) to /tmp/pip-build-Q0YZ23/coverage
Collecting cov-core from git+git://github.com/bukzor/cov-core.git@master#egg=cov-core
  Cloning git://github.com/bukzor/cov-core.git (to master) to /tmp/pip-build-Q0YZ23/cov-core
Installing collected packages: cov-core, coverage
  Running setup.py install for cov-core
  Running setup.py install for coverage
    Traceback (most recent call last):
      File "/home/buck/venv/fresh/lib/python2.7/site.py", line 703, in <module>
        main()
      File "/home/buck/venv/fresh/lib/python2.7/site.py", line 683, in main
        paths_in_sys = addsitepackages(paths_in_sys)
      File "/home/buck/venv/fresh/lib/python2.7/site.py", line 282, in addsitepackages
        addsitedir(sitedir, known_paths)
      File "/home/buck/venv/fresh/lib/python2.7/site.py", line 204, in addsitedir
        addpackage(sitedir, name, known_paths)
      File "/home/buck/venv/fresh/lib/python2.7/site.py", line 173, in addpackage
        exec(line)
      File "<string>", line 1, in <module>
      File "/home/buck/venv/fresh/local/lib/python2.7/site-packages/cov_core_init.py", line 20, in <module>
        import cov_core  # noqa: register multiprocessing handler
      File "/home/buck/venv/fresh/local/lib/python2.7/site-packages/cov_core.py", line 5, in <module>
        import coverage
    ImportError: No module named coverage

If I reverse the arguments under pip1.6, the packages are listed in that order, but again installed in reversed (double-revered, ie correct) order:

$ COV_CORE_SOURCE= COV_CORE_DATA_FILE=cov COV_CORE_CONFIG=.coveragerc pip install git+git://github.com/bukzor/cov-core.git@master#egg=cov-core  hg+https://bitbucket.org/bukzor/coverage.py@__main__-support#egg=coverage

Collecting cov-core from git+git://github.com/bukzor/cov-core.git@master#egg=cov-core
  Cloning git://github.com/bukzor/cov-core.git (to master) to /tmp/pip-build-qiidwD/cov-core
Collecting coverage from hg+https://bitbucket.org/bukzor/coverage.py@__main__-support#egg=coverage
  Cloning hg https://bitbucket.org/bukzor/coverage.py (to revision __main__-support) to /tmp/pip-build-qiidwD/coverage
Installing collected packages: coverage, cov-core
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -
c coverage/tracer.c -o build/temp.linux-x86_64-2.7/coverage/tracer.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DN
DEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format
-security build/temp.linux-x86_64-2.7/coverage/tracer.o -o build/lib.linux-x86_64-2.7/coverage/tracer.so
    Installing coverage2 script to /home/buck/venv/fresh/bin
    Installing coverage-2.7 script to /home/buck/venv/fresh/bin
    Installing coverage script to /home/buck/venv/fresh/bin
  Running setup.py install for cov-core
Successfully installed cov-core coverage

In summary, I can craft a command that will work under 1.5 or 6, but not both, in the currrent state of things.

The text was updated successfully, but these errors were encountered:

ionelmc · 2014-12-23T08:12:49Z

Note that cov-core is very clever - have you seen the pth hackery in it's setup.py ? That could break all your imports at any time. 😨

dstufft · 2014-12-23T08:56:22Z

We purposely switched the order. The new order installs dependent packages first which means that you won't get a top level dependency installed and then have one of it's dependencies fail to install and leave your system in a state where you have to uninstall and then reinstall the top level.

schlamar · 2014-12-23T09:02:05Z

The new order installs dependent packages first

Apparently not, cov-core depends on coverage. So there is something wrong here...

dstufft · 2014-12-23T10:26:48Z

Oh, actually I didn't read the command very closely. The reason the order is reversed is because you specified them both on the CLI. That tells pip that they are both top level dependencies.

The behavior is correct if you just install cov-core:

1.5.6

$ pip install cov-core
Downloading/unpacking cov-core
  Downloading cov-core-1.15.0.tar.gz
  Running setup.py (path:/Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/build/cov-core/setup.py) egg_info for package cov-core

Downloading/unpacking coverage>=3.6 (from cov-core)
  Downloading coverage-3.7.1.tar.gz (284kB): 284kB downloaded
  Running setup.py (path:/Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/build/coverage/setup.py) egg_info for package coverage

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Installing collected packages: cov-core, coverage
  Running setup.py install for cov-core


    Wrote pth file for subprocess measurement to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/lib/python3.4/site-packages/init_cov_core.pth
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/dstufft/.pyenv/versions/3.4.2/include/python3.4m -c coverage/tracer.c -o build/temp.macosx-10.10-x86_64-3.4/coverage/tracer.o
    gcc -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/dstufft/.pyenv/versions/3.4.2/lib build/temp.macosx-10.10-x86_64-3.4/coverage/tracer.o -o build/lib.macosx-10.10-x86_64-3.4/coverage/tracer.so

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    Installing coverage script to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/bin
    Installing coverage-3.4 script to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/bin
    Installing coverage3 script to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/bin
Successfully installed cov-core coverage
Cleaning up...

6.0.1

$ pip install cov-core
Collecting cov-core
  Using cached cov-core-1.15.0.tar.gz
Collecting coverage>=3.6 (from cov-core)
  Using cached coverage-3.7.1.tar.gz
Installing collected packages: coverage, cov-core
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/dstufft/.pyenv/versions/3.4.2/include/python3.4m -c coverage/tracer.c -o build/temp.macosx-10.10-x86_64-3.4/coverage/tracer.o
    gcc -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/dstufft/.pyenv/versions/3.4.2/lib build/temp.macosx-10.10-x86_64-3.4/coverage/tracer.o -o build/lib.macosx-10.10-x86_64-3.4/coverage/tracer.so
    Installing coverage3 script to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/bin
    Installing coverage script to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/bin
    Installing coverage-3.4 script to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/bin
  Running setup.py install for cov-core
    Wrote pth file for subprocess measurement to /Users/dstufft/.virtualenvs/tmp-9c5dd7548c94620/lib/python3.4/site-packages/init_cov_core.pth
Successfully installed cov-core coverage

You can tell from this that previously pip would install cov-core first, and then coverage, and now it installs coverage first and then cov-core.

dstufft · 2014-12-23T10:39:10Z

I'm going to close this, as the new behavior is what we wanted. It means that when running pip install cov-core it will isntall coverage first. Previously it would install cov-core first, and then if coverage install failed you'd end up with cov-core installed without coverage. This was particularly troublesome because pip install cov-core again wouldn't fix it since it was already installed.

bukzor · 2014-12-23T17:06:43Z

In summary, If I specify two dependant packages in the order that pip would have installed
them, it instead installs them in the opposite order, both as compared to
the arguments, and as compared to the dependency topology, putting me in
exactly the situation you were trying to factor out with this change.

I don't see that as a reasonable state of things.

Also please note that I do need to specify both packages, since I'm
currently using forks of both.

I propose two patches:

Pip should notice when "top level" dependencies are actually dependent.
There are multiple reasons why users would specify both a package and its
dependency in a requirements list.
Independent top level dependencies should have their ordering preserved. It seems to me (please look closely at demos) that pip6 currently preserves and reverses the ordering of top level dependencies.

--phone is hard.
On Dec 23, 2014 2:39 AM, "Donald Stufft" notifications@github.com wrote:

I'm going to close this, as the new behavior is what we wanted. It means
that when running pip install cov-core it will isntall coverage first.
Previously it would install cov-core first, and then if coverage install
failed you'd end up with cov-core installed without coverage. This was
particularly troublesome because pip install cov-core again wouldn't fix
it since it was already installed.

—
Reply to this email directly or view it on GitHub
#2260 (comment).

asottile · 2014-12-23T17:48:56Z

+1

piotr-dobrogost · 2015-03-04T14:33:50Z

There is no reason to install packages given as an input to pip in the reverse order. This is not natural and clearly regression from previous, natural behavior. Please note that this is orthogonal to the change which introduced installation of dependencies first, which is positive. This bug should be reopened and fixed.

@bukzor

Pip should notice when "top level" dependencies are actually dependent.

I wouldn't involve this here. Firstly, it's not required in order to fix the regression which is installation of input packages in reverse order. Secondly, this is territory of dependency resolution which pip can't do - see #988

On its own, this is a fix for a regression vs master, as it passes tests. It is needed for topological handling, as we need to build a dependency graph, and can't do that without resolving unnamed -> named dependencies before adding the depended-on requirements.

ionelmc mentioned this issue Dec 23, 2014

Allow import to fail, a .pth file is not a good place to raise exceptions schlamar/cov-core#10

Closed

dstufft closed this as completed Dec 23, 2014

dstufft reopened this Dec 23, 2014

ionelmc mentioned this issue Dec 23, 2014

Allow import to fail, a .pth file is not a good place to raise exceptions pytest-dev/pytest-cov#38

Merged

dstufft closed this as completed Dec 23, 2014

piotr-dobrogost mentioned this issue Mar 4, 2015

install ordering issue with pip 6.0.8 and setuptools 12.3 #2473

Closed

taion mentioned this issue Mar 4, 2015

Topological sorting of subrequirements #2478

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 5, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip 6.0 installs in reverse order, as compared to 1.5 #2260

pip 6.0 installs in reverse order, as compared to 1.5 #2260

bukzor commented Dec 23, 2014

ionelmc commented Dec 23, 2014

dstufft commented Dec 23, 2014

schlamar commented Dec 23, 2014

dstufft commented Dec 23, 2014

dstufft commented Dec 23, 2014

bukzor commented Dec 23, 2014

asottile commented Dec 23, 2014

piotr-dobrogost commented Mar 4, 2015

pip 6.0 installs in reverse order, as compared to 1.5 #2260

pip 6.0 installs in reverse order, as compared to 1.5 #2260

Comments

bukzor commented Dec 23, 2014

ionelmc commented Dec 23, 2014

dstufft commented Dec 23, 2014

schlamar commented Dec 23, 2014

dstufft commented Dec 23, 2014

dstufft commented Dec 23, 2014

bukzor commented Dec 23, 2014

asottile commented Dec 23, 2014

piotr-dobrogost commented Mar 4, 2015