New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wonky pth file breaks upgrade #2751

Open
rbtcollins opened this Issue May 4, 2015 · 20 comments

Comments

Projects
None yet
7 participants
@rbtcollins
Contributor

rbtcollins commented May 4, 2015

nstalling collected packages: pbr, pip
  Found existing installation: pbr 0.10.0
Cleaning up...
  Removing temporary dir /tmp/pip_build_root...
Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 555, in uninstall
    paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1785, in add_pth
    self.pth[pth_file] = UninstallPthEntries(pth_file)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1868, in __init__
    raise UninstallationError("Cannot remove entries from nonexistent file %s" % pth_file)
UninstallationError: Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented May 4, 2015

pip 1.5.4, trying with newer pip.

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented May 4, 2015

still present in 6.1.1

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented May 4, 2015

Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run
    root=options.root_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 687, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 671, in uninstall
    paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 67, in add_pth
    self.pth[pth_file] = UninstallPthEntries(pth_file)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 162, in __init__
    "Cannot remove entries from nonexistent file %s" % pth_file
UninstallationError: Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented May 4, 2015

this is trying to install pbr, which appears to be registered but isn't really there: its not in pip list.

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented May 4, 2015

There is a /usr/local/lib/python2.7/dist-packages/oslo.config.egg-link file with one line in it /home/robertc/work/openstack/oslo.config

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented May 4, 2015

And

cat /usr/local/lib/python2.7/dist-packages/easy-install.pth

import sys; sys.__plen = len(sys.path)
/home/robertc/work/openstack/oslo.config
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented May 4, 2015

removing both of those files allowed the install to proceed. I suspect ignoring the missing file would have been better?

@cdeil

This comment has been minimized.

cdeil commented Sep 29, 2015

I think I just ran into the same issue here with pip 7.1.2:
https://travis-ci.org/astropy/astroplan/jobs/82742624#L613
https://github.com/astropy/astroplan/blob/master/.continuous-integration/travis/setup_dependencies_common.sh#L58

Am I doing something wrong?
Is there a workaround?

@xavfernandez

This comment has been minimized.

Contributor

xavfernandez commented Sep 29, 2015

@rbtcollins It would be interesting to know how the initial package was installed ?

@cdeil Sorry I'm not familiar with conda and even less on how it behaves/conflicts with pip but since the issue comes upgrading sphinx which comes from pip install sphinx_rtd_theme you could switch it to pip install sphinx_rtd_theme==0.1.7to avoid the need to upgrade sphinx :)

@cdeil

This comment has been minimized.

cdeil commented Sep 30, 2015

@asmeurer from conda – Maybe you know why conda and pip run into the issue mentioned in my last comment.
Also: sphinx_rtd has sphinx declared as a dependency for for pip, but not for conda. Presumably this should be added for conda?

@asmeurer

This comment has been minimized.

asmeurer commented Sep 30, 2015

@stxlvt

This comment has been minimized.

stxlvt commented Dec 17, 2015

I tried filing this as an anaconda bug, but @ilanschnell closed it as a setuptools/pip issue. I'm including a summary of the discussion here:

Setup

D:\workspace>conda info
Current conda install:

             platform : win-32
        conda version : 3.18.5
  conda-build version : 1.14.1
       python version : 2.7.10.final.0
     requests version : 2.8.1
     root environment : D:\anaconda32  (writable)
  default environment : D:\anaconda32
     envs directories : D:\anaconda32\envs
        package cache : D:\anaconda32\pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/win-32/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/win-32/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

D:\workspace>conda create --name tst python
Fetching package metadata: ....
Solving package specifications: ............
Package plan for installation in environment D:\anaconda32\envs\tst:

The following NEW packages will be INSTALLED:

    msvc_runtime: 1.0.1-vc9_0   [vc9]
    pip:          7.1.2-py27_0
    python:       2.7.10-4
    setuptools:   18.4-py27_0
    wheel:        0.26.0-py27_1

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################| 100%
#
# To activate this environment, use:
# > activate tst
#

D:\workspace>activate tst
Activating environment "D:\anaconda32\envs\tst"...

The bug

[tst] D:\workspace>pip install -U setuptools
Collecting setuptools
  Downloading setuptools-18.5-py2.py3-none-any.whl (462kB)
    100% |################################| 462kB 787kB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 18.4
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

Trying to install -U any package depending on setuptools also fails with the same error.

Known workarounds

Adding the --ignore-installed option will sucessfully upgrade setuptools and the problem will be solved for the lifetime of that particular conda environment. Downloading and running ez_setup.py also fixes the problem for that conda environment. The second workaround will create the missing pth file, while the first will not, but in both cases everything seems to work afterwards.

While using conda to upgrade setuptools is also a workaround, it is not a good solution, since a lot of packages on pypi (1822 in 2013) depend on setuptools and therefore try to upgrade setuptools too, when upgraded:

[tst] D:\workspace>pip install -U zc.buildout
Collecting zc.buildout
  Downloading http://python/root/pypi/+f/739/582d22e3ddd5e/zc.buildout-2.5.0-py2
.py3-none-any.whl (261kB)
    100% |################################| 262kB 6.8MB/s
Collecting setuptools>=8.0 (from zc.buildout)
  Downloading http://python/root/pypi/+f/d40/182384798e286/setuptools-18.8.1-py2
.py3-none-any.whl (463kB)
    100% |################################| 466kB 10.2MB/s
Installing collected packages: setuptools, zc.buildout
  Found existing installation: setuptools 18.5
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

A lot of pypi packages are not installable using conda install, and so using conda to upgrade these is not an option.

None of these workarounds are good permanent solutions to this bug because:

  1. They will trip a lot of users who do not know that adding -I fixes the problem and will have to spend a considerable amount of time searching for some solution.
  2. They break automated testing (e.g. tox)
  3. They do not fix the bug permanently on that PC, but just work for the lifetime of the environment, which is no good if the environment is a throwaway environment.

Cause

The error message is correct, there is no easy-install.pth file in site-packages. The error occurs when pip tries to uninstall the old setuptools package, in pip/req/req_uninstall.py:160. The file pip is trying to remove (easy-install.pth) does not exist in a fresh Anaconda/Miniconda installation because no conda packages are able to include this file.

This is a periodically sleeping bug, since it rears its head whenever the conda setuptools version (currently 18.8.1) lags behind the pip setuptools version (currently 19.1.1).

@ilanschnell

This comment has been minimized.

ilanschnell commented Dec 17, 2015

Thanks for bringing this to my attention. I'm not sure what to do about it (from an Anaconda) perspective. The easy-install.pth cannot really be part of any conda package (as it would overlap with others). When building conda packages, we either install them:

  • without setuptools
  • with --single-version-externally-managed
  • create a .pth file for the specific package <package name>.pth

I think that pip should not assume that easy-install.pth always exists.

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented Dec 17, 2015

The assumption of easy-install.pth file is guarded by a check for .egg:

        elif dist.location.endswith('.egg'):
            # package installed by easy_install
            # We cannot match on dist.egg_name because it can slightly vary
            # i.e. setuptools-0.6c11-py2.6.egg vs setuptools-0.6rc11-py2.6.egg
            paths_to_remove.add(dist.location)
            easy_install_egg = os.path.split(dist.location)[1]
            easy_install_pth = os.path.join(os.path.dirname(dist.location),
                                            'easy-install.pth')
            paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)

So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well. Perhaps don't do that?

@ilanschnell

This comment has been minimized.

ilanschnell commented Dec 21, 2015

Correct, conda packages usually include .egg-info (as it is installed by setuptools during the build process). If we removed all .egg-info files from conda packages, pip wouldn't be able to see what's installed at all (and try to overwrite packages). Not including .egg-info files wouldn't be a problem for conda itself. For conda, .egg-info files are just like any other files that are part of a conda package. Many conda packages (the ones which are not Python specific, such as libxml2, hdf5, r, etc.) have no .egg-info files either.

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented Dec 21, 2015

No, you've misunderstood. .egg-info is not the same as .egg. The test I pasted is for "SOMETHING.egg", not "SOMETHING.egg-info". What you are doing with .egg-info files is probably correct (though I haven't read your code to authoritatively say so).

@stxlvt

This comment has been minimized.

stxlvt commented Jan 7, 2016

@ilanschnell, are you still of the opinion that this is a setuptools/pip issue?

slivingston added a commit to tulip-control/tulip-control that referenced this issue Jun 26, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542

johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 2, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542

slivingston added a commit to tulip-control/tulip-control that referenced this issue Jul 8, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542

johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 19, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542

jcfr added a commit to jcfr/slicer_cli_web_plugin that referenced this issue Sep 3, 2016

dockerfile: Use `--ignore-installed` as workaround of conda
This commit fixes the following error:

```
Collecting setuptools==19.4
  Downloading setuptools-19.4-py2.py3-none-any.whl (471kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 26.1.1
Cannot remove entries from nonexistent file /build/miniconda/lib/python2.7/site-packages/easy-install.pth
The command '/bin/sh -c cd $build_path &&     git clone git://github.com/girder/slicer_cli_web.git &&     cd slicer_cli_web &&     pip install -U -r requirements.txt &&     pip install -U setuptools==19.4' returned a non-zero code: 1
```

Here are some more detailed copied from  tulip-control/tulip-control@d6be896:

`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542

@luizirber luizirber referenced this issue Nov 7, 2016

Merged

Fix two installation quirks #1505

9 of 9 tasks complete

jcfr added a commit to scikit-build/scikit-build that referenced this issue Dec 5, 2016

requirements: unpin version of setuptools and require >= 28.0.0
As suggested by @mivade: "Considering the target audience is likely using
anaconda/miniconda, I would think the most sensible thing is to use the
version used there as the minimum version. A quick check indicates that
setuptools is at version 28.8 there, so I think requiring >=28.0.0 should
work."

See #212, pypa/pip#2751 and ContinuumIO/anaconda-issues#542

Suggested-by: "Michael V. DePalatis" <mike@depalatis.net>
@stxlvt

This comment has been minimized.

stxlvt commented Dec 8, 2016

FYI, there's activity over at the corresponding conda issue at ContinuumIO/anaconda-issues#542, but this is the general consensus over there:

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

@pradyunsg

This comment has been minimized.

Member

pradyunsg commented Oct 17, 2017

If someone could provide steps to reproduce this issue, that'd be awesome! :)

@pradyunsg pradyunsg added the K: crash label Oct 17, 2017

@stxlvt

This comment has been minimized.

stxlvt commented Oct 24, 2017

@pradyunsg I present a reliable way to reproduce it in the OP of the anaconda issue. It assumes you have conda installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment