Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wonky pth file breaks upgrade #2751

Closed
rbtcollins opened this issue May 4, 2015 · 24 comments
Closed

wonky pth file breaks upgrade #2751

rbtcollins opened this issue May 4, 2015 · 24 comments

Comments

@rbtcollins
Copy link

@rbtcollins rbtcollins commented May 4, 2015

nstalling collected packages: pbr, pip
  Found existing installation: pbr 0.10.0
Cleaning up...
  Removing temporary dir /tmp/pip_build_root...
Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 555, in uninstall
    paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1785, in add_pth
    self.pth[pth_file] = UninstallPthEntries(pth_file)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1868, in __init__
    raise UninstallationError("Cannot remove entries from nonexistent file %s" % pth_file)
UninstallationError: Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented May 4, 2015

pip 1.5.4, trying with newer pip.

@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented May 4, 2015

still present in 6.1.1

@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented May 4, 2015

Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run
    root=options.root_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 687, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 671, in uninstall
    paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 67, in add_pth
    self.pth[pth_file] = UninstallPthEntries(pth_file)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 162, in __init__
    "Cannot remove entries from nonexistent file %s" % pth_file
UninstallationError: Cannot remove entries from nonexistent file /home/robertc/work/openstack/oslo.config/easy-install.pth
@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented May 4, 2015

this is trying to install pbr, which appears to be registered but isn't really there: its not in pip list.

@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented May 4, 2015

There is a /usr/local/lib/python2.7/dist-packages/oslo.config.egg-link file with one line in it /home/robertc/work/openstack/oslo.config

@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented May 4, 2015

And

cat /usr/local/lib/python2.7/dist-packages/easy-install.pth

import sys; sys.__plen = len(sys.path)
/home/robertc/work/openstack/oslo.config
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented May 4, 2015

removing both of those files allowed the install to proceed. I suspect ignoring the missing file would have been better?

@cdeil
Copy link

@cdeil cdeil commented Sep 29, 2015

I think I just ran into the same issue here with pip 7.1.2:
https://travis-ci.org/astropy/astroplan/jobs/82742624#L613
https://github.com/astropy/astroplan/blob/master/.continuous-integration/travis/setup_dependencies_common.sh#L58

Am I doing something wrong?
Is there a workaround?

@xavfernandez
Copy link
Member

@xavfernandez xavfernandez commented Sep 29, 2015

@rbtcollins It would be interesting to know how the initial package was installed ?

@cdeil Sorry I'm not familiar with conda and even less on how it behaves/conflicts with pip but since the issue comes upgrading sphinx which comes from pip install sphinx_rtd_theme you could switch it to pip install sphinx_rtd_theme==0.1.7to avoid the need to upgrade sphinx :)

@cdeil
Copy link

@cdeil cdeil commented Sep 30, 2015

@asmeurer from conda – Maybe you know why conda and pip run into the issue mentioned in my last comment.
Also: sphinx_rtd has sphinx declared as a dependency for for pip, but not for conda. Presumably this should be added for conda?

@asmeurer
Copy link

@asmeurer asmeurer commented Sep 30, 2015

@stxlvt
Copy link

@stxlvt stxlvt commented Dec 17, 2015

I tried filing this as an anaconda bug, but @ilanschnell closed it as a setuptools/pip issue. I'm including a summary of the discussion here:

Setup

D:\workspace>conda info
Current conda install:

             platform : win-32
        conda version : 3.18.5
  conda-build version : 1.14.1
       python version : 2.7.10.final.0
     requests version : 2.8.1
     root environment : D:\anaconda32  (writable)
  default environment : D:\anaconda32
     envs directories : D:\anaconda32\envs
        package cache : D:\anaconda32\pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/win-32/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/win-32/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

D:\workspace>conda create --name tst python
Fetching package metadata: ....
Solving package specifications: ............
Package plan for installation in environment D:\anaconda32\envs\tst:

The following NEW packages will be INSTALLED:

    msvc_runtime: 1.0.1-vc9_0   [vc9]
    pip:          7.1.2-py27_0
    python:       2.7.10-4
    setuptools:   18.4-py27_0
    wheel:        0.26.0-py27_1

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################| 100%
#
# To activate this environment, use:
# > activate tst
#

D:\workspace>activate tst
Activating environment "D:\anaconda32\envs\tst"...

The bug

[tst] D:\workspace>pip install -U setuptools
Collecting setuptools
  Downloading setuptools-18.5-py2.py3-none-any.whl (462kB)
    100% |################################| 462kB 787kB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 18.4
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

Trying to install -U any package depending on setuptools also fails with the same error.

Known workarounds

Adding the --ignore-installed option will sucessfully upgrade setuptools and the problem will be solved for the lifetime of that particular conda environment. Downloading and running ez_setup.py also fixes the problem for that conda environment. The second workaround will create the missing pth file, while the first will not, but in both cases everything seems to work afterwards.

While using conda to upgrade setuptools is also a workaround, it is not a good solution, since a lot of packages on pypi (1822 in 2013) depend on setuptools and therefore try to upgrade setuptools too, when upgraded:

[tst] D:\workspace>pip install -U zc.buildout
Collecting zc.buildout
  Downloading http://python/root/pypi/+f/739/582d22e3ddd5e/zc.buildout-2.5.0-py2
.py3-none-any.whl (261kB)
    100% |################################| 262kB 6.8MB/s
Collecting setuptools>=8.0 (from zc.buildout)
  Downloading http://python/root/pypi/+f/d40/182384798e286/setuptools-18.8.1-py2
.py3-none-any.whl (463kB)
    100% |################################| 466kB 10.2MB/s
Installing collected packages: setuptools, zc.buildout
  Found existing installation: setuptools 18.5
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

A lot of pypi packages are not installable using conda install, and so using conda to upgrade these is not an option.

None of these workarounds are good permanent solutions to this bug because:

  1. They will trip a lot of users who do not know that adding -I fixes the problem and will have to spend a considerable amount of time searching for some solution.
  2. They break automated testing (e.g. tox)
  3. They do not fix the bug permanently on that PC, but just work for the lifetime of the environment, which is no good if the environment is a throwaway environment.

Cause

The error message is correct, there is no easy-install.pth file in site-packages. The error occurs when pip tries to uninstall the old setuptools package, in pip/req/req_uninstall.py:160. The file pip is trying to remove (easy-install.pth) does not exist in a fresh Anaconda/Miniconda installation because no conda packages are able to include this file.

This is a periodically sleeping bug, since it rears its head whenever the conda setuptools version (currently 18.8.1) lags behind the pip setuptools version (currently 19.1.1).

@ilanschnell
Copy link
Contributor

@ilanschnell ilanschnell commented Dec 17, 2015

Thanks for bringing this to my attention. I'm not sure what to do about it (from an Anaconda) perspective. The easy-install.pth cannot really be part of any conda package (as it would overlap with others). When building conda packages, we either install them:

  • without setuptools
  • with --single-version-externally-managed
  • create a .pth file for the specific package <package name>.pth

I think that pip should not assume that easy-install.pth always exists.

@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented Dec 17, 2015

The assumption of easy-install.pth file is guarded by a check for .egg:

        elif dist.location.endswith('.egg'):
            # package installed by easy_install
            # We cannot match on dist.egg_name because it can slightly vary
            # i.e. setuptools-0.6c11-py2.6.egg vs setuptools-0.6rc11-py2.6.egg
            paths_to_remove.add(dist.location)
            easy_install_egg = os.path.split(dist.location)[1]
            easy_install_pth = os.path.join(os.path.dirname(dist.location),
                                            'easy-install.pth')
            paths_to_remove.add_pth(easy_install_pth, './' + easy_install_egg)

So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well. Perhaps don't do that?

@ilanschnell
Copy link
Contributor

@ilanschnell ilanschnell commented Dec 21, 2015

Correct, conda packages usually include .egg-info (as it is installed by setuptools during the build process). If we removed all .egg-info files from conda packages, pip wouldn't be able to see what's installed at all (and try to overwrite packages). Not including .egg-info files wouldn't be a problem for conda itself. For conda, .egg-info files are just like any other files that are part of a conda package. Many conda packages (the ones which are not Python specific, such as libxml2, hdf5, r, etc.) have no .egg-info files either.

@rbtcollins
Copy link
Author

@rbtcollins rbtcollins commented Dec 21, 2015

No, you've misunderstood. .egg-info is not the same as .egg. The test I pasted is for "SOMETHING.egg", not "SOMETHING.egg-info". What you are doing with .egg-info files is probably correct (though I haven't read your code to authoritatively say so).

@stxlvt
Copy link

@stxlvt stxlvt commented Jan 7, 2016

@ilanschnell, are you still of the opinion that this is a setuptools/pip issue?

slivingston added a commit to tulip-control/tulip-control that referenced this issue Jun 26, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 2, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
slivingston added a commit to tulip-control/tulip-control that referenced this issue Jul 8, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 19, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
jcfr added a commit to jcfr/slicer_cli_web_plugin that referenced this issue Sep 3, 2016
This commit fixes the following error:

```
Collecting setuptools==19.4
  Downloading setuptools-19.4-py2.py3-none-any.whl (471kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 26.1.1
Cannot remove entries from nonexistent file /build/miniconda/lib/python2.7/site-packages/easy-install.pth
The command '/bin/sh -c cd $build_path &&     git clone git://github.com/girder/slicer_cli_web.git &&     cd slicer_cli_web &&     pip install -U -r requirements.txt &&     pip install -U setuptools==19.4' returned a non-zero code: 1
```

Here are some more detailed copied from  tulip-control/tulip-control@d6be896:

`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
@luizirber luizirber mentioned this issue Nov 7, 2016
9 of 9 tasks complete
jcfr added a commit to scikit-build/scikit-build that referenced this issue Dec 5, 2016
As suggested by @mivade: "Considering the target audience is likely using
anaconda/miniconda, I would think the most sensible thing is to use the
version used there as the minimum version. A quick check indicates that
setuptools is at version 28.8 there, so I think requiring >=28.0.0 should
work."

See #212, pypa/pip#2751 and ContinuumIO/anaconda-issues#542

Suggested-by: "Michael V. DePalatis" <mike@depalatis.net>
@stxlvt
Copy link

@stxlvt stxlvt commented Dec 8, 2016

FYI, there's activity over at the corresponding conda issue at ContinuumIO/anaconda-issues#542, but this is the general consensus over there:

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Oct 17, 2017

If someone could provide steps to reproduce this issue, that'd be awesome! :)

@stxlvt
Copy link

@stxlvt stxlvt commented Oct 24, 2017

@pradyunsg I present a reliable way to reproduce it in the OP of the anaconda issue. It assumes you have conda installed.

dp451 added a commit to BigVikingGames/dockerfiles that referenced this issue Jan 18, 2018
julia326 added a commit to openvax/neoantigen-vaccine-pipeline that referenced this issue May 16, 2018
hmpf added a commit to hmpf/nav that referenced this issue Nov 5, 2019
* Build docs using python3
* Work around a bug involving pip and setuptools:
  pypa/pip#2751
@chrahunt
Copy link
Member

@chrahunt chrahunt commented Dec 9, 2019

This seems straightforward, so I'll mark it as "awaiting PR".

If a pth file does not exist, then it does not contain any entries so we shouldn't try to edit it.

The error in question is raised here, which is invoked from add_pth, which itself is invoked from here and here.

One fix would be to simply check whether the incoming file exists in add_pth and if not do not use it to create an UninstallPthEntries. A functional test of reinstalling an editably-installed project and an egg project after removing easy-install.pth should be sufficient. The tests should be done in a separate commit and decorated with pytest.mark.xfail so they can be easily validated. I would put them in tests/functional/test_install_upgrade.py.

hmpf added a commit to hmpf/nav that referenced this issue Jan 27, 2020
* Build docs using python3
* Work around a bug involving pip and setuptools:
  pypa/pip#2751
hmpf added a commit to hmpf/nav that referenced this issue Jan 27, 2020
* Build docs using python3
* Work around a bug involving pip and setuptools:
  pypa/pip#2751
@deveshks
Copy link
Contributor

@deveshks deveshks commented Apr 25, 2020

Since as part of #7891 , the UninstallationError has been converted to a logger.warning in case the pth file is not found, IMO this issue can be closed.

@uranusjr
Copy link
Member

@uranusjr uranusjr commented Apr 25, 2020

I’m marking this as awaiting response. Please comment if the recently-released 20.1b1 still has this problem. Otherwise the bot will auto-close this after some time.

@no-response
Copy link

@no-response no-response bot commented May 10, 2020

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

@no-response no-response bot closed this May 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants