New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install -U setuptools fails in fresh conda environment #542

Open
stxlvt opened this Issue Nov 18, 2015 · 34 comments

Comments

Projects
None yet
@stxlvt

stxlvt commented Nov 18, 2015

Setup:

D:\workspace>conda info
Current conda install:

             platform : win-32
        conda version : 3.18.5
  conda-build version : 1.14.1
       python version : 2.7.10.final.0
     requests version : 2.8.1
     root environment : D:\anaconda32  (writable)
  default environment : D:\anaconda32
     envs directories : D:\anaconda32\envs
        package cache : D:\anaconda32\pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/win-32/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/win-32/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

D:\workspace>conda create --name tst python
Fetching package metadata: ....
Solving package specifications: ............
Package plan for installation in environment D:\anaconda32\envs\tst:

The following NEW packages will be INSTALLED:

    msvc_runtime: 1.0.1-vc9_0   [vc9]
    pip:          7.1.2-py27_0
    python:       2.7.10-4
    setuptools:   18.4-py27_0
    wheel:        0.26.0-py27_1

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################| 100%
#
# To activate this environment, use:
# > activate tst
#

D:\workspace>activate tst
Activating environment "D:\anaconda32\envs\tst"...

The bug:

[tst] D:\workspace>pip install -U setuptools
Collecting setuptools
  Downloading setuptools-18.5-py2.py3-none-any.whl (462kB)
    100% |################################| 462kB 787kB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 18.4
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

Trying to install -U any package depending on setuptools also fails with the same error.

Known workarounds:

  • Adding the --ignore-installed option will sucessfully upgrade setuptools and the problem will be solved for the lifetime of that particular conda environment.
  • Downloading and running ez_setup.py also fixes the problem for that conda environment.

The error message is correct, there is no easy-install.pth file in site-packages. The error occurs when pip tries to uninstall the old setuptools package, in pip/req/req_uninstall.py:160.

The second workaround will create the missing pth file, while the first will not, but in both cases everything seems to work afterwards.

I am unsure if this is a pip issue or a conda issue, but because it is your setuptools build that makes pip choke, I choose to submit it here.

@yosuah

This comment has been minimized.

yosuah commented Dec 11, 2015

Thank you for the report, I just hit the same issue and could solve it using the --ignore-installed option.

@ccordoba12

This comment has been minimized.

ccordoba12 commented Dec 11, 2015

I'd recommend you to not upgrade setuptools using pip. It's better to wait for us to provide a new package because (as you noticed) we don't use easy-install.pth.

I think that was decided to avoid setuptools messing with sys.path (which is one of its least appreciated features :-)

@stxlvt

This comment has been minimized.

stxlvt commented Dec 15, 2015

Thank you for your reply, ccordoba12. The problem with the "then don't do that" advice is that this bug also hits you if you try to upgrade a package that depends on setuptools. For example:

[tst] D:\workspace>pip install -U zc.buildout
Collecting zc.buildout
  Downloading http://python/root/pypi/+f/739/582d22e3ddd5e/zc.buildout-2.5.0-py2
.py3-none-any.whl (261kB)
    100% |################################| 262kB 6.8MB/s
Collecting setuptools>=8.0 (from zc.buildout)
  Downloading http://python/root/pypi/+f/d40/182384798e286/setuptools-18.8.1-py2
.py3-none-any.whl (463kB)
    100% |################################| 466kB 10.2MB/s
Installing collected packages: setuptools, zc.buildout
  Found existing installation: setuptools 18.5
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

To try to figure out how big a problem this is, I downloaded the file pypi-deps.csv.lzma from here, and it showed that at the time that data was compiled in 2013, 1822 pypi packages depended on setuptools, and a further 41 depended on one of those 1822 packages. That data is used in this zoomable graph of pypi dependencies. Setuptools is the red dot in the middle of the biggest blob. It seems like no other package is depended on as much as setuptools.

I can totally understand not wanting to mess with sys.path. Been there, done that. But I had hoped you could find a way to solve this that does not break updating a big portion of the packages on pypi.

@ccordoba12

This comment has been minimized.

ccordoba12 commented Dec 15, 2015

Pinging @ilanschnell about this one.

@ilanschnell

This comment has been minimized.

Contributor

ilanschnell commented Dec 15, 2015

Use conda to update setuptools:

conda update setuptools

Using pip, I get the same error:

Cannot remove entries from nonexistent file /Users/ilan/anaconda/ibuild/tmp/lib/python2.7/site-packages/easy-install.pth

The file pip is trying to remove (easy-install.pth) does not exist in a fresh Anaconda/Miniconda installation because no conda packages are able to include this file. I consider it a pip (or setuptools) bug that pip install -U setuptools fails in the absence of this file.

@bkreider

This comment has been minimized.

Member

bkreider commented Mar 24, 2016

@kalefranz Can you look at this? It's still an issue (as seen on the mailing list)

It's tied to this issue over at pip: pypa/pip#2751 (comment)

So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well.

It's true. The conda package for setuptools installs an egg.

@denfromufa

This comment has been minimized.

denfromufa commented May 3, 2016

this worked for me perfectly on Windows (original recipe for OSX):

tensorflow/tensorflow#135 (comment)

curl https://bootstrap.pypa.io/ez_setup.py -o - | python

slivingston added a commit to tulip-control/tulip-control that referenced this issue Jun 26, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542

johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 2, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542

slivingston added a commit to tulip-control/tulip-control that referenced this issue Jul 8, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
@burkesquires

This comment has been minimized.

burkesquires commented Jul 8, 2016

The --ignore-installed helped to get past the error for upgrading to iPython 5.0.

I was getting the following error when running pip install ipython --upgrade

Installing collected packages: setuptools, traitlets, ipython
Found existing installation: setuptools 23.0.0
Cannot remove entries from nonexistent file /Users/squiresrb/anaconda/lib/python3.5/site-packages/easy-install.pth

Thanks!

@stxlvt

This comment has been minimized.

stxlvt commented Jul 12, 2016

@ilanschnell @ccordoba12

Please consider reopening this bug, or at least responding to pypa/pip#2751 which awaits an answer from you guys. This problem is not going away, and there is a constant stream of references from other projects who have this problem, both here and at pypa/pip#2751. And that's only the tip of the iceberg, those that do not just find this workaround and apply it without telling us about it.

First step to solving this problem is that you guys and the pip guys must determine whose problem it is. :-)

To bring you up to speed on where things stand I'll give you a summary. @rbtcollins pointed out that...

The assumption of easy-install.pth file is guarded by a check for .egg. So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well. Perhaps don't do that?

To which @ilanschnell answered:

Correct, conda packages usually include .egg-info [...]

And @rbtcollins replied:

No, you've misunderstood. .egg-info is not the same as .egg. The test I pasted is for "SOMETHING.egg", not "SOMETHING.egg-info".

So. Could you perhaps make setuptools look more like the pip setuptools by making it a normal package instead of an egg? Would that be an acceptable solution?

johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 19, 2016

TEST: Use `--ignore-installed` as workaround of conda
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
@ssbarnea

This comment has been minimized.

ssbarnea commented Aug 5, 2016

Can we have an update on this? This bg break tons of packages making impossible to deploy anything on conda. Every day I find another case where I reach the infamous:

Cannot remove entries from nonexistent file ~/anaconda3/lib/python3.5/site-packages/easy-install.pth

Here is a very simple way to replicate the problem

conda install flake8
# will install flake 2.6.x with mccabe 0.3.1
python3 -m pip install --upgrade flake8 
# will attempt to install flake8 which needs a new version of mccabe ... and pip will fail to uninstall it

jcfr added a commit to jcfr/slicer_cli_web_plugin that referenced this issue Sep 3, 2016

dockerfile: Use `--ignore-installed` as workaround of conda
This commit fixes the following error:

```
Collecting setuptools==19.4
  Downloading setuptools-19.4-py2.py3-none-any.whl (471kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 26.1.1
Cannot remove entries from nonexistent file /build/miniconda/lib/python2.7/site-packages/easy-install.pth
The command '/bin/sh -c cd $build_path &&     git clone git://github.com/girder/slicer_cli_web.git &&     cd slicer_cli_web &&     pip install -U -r requirements.txt &&     pip install -U setuptools==19.4' returned a non-zero code: 1
```

Here are some more detailed copied from  tulip-control/tulip-control@d6be896:

`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
@chebee7i

This comment has been minimized.

chebee7i commented Sep 12, 2016

An update on this would be nice. I just hit this as well.

musically-ut added a commit to Networks-Learning/l1-ls.py that referenced this issue Oct 12, 2016

DavideBaroliUniLu added a commit to DavideBaroliUniLu/PyGeM that referenced this issue Nov 10, 2016

@kalefranz

This comment has been minimized.

Member

kalefranz commented Dec 7, 2016

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

The bug in pip should be patched. And we should either apply that patch to our setuptools build or hack around it with the easy-install.pth fix. We can add a post-unlink script that removes easy-install.pth if it's empty.

@ilanschnell

This comment has been minimized.

Contributor

ilanschnell commented Dec 7, 2016

The patch to pip could be simply to create an empty easy-install.pth if the files does not exist. This would keep the conda packages clean of hacks.

@ilanschnell

This comment has been minimized.

Contributor

ilanschnell commented Dec 7, 2016

I'm going ahead and writing a little patch for pip which creates an empty easy-install.pth if it does not exist yet.

@csoja

This comment has been minimized.

Contributor

csoja commented Dec 7, 2016

Thank you @ilanschnell !

@ilanschnell

This comment has been minimized.

Contributor

ilanschnell commented Dec 7, 2016

I've added a patch (https://github.com/ContinuumIO/anaconda-recipes/blob/master/pip/ensure_easy_pth.patch) to pip, and build new packages (pip-9.0.1 build 1). However, while this patch is certainly an improvement, pip install -U setuptools still does not work for me because of a different error.

@dstufft

This comment has been minimized.

dstufft commented Dec 8, 2016

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

It's not entirely clear to me that this is the case. As best I can tell pip will only care about an easy-install.pth file in the case that it finds either a .egg in site-packages or it's attempting to uninstall something that had been installed in develop mode. In both of these cases I believe setuptools will always ensure that there is a easy-install.pth file. Do you know which case you're hitting to trigger pip to care about this file at all? .egg or develop install?

mfeurer added a commit to automl/HPOlib2 that referenced this issue Dec 9, 2016

@jgwerner jgwerner referenced this issue Dec 31, 2016

Merged

setuptools fix #2

felliott added a commit to CenterForOpenScience/waterbutler that referenced this issue Jan 24, 2017

add WB's pip reqs to conda's environment.yml to build code docs
 * The conda environment on readthedocs.org could successfully build
   the static docs, but failed to build the code-level docs b/c of
   missing dependencies.  This commit adds the dependencies from
   requirements.txt to the pip section of conda's environment.yml.

 * setuptools had to be removed from the pip section of
   environment.yml because conda has issues with installing setuptools
   via pip. See:
   ContinuumIO/anaconda-issues#542
   Instead, use conda's provided setuptools.

 * Removed extraneous dependencies

alexrudy added a commit to alexrudy/FTR that referenced this issue Feb 3, 2017

@markdem

This comment has been minimized.

markdem commented Feb 13, 2017

Trying to install Kivy (Python framework) on Windows 10 from the VS Command prompt (as admin).
Getting the same error:

 Cannot remove entries from nonexistent file c:\users\m\anaconda3\lib\site-packages\easy-install.pth

I see that ilanschnell created a patch for this problem, but I am unsure which directory I should place this patch in.

Can someone please advise.

@fmelectrical

This comment has been minimized.

fmelectrical commented Feb 19, 2017

Sorry I used Pythonistribution but after some days it did not work so I uninstalled it and when I wanted to reinstall it I amreceiving this message
Doc1.pdf
Doc1.pdf

And I cannot install anaconda again

@fmelectrical

This comment has been minimized.

fmelectrical commented Feb 19, 2017

please help me to solve my problem and instal anaconda again and use it

@denfromufa

This comment has been minimized.

denfromufa commented Feb 19, 2017

@fmelectrical your issue is not related to this thread at all.

@niklasholm

This comment has been minimized.

niklasholm commented Feb 23, 2017

@ilanschnell I get this error instead with the latest build of pip:

OSError: [Errno 2] No such file or directory: '[...]/.local/miniconda3/envs/issue542test/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg'

Is it the same you got?

rafaeldelucena added a commit to luismulinari/waterbutler that referenced this issue Apr 10, 2017

add WB's pip reqs to conda's environment.yml to build code docs
 * The conda environment on readthedocs.org could successfully build
   the static docs, but failed to build the code-level docs b/c of
   missing dependencies.  This commit adds the dependencies from
   requirements.txt to the pip section of conda's environment.yml.

 * setuptools had to be removed from the pip section of
   environment.yml because conda has issues with installing setuptools
   via pip. See:
   ContinuumIO/anaconda-issues#542
   Instead, use conda's provided setuptools.

 * Removed extraneous dependencies
@myidealab

This comment has been minimized.

myidealab commented Apr 29, 2017

@niklasholm
To resolve the error you got, I reinstalled setuptools and the error went away for the package (newspaper3k) I was trying to import.

    MBPRO-D12LK3WJJC45:~ myidealab$ conda install -c anaconda setuptools=27.2.0
    Fetching package metadata ...........
    Solving package specifications: .

    Package plan for installation in environment /Users/myidealab/anaconda:

    The following packages will be SUPERSEDED by a higher-priority channel:

    conda:      4.3.16-py36_0 --> 4.3.16-py36_0 anaconda
    conda-env:  2.6.0-0       --> 2.6.0-0       anaconda
    setuptools: 27.2.0-py36_0 --> 27.2.0-py36_0 anaconda

    Proceed ([y]/n)? y

    conda-env-2.6. 100% |#################################| Time: 0:00:00 187.75 kB/s
    setuptools-27. 100% |#################################| Time: 0:00:00 919.02 kB/s
    conda-4.3.16-p 100% |#################################| Time: 0:00:00 754.75 kB/s
@ronzillia

This comment has been minimized.

ronzillia commented Mar 23, 2018

this can be resolved if you do conda install setuptools
I am using linux

jaimergp added a commit to insilichem/tangram that referenced this issue Apr 12, 2018

dorukozturk added a commit to NeurodataWithoutBorders/pynwb that referenced this issue Aug 17, 2018

dorukozturk added a commit to NeurodataWithoutBorders/pynwb that referenced this issue Aug 17, 2018

KEggensperger pushed a commit to automl/HPOlib2 that referenced this issue Sep 13, 2018

KEggensperger pushed a commit to automl/HPOlib2 that referenced this issue Sep 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment