Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install -U setuptools fails in fresh conda environment #542

Open
stxlvt opened this issue Nov 18, 2015 · 34 comments
Open

pip install -U setuptools fails in fresh conda environment #542

stxlvt opened this issue Nov 18, 2015 · 34 comments

Comments

@stxlvt
Copy link

@stxlvt stxlvt commented Nov 18, 2015

Setup:

D:\workspace>conda info
Current conda install:

             platform : win-32
        conda version : 3.18.5
  conda-build version : 1.14.1
       python version : 2.7.10.final.0
     requests version : 2.8.1
     root environment : D:\anaconda32  (writable)
  default environment : D:\anaconda32
     envs directories : D:\anaconda32\envs
        package cache : D:\anaconda32\pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/win-32/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/win-32/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

D:\workspace>conda create --name tst python
Fetching package metadata: ....
Solving package specifications: ............
Package plan for installation in environment D:\anaconda32\envs\tst:

The following NEW packages will be INSTALLED:

    msvc_runtime: 1.0.1-vc9_0   [vc9]
    pip:          7.1.2-py27_0
    python:       2.7.10-4
    setuptools:   18.4-py27_0
    wheel:        0.26.0-py27_1

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################| 100%
#
# To activate this environment, use:
# > activate tst
#

D:\workspace>activate tst
Activating environment "D:\anaconda32\envs\tst"...

The bug:

[tst] D:\workspace>pip install -U setuptools
Collecting setuptools
  Downloading setuptools-18.5-py2.py3-none-any.whl (462kB)
    100% |################################| 462kB 787kB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 18.4
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

Trying to install -U any package depending on setuptools also fails with the same error.

Known workarounds:

  • Adding the --ignore-installed option will sucessfully upgrade setuptools and the problem will be solved for the lifetime of that particular conda environment.
  • Downloading and running ez_setup.py also fixes the problem for that conda environment.

The error message is correct, there is no easy-install.pth file in site-packages. The error occurs when pip tries to uninstall the old setuptools package, in pip/req/req_uninstall.py:160.

The second workaround will create the missing pth file, while the first will not, but in both cases everything seems to work afterwards.

I am unsure if this is a pip issue or a conda issue, but because it is your setuptools build that makes pip choke, I choose to submit it here.

@yosuah
Copy link

@yosuah yosuah commented Dec 11, 2015

Thank you for the report, I just hit the same issue and could solve it using the --ignore-installed option.

@ccordoba12
Copy link

@ccordoba12 ccordoba12 commented Dec 11, 2015

I'd recommend you to not upgrade setuptools using pip. It's better to wait for us to provide a new package because (as you noticed) we don't use easy-install.pth.

I think that was decided to avoid setuptools messing with sys.path (which is one of its least appreciated features :-)

@stxlvt
Copy link
Author

@stxlvt stxlvt commented Dec 15, 2015

Thank you for your reply, ccordoba12. The problem with the "then don't do that" advice is that this bug also hits you if you try to upgrade a package that depends on setuptools. For example:

[tst] D:\workspace>pip install -U zc.buildout
Collecting zc.buildout
  Downloading http://python/root/pypi/+f/739/582d22e3ddd5e/zc.buildout-2.5.0-py2
.py3-none-any.whl (261kB)
    100% |################################| 262kB 6.8MB/s
Collecting setuptools>=8.0 (from zc.buildout)
  Downloading http://python/root/pypi/+f/d40/182384798e286/setuptools-18.8.1-py2
.py3-none-any.whl (463kB)
    100% |################################| 466kB 10.2MB/s
Installing collected packages: setuptools, zc.buildout
  Found existing installation: setuptools 18.5
Cannot remove entries from nonexistent file d:\anaconda32\envs\tst\lib\site-pack
ages\easy-install.pth

To try to figure out how big a problem this is, I downloaded the file pypi-deps.csv.lzma from here, and it showed that at the time that data was compiled in 2013, 1822 pypi packages depended on setuptools, and a further 41 depended on one of those 1822 packages. That data is used in this zoomable graph of pypi dependencies. Setuptools is the red dot in the middle of the biggest blob. It seems like no other package is depended on as much as setuptools.

I can totally understand not wanting to mess with sys.path. Been there, done that. But I had hoped you could find a way to solve this that does not break updating a big portion of the packages on pypi.

@ccordoba12
Copy link

@ccordoba12 ccordoba12 commented Dec 15, 2015

Pinging @ilanschnell about this one.

@ilanschnell
Copy link
Contributor

@ilanschnell ilanschnell commented Dec 15, 2015

Use conda to update setuptools:

conda update setuptools

Using pip, I get the same error:

Cannot remove entries from nonexistent file /Users/ilan/anaconda/ibuild/tmp/lib/python2.7/site-packages/easy-install.pth

The file pip is trying to remove (easy-install.pth) does not exist in a fresh Anaconda/Miniconda installation because no conda packages are able to include this file. I consider it a pip (or setuptools) bug that pip install -U setuptools fails in the absence of this file.

@bkreider
Copy link
Member

@bkreider bkreider commented Mar 24, 2016

@kalefranz Can you look at this? It's still an issue (as seen on the mailing list)

It's tied to this issue over at pip: pypa/pip#2751 (comment)

So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well.

It's true. The conda package for setuptools installs an egg.

@denfromufa
Copy link

@denfromufa denfromufa commented May 3, 2016

this worked for me perfectly on Windows (original recipe for OSX):

tensorflow/tensorflow#135 (comment)

curl https://bootstrap.pypa.io/ez_setup.py -o - | python

slivingston added a commit to tulip-control/tulip-control that referenced this issue Jun 26, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 2, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
slivingston added a commit to tulip-control/tulip-control that referenced this issue Jul 8, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
@burkesquires
Copy link

@burkesquires burkesquires commented Jul 8, 2016

The --ignore-installed helped to get past the error for upgrading to iPython 5.0.

I was getting the following error when running pip install ipython --upgrade

Installing collected packages: setuptools, traitlets, ipython
Found existing installation: setuptools 23.0.0
Cannot remove entries from nonexistent file /Users/squiresrb/anaconda/lib/python3.5/site-packages/easy-install.pth

Thanks!

@stxlvt
Copy link
Author

@stxlvt stxlvt commented Jul 12, 2016

@ilanschnell @ccordoba12

Please consider reopening this bug, or at least responding to pypa/pip#2751 which awaits an answer from you guys. This problem is not going away, and there is a constant stream of references from other projects who have this problem, both here and at pypa/pip#2751. And that's only the tip of the iceberg, those that do not just find this workaround and apply it without telling us about it.

First step to solving this problem is that you guys and the pip guys must determine whose problem it is. :-)

To bring you up to speed on where things stand I'll give you a summary. @rbtcollins pointed out that...

The assumption of easy-install.pth file is guarded by a check for .egg. So my guess is that your conda package looks like an easy_install egg: this may confuse other tools as well. Perhaps don't do that?

To which @ilanschnell answered:

Correct, conda packages usually include .egg-info [...]

And @rbtcollins replied:

No, you've misunderstood. .egg-info is not the same as .egg. The test I pasted is for "SOMETHING.egg", not "SOMETHING.egg-info".

So. Could you perhaps make setuptools look more like the pip setuptools by making it a normal package instead of an egg? Would that be an acceptable solution?

johnyf added a commit to tulip-control/tulip-control that referenced this issue Jul 19, 2016
`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
@ssbarnea
Copy link

@ssbarnea ssbarnea commented Aug 5, 2016

Can we have an update on this? This bg break tons of packages making impossible to deploy anything on conda. Every day I find another case where I reach the infamous:

Cannot remove entries from nonexistent file ~/anaconda3/lib/python3.5/site-packages/easy-install.pth

Here is a very simple way to replicate the problem

conda install flake8
# will install flake 2.6.x with mccabe 0.3.1
python3 -m pip install --upgrade flake8 
# will attempt to install flake8 which needs a new version of mccabe ... and pip will fail to uninstall it
jcfr added a commit to jcfr/slicer_cli_web_plugin that referenced this issue Sep 3, 2016
This commit fixes the following error:

```
Collecting setuptools==19.4
  Downloading setuptools-19.4-py2.py3-none-any.whl (471kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 26.1.1
Cannot remove entries from nonexistent file /build/miniconda/lib/python2.7/site-packages/easy-install.pth
The command '/bin/sh -c cd $build_path &&     git clone git://github.com/girder/slicer_cli_web.git &&     cd slicer_cli_web &&     pip install -U -r requirements.txt &&     pip install -U setuptools==19.4' returned a non-zero code: 1
```

Here are some more detailed copied from  tulip-control/tulip-control@d6be896:

`conda` ships with setuptools but lacks a corresponding
easy-install.pth. Attempting to upgrade setuptools using `pip` fails
because easy-install.pth is missing, despite the presentation of the
conda-originating setuptools as an easy_install egg. It is a known bug
of `conda` that is documented at the following:

pypa/pip#2751
ContinuumIO/anaconda-issues#542
@chebee7i
Copy link

@chebee7i chebee7i commented Sep 12, 2016

An update on this would be nice. I just hit this as well.

@ilanschnell
Copy link
Contributor

@ilanschnell ilanschnell commented Dec 7, 2016

I'm going ahead and writing a little patch for pip which creates an empty easy-install.pth if it does not exist yet.

@csoja
Copy link
Contributor

@csoja csoja commented Dec 7, 2016

Thank you @ilanschnell !

@ilanschnell
Copy link
Contributor

@ilanschnell ilanschnell commented Dec 7, 2016

I've added a patch (https://github.com/ContinuumIO/anaconda-recipes/blob/master/pip/ensure_easy_pth.patch) to pip, and build new packages (pip-9.0.1 build 1). However, while this patch is certainly an improvement, pip install -U setuptools still does not work for me because of a different error.

@dstufft
Copy link

@dstufft dstufft commented Dec 8, 2016

I squarely consider this a bug in pip. So while it's pip's fault, it's our problem, because it's causing problems for our users.

It's not entirely clear to me that this is the case. As best I can tell pip will only care about an easy-install.pth file in the case that it finds either a .egg in site-packages or it's attempting to uninstall something that had been installed in develop mode. In both of these cases I believe setuptools will always ensure that there is a easy-install.pth file. Do you know which case you're hitting to trigger pip to care about this file at all? .egg or develop install?

mfeurer added a commit to automl/HPOlib1.5 that referenced this issue Dec 9, 2016
felliott added a commit to CenterForOpenScience/waterbutler that referenced this issue Jan 24, 2017
 * The conda environment on readthedocs.org could successfully build
   the static docs, but failed to build the code-level docs b/c of
   missing dependencies.  This commit adds the dependencies from
   requirements.txt to the pip section of conda's environment.yml.

 * setuptools had to be removed from the pip section of
   environment.yml because conda has issues with installing setuptools
   via pip. See:
   ContinuumIO/anaconda-issues#542
   Instead, use conda's provided setuptools.

 * Removed extraneous dependencies
alexrudy added a commit to alexrudy/FTR that referenced this issue Feb 3, 2017
@ghost
Copy link

@ghost ghost commented Feb 13, 2017

Trying to install Kivy (Python framework) on Windows 10 from the VS Command prompt (as admin).
Getting the same error:

 Cannot remove entries from nonexistent file c:\users\m\anaconda3\lib\site-packages\easy-install.pth

I see that ilanschnell created a patch for this problem, but I am unsure which directory I should place this patch in.

Can someone please advise.

@fmelectrical
Copy link

@fmelectrical fmelectrical commented Feb 19, 2017

Sorry I used Pythonistribution but after some days it did not work so I uninstalled it and when I wanted to reinstall it I amreceiving this message
Doc1.pdf
Doc1.pdf

And I cannot install anaconda again

@fmelectrical
Copy link

@fmelectrical fmelectrical commented Feb 19, 2017

please help me to solve my problem and instal anaconda again and use it

@denfromufa
Copy link

@denfromufa denfromufa commented Feb 19, 2017

@fmelectrical your issue is not related to this thread at all.

@niklasholm
Copy link

@niklasholm niklasholm commented Feb 23, 2017

@ilanschnell I get this error instead with the latest build of pip:

OSError: [Errno 2] No such file or directory: '[...]/.local/miniconda3/envs/issue542test/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg'

Is it the same you got?

rafaeldelucena added a commit to luismulinari/waterbutler that referenced this issue Apr 10, 2017
 * The conda environment on readthedocs.org could successfully build
   the static docs, but failed to build the code-level docs b/c of
   missing dependencies.  This commit adds the dependencies from
   requirements.txt to the pip section of conda's environment.yml.

 * setuptools had to be removed from the pip section of
   environment.yml because conda has issues with installing setuptools
   via pip. See:
   ContinuumIO/anaconda-issues#542
   Instead, use conda's provided setuptools.

 * Removed extraneous dependencies
@myidealab
Copy link

@myidealab myidealab commented Apr 29, 2017

@niklasholm
To resolve the error you got, I reinstalled setuptools and the error went away for the package (newspaper3k) I was trying to import.

    MBPRO-D12LK3WJJC45:~ myidealab$ conda install -c anaconda setuptools=27.2.0
    Fetching package metadata ...........
    Solving package specifications: .

    Package plan for installation in environment /Users/myidealab/anaconda:

    The following packages will be SUPERSEDED by a higher-priority channel:

    conda:      4.3.16-py36_0 --> 4.3.16-py36_0 anaconda
    conda-env:  2.6.0-0       --> 2.6.0-0       anaconda
    setuptools: 27.2.0-py36_0 --> 27.2.0-py36_0 anaconda

    Proceed ([y]/n)? y

    conda-env-2.6. 100% |#################################| Time: 0:00:00 187.75 kB/s
    setuptools-27. 100% |#################################| Time: 0:00:00 919.02 kB/s
    conda-4.3.16-p 100% |#################################| Time: 0:00:00 754.75 kB/s
@ronzillia
Copy link

@ronzillia ronzillia commented Mar 23, 2018

this can be resolved if you do conda install setuptools
I am using linux

jaimergp added a commit to insilichem/tangram that referenced this issue Apr 12, 2018
dorukozturk pushed a commit to NeurodataWithoutBorders/pynwb that referenced this issue Aug 17, 2018
dorukozturk pushed a commit to NeurodataWithoutBorders/pynwb that referenced this issue Aug 17, 2018
KEggensperger pushed a commit to automl/HPOlib1.5 that referenced this issue Sep 13, 2018
KEggensperger pushed a commit to automl/HPOlib1.5 that referenced this issue Sep 13, 2018
cwmeijer added a commit to NLeSC/mcfly that referenced this issue Aug 19, 2019
This should fix a specific build failure (to all current failures unfortunately)  
Add --ignore-installed in next line because of failures like in
  https://ci.appveyor.com/project/NLeSC/mcfly/builds/26750694/job/1klik02e6464okfc
  for which the used fix is suggested here: ContinuumIO/anaconda-issues#542.
saimn added a commit to GeminiDRSoftware/DRAGONS that referenced this issue Feb 4, 2020
Conda's astroquery depends on keyring which comes with secretstorage
v2.3 but keyring's metadata declares a dependency on secretstorage>3.
And then pip tries to update secretstorage and fails because of an old
issue between pip and conda
(ContinuumIO/anaconda-issues#542).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet