Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Do not run `egg_info` to each package in requirements list before installing the previous packages #25

Closed
vbabiy opened this Issue Mar 15, 2011 · 22 comments

Comments

Projects
None yet
Contributor

vbabiy commented Mar 15, 2011

There are tons of packages using setup.py wrongly, and if we have a requirements file like:

# requirements.txt

numpy
scipy

What pip does is to download both, run python setup.py egg_info to numpy, and then python setup.py egg_info to scipy.

The problem is that scipy's setup.py tries to import numpy, which breaks the installation, because there is no numpy installed yet.

I would suggest to download each package and then instead of running egg_info, running install, but I am not seeing the drawbacks now.

Could you please give suggestions here?

PS.: This issue is a split of <<issue 178>> - Non-alphabetical installation of requirements.


Member

hltbra commented Apr 28, 2011

There was a new report of this issue. The issue is #272.

Contributor

carljm commented Apr 28, 2011

The suggestion made here (installing each package fully before moving on to the next package) is not an option. One of the core advantages of pip over easy_install is "look before you leap" - pip ensures that all dependencies can be found, unpacked, and have working setup.py before it installs any of them. This avoids leaving the system in a half-broken state when an error is hit after three-quarters of the requirements have already been installed.

This does cause an issue when one package depends on another package in order to even run its setup.py. Really, this is a broken setup.py, and should be fixed in scipy.

For a workaround in pip, the idea presented in #272 is the only one I've seen so far that I think might be an ok option -- allowing some kind of per-line marker in a requirements file that means "install this in a separate first round of installs". I'm not sure I like this, because it feels like going down the road of turning requirements files into a mini-scripting-language for pip, and I'm not sure we want to go that way. But if someone actually turned this proposal into a patch, I'd be interested in seeing how it looks.

Member

hltbra commented Apr 28, 2011

Hi Carl,

I agree with not adding this kind of behavior to pip. Now I see if we do it or stuff like this, we are not fixing the right place, but only hiding the real problem.

Maybe SciPy folks don't know about this problem... It would be good to send them an email talking about this issue.

I would not like to do it these days. Can someone do it?

Thanks for the response ... this makes sense to me. I have raised a ticket on scipy
http://projects.scipy.org/scipy/ticket/1429

In the meantime I simply split my requirements file in two and ran pip twice. Not too arduous!

@carljm: this is not a bug in scipy: scipy setup.py requires numpy, that is numpy is a build dependency of scipy. This problem is inherent to any package which requires another package to build - without pip supporting this, there is nothing we can do in scipy to solve this.

Contributor

carljm commented Apr 29, 2011

Hi David,

scipy setup.py requires numpy, that is numpy is a build dependency of scipy.

The two clauses there are not equivalent. A build dependency is not the same thing as a running-setup.py dependency.
Even if scipy requires numpy to build, it still should not require it unconditionally in setup.py -- it should be possible to get package metadata out of setup.py (which is all pip is doing here) without having numpy available.

That said, I see your point that even if that were fixed, the scipy install would still fail, just later on at the build phase.

The only currently-available means for explicitly expressing a build dependency in Python packaging is setuptools' "setup_requires" keyword, which pip already supports. I understand that using setuptools and setup_requires may not be feasible for scipy, but I'm not sure what else pip can or should do to support this case.

Closing this bug -- it's clear that serial installation is the only answer here, and I'm not convinced that pip's requirements file format should be extended to support multiple serial installation within a single requirements file. What we really need is a standard way to express build dependencies in Python packaging - I'll inquire on the distutils2 mailing list about that.

@carljm carljm closed this Apr 29, 2011

Contributor

carljm commented Apr 29, 2011

Actually, I think if scipy really only had a build dependency on numpy, rather than an importing-setup.py dependency, it would work fine in pip, as long as numpy was listed above scipy in requirements.txt. We don't separate build and install steps, and IIRC current pip versions now process installations in order, so if the egg_info step for scipy could succeed, numpy would be installed before pip tries to build/install scipy.

I don't understand the distinction you make between setup.py dependency vs build dependency. setup.py is used to build scipy, and the setup.py certainly needs to import numpy.distutils for monkey patching to work. This is not particular to scipy, but will be the same for any package requiring distutils extensions provided by 3rd party tools (starting from setuptools, BTW).

Contributor

carljm commented Jun 20, 2011

@cournape - setup.py is used to build scipy, but it is not only used to build scipy. It is also used for much simpler things, such as e.g. discovering the version of scipy via "python setup.py --version". What pip is doing ("python setup.py egg_info") is simply an extended version of that: metadata discovery.

Scipy should not require an external package in order to make its core metadata discoverable via setup.py. The import of numpy.distutils should be conditional on the use of a setup.py command (e.g. build or install) that actually requires it.

merwok commented Jun 20, 2011

Scipy should not require an external package in order to make its core metadata discoverable via setup.py. The import of
numpy.distutils should be conditional on the use of a setup.py command (e.g. build or install) that actually requires it.

Then you would have the setup script look at sys.argv? It would be messy.

Contributor

carljm commented Jun 20, 2011

Then you would have the setup script look at sys.argv? It would be messy.

Yes, it sure would be messy. Packaging is hard - this is why we are
moving to static metadata in d2 :-) But it's better than having a
package which can't even construct its own metadata without external
dependencies. There's no reasonable workaround pip can be expected to
make in this case.

Someone can try to add more hacks to our distutils extensions if they want to support this.

This (and many other things) will go away once we use our own packaging infrastructure in scipy, so I am not really interested in adding more workaround for distutils limitations.

Would it not be possible to have an option to make pip install packages sequentially for problematic cases? Note that I'm not suggesting that it be the default behavior, but more the equivalent of --force or something like that that only people who know what they are doing will use?

Contributor

carljm commented Sep 4, 2012

@astrofrog Not inherently opposed to that, feel free to work on an implementation and see what it looks like.

@qwcode qwcode added a commit that referenced this issue Feb 21, 2013

@qwcode qwcode Merge pull request #25 from pfmoore/wheel_priority
Give wheels priority over sdists when installing
28ffb1b

FYI: Just proposed a change in setup.py in scipy: scipy/scipy#453
Hopefully they will accept it. For now using my branch.
Without this fix, it's really annoying to work with tox, when scipy is in requirements.

wrought commented May 3, 2013

What about a more helpful error response that recommends to manually install dependencies and then run through requirements.txt a la http://stackoverflow.com/a/11015904/1445241

mattsenate: To show this message pip will need to parse ImportError exception output from python setup.py egg_info command it executes - this may lead to suppressing issue of misspelled module name and some other caveats.

If the world was perfect - everybody would not import external modules if egg_info command is executed.

bjodah commented Dec 25, 2013

I want to second what @wrought said. I got bitten here - maybe you could just print a hint that this might be the issue and re-raise the exception?

EDIT: is it enough to check for 'build', 'build_ext', 'install' in sys.argv and only for these cases import from dependencies?

@bjodah It's better to handle egg_info, clean and help commands as special case (at least this is what has been done in scipy (https://github.com/scipy/scipy/blob/master/setup.py) and some other libraries that depend on numpy):

if '--help' in sys.argv[1:] or \
    sys.argv[1] in ('--help-commands', 'egg_info', 'clean', '--version'):
    ...
else:
    import numpy
    ...

@jsvine jsvine added a commit to jsvine/virtualenv-recipes that referenced this issue Apr 3, 2014

@jsvine jsvine Change recipe-installation instructions
Per problems with `pip install -r` when installing scipy and others.

c.f. pypa/pip#25
378a1dd

griesmey commented Mar 3, 2016

Was this ever fixed? I'm having the same issue when trying to install scipy from the requirements file using wheel...

shindere commented Mar 4, 2016

Robert Griesmeyer (2016/03/03 15:56 -0800):

Was this ever fixed? I'm having the same issue when trying to install
scipy from the requirements file using wheel...

No idea, sorry.

Contributor

xavfernandez commented Mar 4, 2016

I don't think this needs fixing on pip's end.
Package should declare their setup_requires and/or users should either provide them or expect setuptools to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment