New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't get setup_requires dependencies without talking to PyPI #410

Closed
slacy opened this Issue Dec 17, 2011 · 16 comments

Comments

Projects
None yet
6 participants
@slacy

slacy commented Dec 17, 2011

Here's the sequence to reproduce this problem. When pip is installing the dependencies for django-jenkins, it goes out and talks to pypi.python.org.

$ virtualenv env
$ . ./env/bin/activate
$ mkdir ./sdists
$ pip install -d ./sdists django-appconf

Then, with NETWORKING DISABLED, run:

$ pip install --find-links=file:///$(pwd)/sdists --no-index --index-url=file:///dev/null django-appconf

The install will fail with a message saying:

distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('versiontools>=1.6')

But, if you enable networking and re-run the exact same command, then it will succeed with:

"Installed /home/slacy/tmp/env/build/django-appconf/versiontools-1.8.2-py2.7.egg"

I'm fairly certain that it goes out to pypi.python.org to figure this out. Earlier today, when pypi.python.org wasn't responding, this would print out some error messages that indicated as such, but now that it's back online, it's harder to see what's going on (and adding logs/verbose doesn't show much)

If there's anything more you need me to do to debug this, let me know.

@carljm

This comment has been minimized.

Contributor

carljm commented Dec 17, 2011

This is a bit more subtle - pip handles normal runtime dependencies (install_requires) correctly (AFAIK), but django-appconf has a setuptools setup_requires dependency; meaning a dependency that isn't needed for the package at runtime, but has to be satisfied in order to even create the package metadata. Pip has no special handling of setup_requires dependencies, and it would be quite tricky for it to do so, since pip gets metadata about a package it has just downloaded (like, the package's dependencies) by running setup.py egg_info, and that in and of itself triggers setuptools to download the setup_requires dependencies!

I'm not sure I see a good solution here, other than to discourage use of setup_requires (@jezdez, take note! :P) - the root of the problem is that setup_requires is kind of an odd recursive feature of setuptools in the first place.

It might be possible to solve if there's a way to set environment vars or pass flags to setup.py egg_info to tell setuptools to use a different index (or none at all) and/or add find-links when downloading setup_requires dependencies. I would guess this isn't even possible, in which case it would require patching distribute, and then some code in pip to check if the right version of distribute is available and only pass the flags in that case.

@carljm

This comment has been minimized.

Contributor

carljm commented Dec 17, 2011

One other workaround on the package-author side that I think would also solve this problem is if the package author actually adds an egg file for the setup_requires dependency to the sdist of their package, at the root right next to setup.py (so for instance, django-appconf would come bundled with a versiontools egg in the sdist). That's where setuptools puts the egg when it downloads it, and I haven't tested recently but I think if it's already there setuptools never goes looking for it in the first place.

@slacy

This comment has been minimized.

slacy commented Dec 17, 2011

I think they are including the .egg in their package, but for some reason it's going out to pypi first to check versions or something.

@carljm

This comment has been minimized.

Contributor

carljm commented Dec 17, 2011

I just checked and the sdist for django-appconf definitely does not have a versiontools egg in it. And I also tested - once the egg is placed there, I can run setup.py egg_info on the unpacked django-appconf sdist with networking disabled. So I think that workaround in django-appconf would fix it.

@carljm

This comment has been minimized.

Contributor

carljm commented Dec 17, 2011

You might have thought it was included because you looked in an unpacked sdist after setup.py egg_info had already been run on it? In that case the egg would be there.

@jezdez

This comment has been minimized.

Contributor

jezdez commented Dec 17, 2011

Sigh, I think I'll start to not require versiontools anymore, this is exactly the reason why I really don't enjoy the odd behaviours of setuptools sometimes. Thanks for letting me know!

@zyga

This comment has been minimized.

zyga commented Dec 17, 2011

jezdes, that would be unfortunate, is there anything I can do to help you fix this without throwing the idea of versiontools away?

@zyga

This comment has been minimized.

zyga commented Dec 17, 2011

@jezdez (sorry for mis-typing your name before), I could rewrite versiontools to generate package meta-data on setup time when you sdist upload your code. The, it would not be a setup-time dependency (it could, for example, edit setup.cfg instead)

@jezdez

This comment has been minimized.

Contributor

jezdez commented Feb 4, 2012

@zyga Yeah, I think that would be an interesting approach instead.

@zyga

This comment has been minimized.

zyga commented Feb 4, 2012

@jezdez I'll have a look at what can be done over the weekend.

@zyga

This comment has been minimized.

zyga commented Feb 5, 2012

@jezdez Please have a look at https://code.launchpad.net/~zkrynicki/versiontools/1.9 This branch has a zero-dependency mode. Everything is documented but to keep it short you need to distribute a new helper file along with your release. This allows you to drop the dependency on versiontools (and no, the bundled file is not versiontools ;-) at setup time.

@qwcode

This comment has been minimized.

Contributor

qwcode commented Feb 28, 2013

this solution wasn't mentioned explicitly, but it works (from real experience)

add this to ~/.pydistutils.cfg
(http://docs.python.org/2/install/index.html#location-and-names-of-config-files)

[easy_install]
allow_hosts = ''
find_links = file:///path/to/setup_requires_dists/

since easy_install is what is processing "setup_requires", this tells it to disallow all index hosts, but look in the local find_links.

I'll leave this open until I add a cookbook entry for this to the docs.

I'm not sure we can do anything more than document this for now.

@qwcode

This comment has been minimized.

Contributor

qwcode commented Mar 17, 2013

@qwcode

This comment has been minimized.

Contributor

qwcode commented Aug 13, 2013

the associated setuptools issue that would give pip something to set. https://bitbucket.org/tarek/distribute/issue/338/unable-to-specify-custom-index-for

@piotr-dobrogost

This comment has been minimized.

piotr-dobrogost commented Oct 2, 2014

new cookbook entry
http://www.pip-installer.org/en/latest/cookbook.html#controlling-setup-requires

This now leads to the User Guide where searching for pydistutils does not bring any results.

@qwcode

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment