Can't get setup_requires dependencies without talking to PyPI #410

Closed
slacy opened this Issue Dec 17, 2011 · 16 comments

Comments

Projects
None yet
6 participants

slacy commented Dec 17, 2011

Here's the sequence to reproduce this problem. When pip is installing the dependencies for django-jenkins, it goes out and talks to pypi.python.org.

$ virtualenv env
$ . ./env/bin/activate
$ mkdir ./sdists
$ pip install -d ./sdists django-appconf

Then, with NETWORKING DISABLED, run:

$ pip install --find-links=file:///$(pwd)/sdists --no-index --index-url=file:///dev/null django-appconf

The install will fail with a message saying:

distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('versiontools>=1.6')

But, if you enable networking and re-run the exact same command, then it will succeed with:

"Installed /home/slacy/tmp/env/build/django-appconf/versiontools-1.8.2-py2.7.egg"

I'm fairly certain that it goes out to pypi.python.org to figure this out. Earlier today, when pypi.python.org wasn't responding, this would print out some error messages that indicated as such, but now that it's back online, it's harder to see what's going on (and adding logs/verbose doesn't show much)

If there's anything more you need me to do to debug this, let me know.

Contributor

carljm commented Dec 17, 2011

This is a bit more subtle - pip handles normal runtime dependencies (install_requires) correctly (AFAIK), but django-appconf has a setuptools setup_requires dependency; meaning a dependency that isn't needed for the package at runtime, but has to be satisfied in order to even create the package metadata. Pip has no special handling of setup_requires dependencies, and it would be quite tricky for it to do so, since pip gets metadata about a package it has just downloaded (like, the package's dependencies) by running setup.py egg_info, and that in and of itself triggers setuptools to download the setup_requires dependencies!

I'm not sure I see a good solution here, other than to discourage use of setup_requires (@jezdez, take note! :P) - the root of the problem is that setup_requires is kind of an odd recursive feature of setuptools in the first place.

It might be possible to solve if there's a way to set environment vars or pass flags to setup.py egg_info to tell setuptools to use a different index (or none at all) and/or add find-links when downloading setup_requires dependencies. I would guess this isn't even possible, in which case it would require patching distribute, and then some code in pip to check if the right version of distribute is available and only pass the flags in that case.

Contributor

carljm commented Dec 17, 2011

One other workaround on the package-author side that I think would also solve this problem is if the package author actually adds an egg file for the setup_requires dependency to the sdist of their package, at the root right next to setup.py (so for instance, django-appconf would come bundled with a versiontools egg in the sdist). That's where setuptools puts the egg when it downloads it, and I haven't tested recently but I think if it's already there setuptools never goes looking for it in the first place.

slacy commented Dec 17, 2011

I think they are including the .egg in their package, but for some reason it's going out to pypi first to check versions or something.

Contributor

carljm commented Dec 17, 2011

I just checked and the sdist for django-appconf definitely does not have a versiontools egg in it. And I also tested - once the egg is placed there, I can run setup.py egg_info on the unpacked django-appconf sdist with networking disabled. So I think that workaround in django-appconf would fix it.

Contributor

carljm commented Dec 17, 2011

You might have thought it was included because you looked in an unpacked sdist after setup.py egg_info had already been run on it? In that case the egg would be there.

Contributor

jezdez commented Dec 17, 2011

Sigh, I think I'll start to not require versiontools anymore, this is exactly the reason why I really don't enjoy the odd behaviours of setuptools sometimes. Thanks for letting me know!

zyga commented Dec 17, 2011

jezdes, that would be unfortunate, is there anything I can do to help you fix this without throwing the idea of versiontools away?

zyga commented Dec 17, 2011

@jezdez (sorry for mis-typing your name before), I could rewrite versiontools to generate package meta-data on setup time when you sdist upload your code. The, it would not be a setup-time dependency (it could, for example, edit setup.cfg instead)

Contributor

jezdez commented Feb 4, 2012

@zyga Yeah, I think that would be an interesting approach instead.

zyga commented Feb 4, 2012

@jezdez I'll have a look at what can be done over the weekend.

zyga commented Feb 5, 2012

@jezdez Please have a look at https://code.launchpad.net/~zkrynicki/versiontools/1.9 This branch has a zero-dependency mode. Everything is documented but to keep it short you need to distribute a new helper file along with your release. This allows you to drop the dependency on versiontools (and no, the bundled file is not versiontools ;-) at setup time.

Contributor

qwcode commented Feb 28, 2013

this solution wasn't mentioned explicitly, but it works (from real experience)

add this to ~/.pydistutils.cfg
(http://docs.python.org/2/install/index.html#location-and-names-of-config-files)

[easy_install]
allow_hosts = ''
find_links = file:///path/to/setup_requires_dists/

since easy_install is what is processing "setup_requires", this tells it to disallow all index hosts, but look in the local find_links.

I'll leave this open until I add a cookbook entry for this to the docs.

I'm not sure we can do anything more than document this for now.

qwcode closed this Mar 17, 2013

Contributor

qwcode commented Aug 13, 2013

the associated setuptools issue that would give pip something to set. https://bitbucket.org/tarek/distribute/issue/338/unable-to-specify-custom-index-for

new cookbook entry
http://www.pip-installer.org/en/latest/cookbook.html#controlling-setup-requires

This now leads to the User Guide where searching for pydistutils does not bring any results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment