--process-dependency-links broken

[EvaSDK]

Hi,

in the company I work for, we have lots of python packages depending on a forked version of celery (https://github.com/EvaSDK/celery/tree/add-riak-backend).

Some of these packages depend on a common module which depend on this forked version which is only hosted on git. Let's say we have package A which depends on package B which depends on forked version of celery.

With pip 1.4, having a dependency_links in B was enough to allow installation from our internal git copy (which is verified and secure) when running "pip install -e ." on project A.

pip 1.5 breaks that, I tried all possible combination of allow-external, allow-unverified and process-dependency-links to no avail.

I wish I could only use released software, but as you may have understood by now, this is just not possible because of constraints from upstream work. Do you have a recommended solution for that case ?

[kdeldycke]

👍

[dstufft]

Can you give me a package and a command you ran to reproduce this error?

[EvaSDK]

The packages are not public but I can write an abbreviated example:

In package A setup.py:

setup(
  name='modulea',
  requires=['celery >=3.2, ==dev'],
  dependency_links=['git+ssh://git@github.com/EvaSDK/celery.git@v3.1.6+feature#egg=celery-dev'],
)

and in package B setup.py:

setup(
  name='moduleb',
  requires=['modulea'],
)

Then running pip install --process-dependency-links -e . on package A works, but not on package B.

[qwcode]

@EvaSDK don't you mean install_requires? you say "but not on package B"? what's the full context of the pip command in that case? what options?

[EvaSDK]

yeah install_requires, sorry for typo.

For the context of the pip command:

$ cd projectA
projectA$ pip install --process-dependency-links -e .

and that works (properly fetches the git repository as long as the is no released version of celery >=3.2.
For project B:

$ cd projectB
projectB$ pip install --process-dependency-links -e .

and that does not work, it tries to install dependencies of project A but fails to select a version of celery because it does not try to fetch it from git.

[qwcode]

$ cd projectB
$ pip install --process-dependency-links -e .

this command doesn't know about about the abstract modulea dependency.
modulea would need to be packaged locally in a directory and referenced with --find-links=local/dir/

[EvaSDK]

But this information is available in egg-info in the dependency_links.txt file. There shall be no difference between the installation of already downloaded archives and those coming from pypi.

[qwcode]

ok, I didn't follow that the 2 commands were being run back to back in the same environment. got it. given that, let me look at it again later today.

[qwcode]

I'm having trouble with your example. I don't see v3.1.6+feature in your clone? maybe post a full working example somewhere that works in 1.4 (like you say), but not in 1.5. the changes in 1.5 were pretty minor. they just make the use of dependency links optional based on a parameter.

but considering the uncertain future of dependency_links, I have to recommend using the logic of requirements files to handle your fork/override needs.

e.g. take a requirements file like so:

-e git+https://github.com/EvaSDK/celery@fork-branch@egg=celery
someproject==1.3  # install_requires=['celery']

you'll end up getting the fork-branch version of celery, not celery from pypi.

someproject could also be an editable project like so

-e git+https://github.com/EvaSDK/celery@fork-branch@egg=celery
-e ./someproject  # install_requires=['celery']

BUT you can't have multiple editable path projects referring to each other abstractly. pip doesn't sort it out. only one top-level editable project that's path based.

so, NOT this, it won't work

-e git+https://github.com/EvaSDK/celery@fork-branch@egg=celery
-e ./someproject  # install_requires=['celery']
-e ./anotherproject  # install_requires=['someproject']

to make this, work, someproject, could be in vcs (with #egg= syntax), or it would need to be packaged, and have pip find it with --find-links.

[dstufft]

I'm unable to reproduce this. Unless someone can package up a reproducible test that works on pip 1.4.1 but doesn't on 1.5 I'm going to vote that we close this issue.

[dstufft]

Closing this, if you're able to provide steps or similar that we can reproduce please recomment and we can open it again.

[EvaSDK]

I'll look into providing a complete example. Thanks for your time.

[Ivoz]

If you wanted, you could run your own package repository, with your own versions of packages.

e.g, run simplepypi or mypypi on a private work server and point pip to it when installing with -i. You can also build wheels in a "wheelhouse" (folder) and install from that, which works from anyone's local filesystem. The pip wheel command makes this very easy.

Then build your own celery fork and put it on your own index.

[bittner]

I'm not sure whether this is 100% the same problem, but I can at least provide a reproducible example. My project installs fine (in version v0.1) with pip 1.4.1, but pip 1.5.4 can't find a version to install:

(myvirtenv)$ pip install django-organice
...
Downloading/unpacking django-cms==2.4.3-support (from django-organice)
  Could not find a version that satisfies the requirement django-cms==2.4.3-support (from django-organice) (from versions: 2.4.3, 3.0, 3.0.1, 3.0.2, 2.0.1, 2.0.2, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.2.1, 2.2, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 3.0.1, 3.0.2, 3.0)
Cleaning up...
No distributions matching the version for django-cms==2.4.3-support (from django-organice)
Storing debug log for failure in /home/peter/.pip/pip.log

The setup refers to two Git repositories using dependency_links= (see https://github.com/bittner/django-organice/blob/v0.1/setup.py#L70).

I'm unable to make this work with pip 1.5. Adding --allow-external and --allow-unverified on the command line doesn't change a thing. The logfile doesn't help either:

Downloading/unpacking django-cms==2.4.3-support (from django-organice)
  Getting page https://pypi.python.org/simple/django-cms/
  URLs to search for versions for django-cms==2.4.3-support (from django-organice):
  * https://pypi.python.org/simple/django-cms/2.4.3-support
  * https://pypi.python.org/simple/django-cms/
  Getting page https://pypi.python.org/simple/django-cms/2.4.3-support
  Could not fetch URL https://pypi.python.org/simple/django-cms/2.4.3-support: 404 Client Error: Not Found
  Will skip URL https://pypi.python.org/simple/django-cms/2.4.3-support when looking for download links for django-cms==2.4.3-support (from django-organice)
  Analyzing links from page https://pypi.python.org/simple/django-cms/
  Found link https://pypi.python.org/packages/2.7/d/django-cms/...
...
  Ignoring link https://pypi.python.org/packages/2.7/d/django-cms/...
...
  Could not find a version that satisfies the requirement django-cms==2.4.3-support (from django-organice) (from versions: 2.4.3, 3.0, 3.0.1, 3.0.2, 2.0.1, 2.0.2, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.2.1, 2.2, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 3.0.1, 3.0.2, 3.0)
Cleaning up...
  Removing temporary dir /home/peter/.virtualenvs/myvirtenv/build...
No distributions matching the version for django-cms==2.4.3-support (from django-organice)
...

Looks like pip 1.5 is simply ignoring my dependency_links. What's wrong here?

[Ivoz]

@bittner did you add --process-dependency-links?

[bittner]

Aha, thanks, that works.

So, in order to cleanly migrate to pip 1.5+ (and make the plain pip install mypackage work again) I have to get rid of PIL and the dependency_links in the setup, correct? In other words, I have to create helper packages on PyPI for everything I currently install directly off a Git repository. Kind of annoying... )-:

[Ivoz]

You probably wish to read the extended discussion of the issue at #1519

Specifically in regards to PIL I'd recommend projects switching to Pillow whenever possible, which is not externally hosted.