Error with pip requirement files and URL packages

[vbabiy]

When you put a URL in a requirements file, pip doesn't know what package it is dealing with, and tells you to add #egg=PackageName to the URL. However, even when this is done, it still doesn't parse out the package name and gives the same error.

To reproduce, put the following URL in a requirements.txt and run pip freeze -r requirements.txt:

http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz#egg=Django}}}
{{{http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz#egg=Django

---

• Bitbucket: https://bitbucket.org/ianb/pip/issue/118
• Originally Reported By: Thomas Johansson
• Originally Created At: 2010-05-09 15:10:04

[vbabiy]

• Changed content.

---

Original Comment By: Thomas Johansson

[vbabiy]

I can duplicate this, looks like a bug to me.

---

Original Comment By: Carl Meyer

[vbabiy]

If I drop that URL into a requirements file and run:

$ pip install -r req.txt

Pip downloads the source distribution and installs it. Is the desired result
to have the same source URL recorded when running "pip freeze -r"?

Unless pip verifies the link, shouldn't pip just include the URL as-is as part
of the the output?

Is the goal to parse the URL, extract the distribution name and version, then
add something like "Django==1.2-rc-1" to the list of requirements?

---

Original Comment By: Kelsey Hightower

[vbabiy]

The follow changeset/patch allows the following to work:

# cat req.txt

http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz



# /opt/OpenPython-2.7.1/bin/pip freeze -r req.txt

http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz

## The following requirements were added by pip --freeze:

Distutils2==1.0a3

Pygments==1.4

bpython==0.9.7.1

distribute==0.6.14

nose==1.0.0

virtualenv==1.5.1

wsgiref==0.1.2

yamlconfig==0.3

Following the logic for how pip inspects each line in the incoming
requirements file, I added a simple check to see if the line starts with
'http'. This could be expanded to cover other valid direct URLs supported by
pip.

Link to changeset

https://bitbucket.org/khightower/pip/changeset/dd1551ceade0

---

Original Comment By: Kelsey Hightower

[Ivoz]

Current behaviour:

 tmp-77f042ce53b5347 ivo·ivosung ~/.virtualenvs/tmp-77f042ce53b5347 » cat req.txt 
http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz#egg=Django
 tmp-77f042ce53b5347 ivo·ivosung ~/.virtualenvs/tmp-77f042ce53b5347 » pip freeze -r req.txt 
Django==1.2-rc-1
## The following requirements were added by pip --freeze:
wsgiref==0.1.2

At the moment freeze is not rewriting the entire url, which would result in a failed install if this package was not on the index used to install packages.

[Ivoz]

@pypa does this seem like behaviour that should be fixed?

[pfmoore]

@Ivoz my first impression is that it could be pretty hard to fix, as the original URL won't be recorded anywhere in the package metadata (I haven't checked this - correct me if I'm wrong). So a "proper" fix probably involves adding new package metadata somewhere to record this information. (Which probably implies it needs adding to the Metadata 2.0 spec).

Assuming that's the case, and given that it's easy enough to manually add the URL to the requirements file, I don't think it's worth fixing. But I don't use freeze at all myself - how much of an issue is this in practice to people who rely on freeze?

[dstufft]

I had a half written PR to record where something was insalled from before, but as I reemmber it @pfmoore is correct in that once it's been installed we no longer have that information to completely recreate it.

[dstufft]

Found it, my PR was #453 which someone else took over and submitted as #507 which is still open.

[Ivoz]

@pfmoore my initial impression is that at least we could keep the url in the requirements.txt given in the freeze command, which is what is currently being lost, not the loftier goal of always remembering urls of all installed packages.

[qwcode]

When you put a URL in a requirements file,
pip doesn't know what package it is dealing with,
and tells you to add #egg=PackageName to the URL

btw, it's no longer true that we require #egg= for pkg urls, but it is true, that pip doesn't try to parse the name/version up front from the archive name, which it could in most case, but instead puts it thru this "unnamed" code path in the req package (search for "unnamed" in the code). see #804 (comment) . I'm going to break the #804 comment into it's own issue.

as for this issue, I say close. technically, it's about an error that doesn't occur anymore. let the PR @dstuff reference be about the possibility of storing the URL for pip freeze

[pfmoore]

@Ivoz ah, I see - you're saying that if there's a URL currently in requirements.txt that matches up with an installed distribution, then use that unchanged rather than writing out the normal requirement. Yes that's possible, although it's arguably a behaviour change - this is what the help currently says for -r:

  -r, --requirement <file>    Use the order in the given requirements file and
                              it's comments when generating output.

But that's a minor nitpick, really just a reminder to update the help if this gets implemented. Also, it does mean that pip freeze and pip freeze -r req.txt could give different results, which may be an issue (although the point of the issue is that it's changing 2 wrong but consistent results to one right and one wrong, so maybe that's still an improvement!)

I guess I'm happy if people who actually use the feature would find it of use.

[Ivoz]

Basically the concerned use case would be that pip freeze -r requirements.txt > requirements.txt would be deleting information (i.e url specified packages)

[qwcode]

btw, it's no longer true that we require #egg= for pkg urls

oh, when using pip freeze -r requirements.txt, it does spit out errors like this when's there's no fragment at all.

Skipping line because it's not clear what it would install: http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz
  (add #egg=PackageName to the URL to avoid this warning)

but when there is a fragment, no error, which is counter to what the issue reports.

[Ivoz]

Alright, see #1690