Error with pip requirement files and URL packages #43

Closed
vbabiy opened this Issue Mar 15, 2011 · 15 comments

Comments

Projects
None yet
5 participants
Contributor

vbabiy commented Mar 15, 2011

When you put a URL in a requirements file, pip doesn't know what package it is dealing with, and tells you to add #egg=PackageName to the URL. However, even when this is done, it still doesn't parse out the package name and gives the same error.

To reproduce, put the following URL in a requirements.txt and run pip freeze -r requirements.txt:

http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz#egg=Django}}}
{{{http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz#egg=Django

Contributor

vbabiy commented Mar 15, 2011

  • Changed content.

Original Comment By: Thomas Johansson
Contributor

vbabiy commented Mar 15, 2011

I can duplicate this, looks like a bug to me.


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

If I drop that URL into a requirements file and run:

$ pip install -r req.txt

Pip downloads the source distribution and installs it. Is the desired result
to have the same source URL recorded when running "pip freeze -r"?

Unless pip verifies the link, shouldn't pip just include the URL as-is as part
of the the output?

Is the goal to parse the URL, extract the distribution name and version, then
add something like "Django==1.2-rc-1" to the list of requirements?


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

The follow changeset/patch allows the following to work:

# cat req.txt

http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz



# /opt/OpenPython-2.7.1/bin/pip freeze -r req.txt

http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz

## The following requirements were added by pip --freeze:

Distutils2==1.0a3

Pygments==1.4

bpython==0.9.7.1

distribute==0.6.14

nose==1.0.0

virtualenv==1.5.1

wsgiref==0.1.2

yamlconfig==0.3

Following the logic for how pip inspects each line in the incoming
requirements file, I added a simple check to see if the line starts with
'http'. This could be expanded to cover other valid direct URLs supported by
pip.

Link to changeset

https://bitbucket.org/khightower/pip/changeset/dd1551ceade0


Original Comment By: Kelsey Hightower
Member

Ivoz commented Mar 27, 2014

Current behaviour:

 tmp-77f042ce53b5347 ivo·ivosung ~/.virtualenvs/tmp-77f042ce53b5347 » cat req.txt 
http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz#egg=Django
 tmp-77f042ce53b5347 ivo·ivosung ~/.virtualenvs/tmp-77f042ce53b5347 » pip freeze -r req.txt 
Django==1.2-rc-1
## The following requirements were added by pip --freeze:
wsgiref==0.1.2

At the moment freeze is not rewriting the entire url, which would result in a failed install if this package was not on the index used to install packages.

Member

Ivoz commented Mar 27, 2014

@pypa does this seem like behaviour that should be fixed?

Owner

pfmoore commented Mar 27, 2014

@Ivoz my first impression is that it could be pretty hard to fix, as the original URL won't be recorded anywhere in the package metadata (I haven't checked this - correct me if I'm wrong). So a "proper" fix probably involves adding new package metadata somewhere to record this information. (Which probably implies it needs adding to the Metadata 2.0 spec).

Assuming that's the case, and given that it's easy enough to manually add the URL to the requirements file, I don't think it's worth fixing. But I don't use freeze at all myself - how much of an issue is this in practice to people who rely on freeze?

Owner

dstufft commented Mar 27, 2014

I had a half written PR to record where something was insalled from before, but as I reemmber it @pfmoore is correct in that once it's been installed we no longer have that information to completely recreate it.

Owner

dstufft commented Mar 27, 2014

Found it, my PR was #453 which someone else took over and submitted as #507 which is still open.

Member

Ivoz commented Mar 27, 2014

@pfmoore my initial impression is that at least we could keep the url in the requirements.txt given in the freeze command, which is what is currently being lost, not the loftier goal of always remembering urls of all installed packages.

Contributor

qwcode commented Mar 27, 2014

When you put a URL in a requirements file,
pip doesn't know what package it is dealing with,
and tells you to add #egg=PackageName to the URL

btw, it's no longer true that we require #egg= for pkg urls, but it is true, that pip doesn't try to parse the name/version up front from the archive name, which it could in most case, but instead puts it thru this "unnamed" code path in the req package (search for "unnamed" in the code). see #804 (comment) . I'm going to break the #804 comment into it's own issue.

as for this issue, I say close. technically, it's about an error that doesn't occur anymore. let the PR @dstuff reference be about the possibility of storing the URL for pip freeze

Owner

pfmoore commented Mar 27, 2014

@Ivoz ah, I see - you're saying that if there's a URL currently in requirements.txt that matches up with an installed distribution, then use that unchanged rather than writing out the normal requirement. Yes that's possible, although it's arguably a behaviour change - this is what the help currently says for -r:

  -r, --requirement <file>    Use the order in the given requirements file and
                              it's comments when generating output.

But that's a minor nitpick, really just a reminder to update the help if this gets implemented. Also, it does mean that pip freeze and pip freeze -r req.txt could give different results, which may be an issue (although the point of the issue is that it's changing 2 wrong but consistent results to one right and one wrong, so maybe that's still an improvement!)

I guess I'm happy if people who actually use the feature would find it of use.

Member

Ivoz commented Mar 27, 2014

Basically the concerned use case would be that pip freeze -r requirements.txt > requirements.txt would be deleting information (i.e url specified packages)

Contributor

qwcode commented Mar 27, 2014

btw, it's no longer true that we require #egg= for pkg urls

oh, when using pip freeze -r requirements.txt, it does spit out errors like this when's there's no fragment at all.

Skipping line because it's not clear what it would install: http://media.djangoproject.com/releases/1.2/Django-1.2-rc-1.tar.gz
  (add #egg=PackageName to the URL to avoid this warning)

but when there is a fragment, no error, which is counter to what the issue reports.

Member

Ivoz commented Mar 27, 2014

Alright, see #1690

@Ivoz Ivoz closed this Mar 27, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment