Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

pip freeze fails if there is a .git directory #58

Closed
vbabiy opened this Issue · 26 comments
@vbabiy

Using pip 0.7.2 from Ubuntu 10.10.
Using a standard TurboGears 2.1 installation (see http://turbogears.org/2.1/docs/main/DownloadInstall.html and http://turbogears.org/2.1/docs/main/AltInstall.html#pipinstall), after the paster quickstart, cd into my projects directory and the setup.py develop, I created a Git repo with git init.

Now, pip freeze fails with:

% pip freeze
Command /usr/bin/git config remote.origin.url failed with error code 1
Storing complete log in /home/niklas/.pip/pip.log

If I move the .git directory away (e.g to the parent folder), pip freeze works.

/home/niklas/.pip/pip.log:

/home/niklas/src/feedboard/tg2env/bin/pip run on Tue Nov 30 19:47:40 2010
Found command 'git' at '/usr/bin/git'
Running command /usr/bin/git config remote.origin.url
Command /usr/bin/git config remote.origin.url failed with error code 1
Exception information:
Traceback (most recent call last):
File "/home/niklas/src/feedboard/tg2env/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/basecommand.py", line 120, in main
self.run(options, args)
File "/home/niklas/src/feedboard/tg2env/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/commands/freeze.py", line 67, in run
req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
File "/home/niklas/src/feedboard/tg2env/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/init.py", line 118, in from_dist
req = get_src_requirement(dist, location, find_tags)
File "/home/niklas/src/feedboard/tg2env/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/vcs/init.py", line 224, in get_src_requirement
return version_control().get_src_requirement(dist, location, find_tags)
File "/home/niklas/src/feedboard/tg2env/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/vcs/git.py", line 148, in get_src_requirement
repo = self.get_url(location)
File "/home/niklas/src/feedboard/tg2env/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/vcs/git.py", line 113, in get_url
show_stdout=False, cwd=location)
File "/home/niklas/src/feedboard/tg2env/lib/python2.6/site-packages/pip-0.7.2-py2.6.egg/pip/init.py", line 226, in call_subprocess
% (command_desc, proc.returncode))
InstallationError: Command /usr/bin/git config remote.origin.url failed with error code 1


@vbabiy

Confirming the error. Same happens for me with TG2.1.


Original Comment By: Anonymous
@vbabiy

I have tried to reproduce it using 0.8.2 and 0.7.2 versions using a Mac OS X,
and did not have problems here.

All my output is save here: http://paste.pocoo.org/show/316040/

Can someone paste a step-by-step showing versions and etc?

Cheers, Hugo.


Original Comment By: Hugo Lopes Tavares
@nisavid

this issue probably only affects git projects that don't have a remote.origin.url. this includes any git projects that originate on the local machine (which were not cloned from a remote repository) and any git-svn projects (since git-svn does not use git remotes).

it is easily reproducible doing pip install -e . in a project checked out from git-svn or in a project whose git repository was initialized locally, then trying pip freeze. you can see a minimal example of the latter here: http://pastebin.com/Jr1VNkDU

@slinkp

But then why does the pip command fail, when the same git command works fine when i run it manually?
http://pastebin.com/errheWeS

@nisavid

slinkp, probably because the error is produced when pip is inspecting a different project than the one that you tested. try it with a fresh virtualenv with --no-site-packages, and do pip install -e only on the openblock project. my guess is that a subsequent pip freeze will work.

to be certain, on a virtualenv that causes pip freeze to fail, i think you can run it with a debugger and inspect the value of location at pip/vcs/git.py:117.

@slinkp

nisavid, correct! it turns out the failure was on another test package, which exists only in a local repository.

I'm trying to think of a good way that the error message could be more helpful, since it provided no clues.
Maybe call_subprocess() should, on errors, also log the cwd and extra_environ args?

@aptivate

I'm experiencing this error and I don't understand it. I don't see how it could be a different repo, as there's only one in my Python environment, and I did do a clone so I should have a remote.origin.url. The same command works fine from the command line.

I'm not even doing a pip freeze, just pip install -r (a file with some packages, one of which is from Git). This is using msysgit on Windows which may be the cause of the problem.

This is what I'm getting:

D:\inetpub\wwwroot\idsapi-staging\django\idsapi>python manage.py update_ve
Requirement already satisfied (use --upgrade to upgrade): Django==1.3 in c:\python26\lib\site-packages (from -r D:\inetpub\wwwroot\idsapi-staging\deploy\pip_packages.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): mysql-python==1.2.3 in c:\users\aptivate\appdata\roaming\python\python26\site-packages (from -r D:\inetpub\wwwroot\idsapi-staging\deploy\pip_packages.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): httplib2==0.6.0 in c:\python26\lib\site-packages (from -r D:\inetpub\wwwroot\idsapi-staging\deploy\pip_packages.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): sunburnt==0.5 in c:\python26\lib\site-packages (from -r D:\inetpub\wwwroot\idsapi-staging\deploy\pip_packages.txt (line 4))
Obtaining djangorestframework from git+https://github.com/tomchristie/django-rest-framework.git#egg=djangorestframework (from -r D:\inetpub\wwwroot\idsapi-staging\deploy\pip_packages.txt (line 5))
Command "C:\Program Files (x86)\Git\cmd\git.cmd" config remote.origin.url failed with error code 1
Storing complete log in C:\Users\aptivate\AppData\Roaming\pip\pip.log

Any ideas what could be the problem here? Thanks in advance, and sorry if it's off-topic.

@nisavid

aptivate, you may have found another way to reproduce this issue. check the pip.log referenced in the error message for clues. if that doesn't help, find out specifically which Git project is triggering the error using a debugger or by throwing a print(location) into pip/vcs/git.py (see my comment above). once you determine the location, change to its directory and check the output of git remote -v show.

@rockyburt

Same issue... I went in and changed lines 114-118 of pip/vcs/git.py to:

def get_url(self, location):                                                                                                                                    
    url = call_subprocess(                                                                                                                                      
        [self.cmd, 'config', 'remote.origin.url'],                                                                                                              
        show_stdout=False, cwd=location, raise_on_returncode=False)                                                                                             
    return (url or '').strip()

It works for me now besides warning it can't find the vcs url.

@akaihola

This also happens if any of the packages installed with setup.py develop have .git/ but no remote.origin.url. Reproducible like this:

mkdir /tmp/freezefail ; cd /tmp/freezefail
virtualenv --distribute .
. bin/activate
cd /home/me/mypackage  # a python package with a setuptools setup.py
rm -rf .git ; git init
python setup.py develop
cd /tmp/freezefail
pip freeze
@piotr-dobrogost

Is there any reason patch suggested by @rockyburt above has not been applied?

@carljm
Owner

@rockyburt's suggested patch looks ok to me. If someone wants to move it along, make it a pull request with a test demonstrating the current failure.

@ndevenish

I just had this same issue, and @rockyburt's patch worked fine. The pip log didn't seem to identify the repository it was failing on either, it only listed the commit hash.

@akaihola

@ndevenish This fix should make pip identify the repository path: #445

@akaihola

I also had this issue after having sudo python setup.py developed a package globally from a local repository where the remote repository was not called "origin".

Is there a way to detect the name of the remote repository instead of using "origin" as a hard-coded value?

@carljm
Owner

@akaihola How would you propose that work? There is no such concept as the remote repository in git, a clone can have many remotes, and all are (AFAIK) treated equally. In order for pip's support to work, however, there must be a single remote repository identified; using "origin" seems as reasonable as anything.

I suppose the one thing that would be possible would be in the special case that a clone has only a single remote, use that one regardless of its name. I'm not sure that this edge case is worth the extra complication in the git backend code, which has already been quite troublesome maintenance-wise. In almost all cases, clones do have a remote named "origin" if they have any remotes at all.

@ndevenish

It's probably better keeping it with the simple case, but I really doubt that it should be a sudden-termination issue in this case. What does it need to do with the remote origin url that is needed for calculating the current version number?

@carljm
Owner

@ndevenish Freeze is trying to create a requirement line that would reliably reproduce what you have installed, anywhere. That's the purpose of the freeze command. In the case of a git editable, this means it has to have a git URL that the commit you have cloned can be cloned from. What would you suggest freeze output in the case where it can't figure out what that URL should be?

I guess I'd be open to a pull request that would cause it to output <unknown repo URL> or some such - that wouldn't be a working requirements file, but it'd be arguably better than a crash.

@nisavid

@carljm said:

@ndevenish Freeze is trying to create a requirement line that would reliably reproduce what you have installed, anywhere. That's the purpose of the freeze command. In the case of a git editable, this means it has to have a git URL that the commit you have cloned can be cloned from. What would you suggest freeze output in the case where it can't figure out what that URL should be?

the same thing it outputs in all other such cases where it lacks the requisite information--the project name and version. why should a Git-versioned project be treated differently?

when i perform an editable installation of a project from a source tree that isn't version-controlled, pip doesn't say <unknown source>. it just lists the name and version.

if i install a project from a custom cheeseshop (not PyPI), pip doesn't know that. pip freeze doesn't then tell me <unknown cheeseshop>. it just lists the name and version.

but if i do a git init in my non-version-controlled project's source tree, then the behavior of pip freeze suddenly changes, even in the case of @carljm's <unknown repository> suggestion. why should this be the case? why not just list the name and version?

i agree with the goal that @carljm stated for pip freeze--to try to create a requirement line that reliably reproduces the installation. but if it can't infer enough information about the installation to produce that, then i don't think it should pollute the output with typically useless unknown text.

perhaps there's a situation where the unknown text wouldn't be useless. for example, maybe some developer is trying to use pip freeze in a situation where it's critical to be able to reproduce the installation exactly. in such a situation, not having enough information would be treated as an error by that developer. in this situation, it would be handy if pip freeze had an option (e.g. --exact) that did in fact write the appropriate unknown text in all of the cases that i listed above. but that's a separate issue, and i think that this usage scenario is less common than the typical one, wherein a user is simply trying to get a list of package names and versions.

returning to the current problem, here's how i see it. we're dealing with an editable installation of a project that is in a local directory that happens to be version-controlled with Git. here are some interesting states in which that directory's contents could be:

1.1. the project was initialized locally. it has no remotes.
1.2. like 1.1, but the project has remotes; however, HEAD points to a local commit that is unrelated to the remotes.
1.3. the project was initialized locally. it has a remote, 'amy'. HEAD points to some commit on 'amy'. there are no staged or unstaged changes.
1.4. like 1.3, but HEAD points to a commit that is a descendant of some commit on 'amy'.
1.5. like 1.3, but there are some staged changes and some other unstaged changes.
2.1. the project was cloned from elsewhere, so it has an 'origin' remote. HEAD points to some commit on 'origin'. there are no staged or unstaged changes.
2.2. like 2.1, but HEAD points to a commit that is a descendant of some commit on 'origin'.
2.3. like 2.1, but there are some staged changes and some other unstaged changes.
3.1. the project was cloned from elsewhere, so it has an 'origin' remote. it also has another remote, 'amy'. HEAD points to some branch on 'amy'. there are no staged or unstaged changes.
3.2. like 3.1, but HEAD points to a commit that is a descendant of some commit on 'origin'.
3.3. like 3.1, but there are some staged changes and some other unstaged changes.

in this list, 1.3, 2.1, and 3.1 are the only cases that afford a requirement line that reliably reproduces the installation. in all of these cases, pip freeze should provide the corresponding repo+commit URI. note that the commit should not be a branch or tag name if the installation should be exactly reproducible.

if pip freeze provided a repo+commit URI for any of the other cases, then that URI would be wrong--if used on any other host (or in some cases on the same host), it would produce an installation that was different from the one inspected by pip freeze. so in all cases except ones like 1.3, 2.1, and 3.1, pip freeze should list only the project name and version (or <unknown repository> if we decide to do that).

the common pattern among 1.3, 2.1, and 3.1 is that HEAD points to a remote commit. in such cases, we need to get the remote's address and commit hash. here's how to do it in Bash, assuming that the working directory is the project's root:

git log -1 --format=format:%d \
    | sed 's/.*(\(.*\)).*/\1/' \
    | sed 's/, /\n/g' \
    | while read refname; do
          reffile=".git/refs/remotes/$refname"
          if [ -f "$reffile" ]; then
              remote="`basename \"$(dirname \"$reffile\")\"`"
              remote_addr="`git remote show \"$remote\" \
                                | grep 'Fetch URL:' \
                                | sed 's/.*Fetch URL: \(.*\)/\1/'`"
              commit="`cat $reffile`"
              echo $remote_addr
              echo $commit
              break
          fi
      done

this script outputs the address and commit in the relevant cases and nothing in other cases. perhaps pip freeze should use a similar technique instead of what it does now.

@carljm
Owner

@nisavid You're totally right, fallback to simple name==version would be sensible behavior. Pull request for that welcome.

@akaihola

Here's an attempt to reproduce the exact use case in which I encountered this:

git svn clone -s http://pywo.googlecode.com/svn PyWO
cd PyWO
git remote add github git@github.com:akaihola/PyWO.git
git push github HEAD
git checkout github/master
git checkout -b akaihola
sudo python setup.py develop

In this case it would make sense for pip freeze to yield

-e git://github.com/akaihola/PyWO.git@35a9e83f94df0323d8bfc46f633b69281e1ee8f1#egg=PyWO-dev

(which it does if I rename the remote to origin).

Also, if I now do

git remote add otherguy git@github.com:otherguy/PyWO.git
git checkout otherguy/master
git checkout -b otherguy

(without repeating python setup.py develop or python setup.py install), then pip freeze should output

-e git://github.com/otherguy/PyWO.git@<SHA1-of-otherguy-repo-HEAD>#egg=PyWO-dev
@carljm
Owner

@akaihola It looks like you're proposing that pip should make use of tracking-branch info to determine which remote's url to output. This makes sense on the surface; I'd have to see what the code for that looks like in a pull request. It's a pretty significant change to the logic of the git backend, I think, which is a bit worrying since the backend has already been quite difficult to maintain.

@akaihola

@carljm When dealing with project dependencies, I probably would make sure to use origin as remote names and not have this problem at all. Also, pip freeze -l would prevent surprises from global packages.

In the case I described, however, I was interested in what versions of packages I had installed globally, and got pip to crash because of my past experiments with the PyWO package. In this case it would suffice if pip mentioned the unexpected state of that package.

Maybe it would be best to output information about the problematic package as commented-out lines with a warning on stderr? The output would still work as input for pip install -r, albeit omitting one package.

For example:

$ pip freeze
[...]
PIL==1.1.7
## Note: the following package was installed from /home/akaihola/PyWO
## but its remote origin can't be determined:
#PyWO
Pygments==1.4
[...]

Warning: The remote origin for some installed packages could not be determined
and they have been omitted from the listing.
@rory

Having been struck by this bug today, I've made a simple work around, which will put in a simple name==version for the offending repo. See pull request #489

@carljm
Owner

Fixed with merge of #489, thanks @rory

@carljm carljm closed this
@reubano

I'm getting the following error when there is no origin

Error when trying to get requirement for VCS system Command /opt/local/bin/git config remote.origin.url failed with error code 1 in /Users/reubano/Documents/Projects/lego, falling back to uneditable format
Could not determine repository location of /Users/reubano/Documents/Projects/lego
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.