The git push command hangs after upgrade to LFS v1.2.0

[sarxos]

Hi,

We found a problem which does not appear on v1.1.1 but can be reproduced after upgrade to v1.2.0. The problem is that git push hangs when trying to push changes to Gerrit.

Our test stack consists of the following:

• Server:
  • CentOS v6.5,
  • Gerrit (tested with v2.10.2 and v2.12.4),
  • LFS server (https://github.com/git-lfs/lfs-test-server),
• Client:
  • Ubuntu 14.04,
  • Git (tested with v1.9.1 and v2.10.0),
  • Git LFS (tested with v1.1.1, v1.2.0 and v1.4.1).

Working setting (LFS v1.1.1):

$ sudo apt-get install git-lfs=1.1.1

But when switched to LFS v1.2.0 (or newer) the git push hangs:

$ sudo apt-get remove git-lfs
$ sudo apt-get install git-lfs=1.2.0

The .git/config:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = ssh://admin@localhost:29418/swim
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master
[lfs]
        url = http://admin:admin@127.0.0.1:9999

The .gitattributes:

*.psd filter=lfs diff=lfs merge=lfs -text

Output from git push:

$ GIT_TRACE=1 git push origin HEAD:refs/for/master
trace: built-in: git 'push' 'origin' 'HEAD:refs/for/master'
trace: run_command: 'ssh' '-p' '29418' 'admin@localhost' 'git-receive-pack '\''/swim'\'''
trace: run_command: '.git/hooks/pre-push' 'origin' 'ssh://admin@localhost:29418/swim'
trace: exec: 'git-lfs' 'pre-push' 'origin' 'ssh://admin@localhost:29418/swim'
trace: run_command: 'git-lfs' 'pre-push' 'origin' 'ssh://admin@localhost:29418/swim'

The output from ps -ef | grep -i git:

$ ps -ef | grep -i git
sarxos    8107  8075  4 14:43 ?        00:03:25 /usr/lib/virtualbox/VirtualBox --comment GIT_SVN --startvm de1fb427-6afa-40db-ae20-2839fb0294f8 --no-startvm-errormsgbox
sarxos   10735  9773  0 15:53 pts/25   00:00:00 git push origin HEAD:refs/for/master
sarxos   10736 10735  0 15:53 pts/25   00:00:00 ssh -p 29418 admin@localhost git-receive-pack '/swim'
sarxos   10738 10735  0 15:53 pts/25   00:00:00 /bin/sh .git/hooks/pre-push origin ssh://admin@localhost:29418/swim
sarxos   10739 10738  0 15:53 pts/25   00:00:00 git lfs pre-push origin ssh://admin@localhost:29418/swim
sarxos   10740 10739  0 15:53 pts/25   00:00:00 git-lfs pre-push origin ssh://admin@localhost:29418/swim
sarxos   10749 10740  0 15:53 pts/25   00:00:00 [git] <defunct>
sarxos   10750 10740  0 15:53 pts/25   00:00:00 git ls-remote --heads --tags -q origin
sarxos   10751 10750  0 15:53 pts/25   00:00:00 ssh -p 29418 admin@localhost git-upload-pack '/swim'
sarxos   10873 10826  0 16:03 pts/26   00:00:00 grep --color=auto -i git

However, if I do the same without git-lfs configured for this repo, everything works as expected. When I downgrade to LFS v1.1.1, everything works as expected as well.

[ttaylorr]

Hi @sarxos - thanks for the detailed report. Would you mind running the same git push command with an additional environment variable, GIT_CURL_VERBOSE=1, and pasting the output? I have a hunch this could be an issue with the test server, not git-lfs, but I want to make sure before diagnosing further.

Thanks!

[sarxos]

Hi @ttaylorr,

Unfortunately, it fails before any CURL is done. The git push with GIT_CURL_VERBOSE=1 prints nothing. With this in mind I played a little bit more and inspected logs in my Gerrit to see what is happening there and identify the difference. This is how it looks like:

With LFS v1.2.0 (failing scenario):

The git push causes 2 ssh connections to be made to Gerrit, and hangs there:

[2016-09-14 15:00:49,310 +0200] f1124be1 admin a/1 LOGIN FROM 10.0.2.2
[2016-09-14 15:00:49,436 +0200] 110a7faa admin a/1 LOGIN FROM 10.0.2.2

After I kill git push which hanged, in Gerrit log I see the following:

[2016-09-14 15:05:43,617 +0200] 110a7faa admin a/1 LOGOUT
[2016-09-14 15:05:43,618 +0200] f1124be1 admin a/1 LOGOUT
[2016-09-14 15:05:43,618 +0200] 110a7faa admin a/1 git-upload-pack./swim 0ms 294176ms killed
[2016-09-14 15:05:43,619 +0200] f1124be1 admin a/1 git-receive-pack./swim 3ms 294293ms 127

With LFS v.1.1.1 (working scenario):

The git push causes only 1 ssh connection (without git-upload-pack) to be made to Gerrit (which does not hang):

[2016-09-14 15:10:04,121 +0200] 3128e31b admin a/1 LOGIN FROM 10.0.2.2
[2016-09-14 15:10:04,218 +0200] 3128e31b admin a/1 git-receive-pack./swim 1ms 87ms 0
[2016-09-14 15:10:04,239 +0200] 3128e31b admin a/1 LOGOUT

From the above I assume these two commands (git-upload-pack and git-receive-pack) probably deadlocked each other, but I should rather wonder why with LFS v1.2.0 these two commands has been ran in parallel and not one after the other or why there is a git-upload-pack executed when v1.1.1 did only git-receive-pack.

[technoweenie]

@sarxos The only significant change from v1.1.1 to v1.2.0 related to SSH is #1134. LFS only calls git-lfs-authenticate. The git-upload-pack and git-receive-pack commands are run by Git itself, and we certainly did not change it :)

There were some big changes to the pre-push hook though: #1128. I wonder if something in the LFS pre-push hook is calling git-upload-pack...?

[ttaylorr]

Hi again -- I just dug into this today in preparing for the next minor release of LFS. I was unable to reproduce the hang myself, after starting v1.1.1 and then upgrading to v1.2.0. My hunch is that the ls-remote command that Git is running is hanging. (See: here).

Would you mind trying the following command, and posting what happens?

$ git ls-remote --heads --tags -q origin