git push hangs when GIT_SSH is OpenSSH

[AnthonyMastrean]

"OpenSSH for Windows" version

PS> ((Get-Item (Get-Command sshd).Source).VersionInfo.FileVersion)
7.7.2.2

Client OperatingSystem

PS> ((Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows nt\CurrentVersion\" -Name ProductName).ProductName)
Windows 10 Pro

What is failing

When GIT_SSH is set to OpenSSH, per the advice here dahlbyk/posh-git#575 (comment), seemingly any git push hangs indefinitely? I had to send an interrupt (CTRL+C) to get the console back.

GIT_SSH=C:\WINDOWS\System32\OpenSSH\ssh.exe

Maybe related to #389?

Expected output

A successful push 🤗

Actual output

$ GIT_TRACE=1 git push --force-with-lease
16:04:28.047029 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
16:04:28.050024 git.c:419               trace: built-in: git push --force-with-lease
16:04:28.068027 run-command.c:643       trace: run_command: unset GIT_PREFIX; 'C:\WINDOWS\System32\OpenSSH\ssh.exe' git@gitlab.com 'git-receive-pack '\''vaeit/nextgen/packaging.git'\'''
16:04:29.069626 run-command.c:643       trace: run_command: .git/hooks/pre-push origin git@gitlab.com:vaeit/nextgen/packaging.git
16:04:29.217805 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
16:04:29.219805 git.c:676               trace: exec: git-lfs pre-push origin git@gitlab.com:vaeit/nextgen/packaging.git
16:04:29.220805 run-command.c:643       trace: run_command: git-lfs pre-push origin git@gitlab.com:vaeit/nextgen/packaging.git
16:04:29.249803 trace git-lfs: exec: git 'version'
16:04:29.294969 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
16:04:29.380835 trace git-lfs: exec: git 'config' '-l'
16:04:29.403839 trace git-lfs: pre-push: refs/heads/INFRA-489 47d6d3948f7a15c31e9c019256d2735e452195f6 refs/heads/INFRA-489 963168ae70b8ac4fc406fc1440b2bb9f12a77630
16:04:30.887486 trace git-lfs: run_command: C:\WINDOWS\System32\OpenSSH\ssh.exe -- git@gitlab.com git-lfs-authenticate vaeit/nextgen/packaging.git upload
Connection reset by 35.231.145.151 port 22
16:07:26.575373 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
16:07:26.575373 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
16:07:26.575373 trace git-lfs: filepathfilter: accepting "tmp"

Exiting because of "interrupt" signal.

[rkeithhill]

One possibility is that GitLab is using a cipher suite that is not configured by default with OpenSSH. I ran into this issue with TFS Git on our TFS 2015 server. Try this to enable verbose logging:

$env:GIT_SSH_COMMAND = '"C:\Windows\System32\OpenSSH\ssh.exe" -vvv'

Try the push and see what you get in the output.

[seed-of-apricot]

I ran into the same issue, with multiple SSH keys managed.
The problem is, I believe, in git lfs because when I don't install lfs it works perfectly.

Related issue

[obrienmd]

I see this with repos with git-lfs as well, but no others. Turning on ssh -vvv and git trace provides additional clues. The initial ssh connection works and connects as expected, but the process gets stuck here with no additional debug output:

debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
14:16:36.985296 run-command.c:663       trace: run_command: .git/hooks/pre-push origin git@a.gitserver.com:mygroup/my-repo.git
14:16:37.171576 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:16:37.196030 git.c:704               trace: exec: git-lfs pre-push origin git@a.gitserver.com:mygroup/my-repo.git
14:16:37.197028 run-command.c:663       trace: run_command: git-lfs pre-push origin git@a.gitserver.com:mygroup/my-repo.git
14:16:38.022938 trace git-lfs: exec: git 'version'
14:16:38.119177 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
14:16:38.474769 trace git-lfs: exec: git 'config' '-l'
14:16:39.907192 trace git-lfs: run_command: sh -c "C:\Windows\System32\OpenSSH\ssh.exe" -vvv -- git@a.gitserver.com 'git-lfs-authenticate mygroup/my-repo.git upload'

I think it's able to connect just fine, but when the run_command is dispatched it hangs.

Same openssh and Win10 versions as original poster. Other versions:

git-lfs/2.7.2 (GitHub; windows amd64; go 1.12.2; git 08a08ae0)
git version 2.22.0.windows.1
GIT_EXEC_PATH=C:/Program Files/Git/mingw64/libexec/git-core
GIT_LFS_PATH=C:\Program Files\Git LFS
GIT_SSH_COMMAND="C:\Windows\System32\OpenSSH\ssh.exe" -vvv
GIT_TRACE=1

[lephyrus]

I'm having the exact same issue. The trace looks the same.

==== snip ====
11:27:30.147136 trace git-lfs: pre-push: refs/heads/nvm-test e6df7fd73686a4a16617f5b4ddeca75c37abf625 refs/heads/nvm-test 0000000000000000000000000000000000000000
11:27:31.818155 trace git-lfs: run_command: C:\Users\REDACTED\scoop\apps\win32-openssh\current\ssh.exe -- git@github.com git-lfs-authenticate myorg/myrepo.git upload
=== hangs here, output below after CTRL-C ===
11:30:27.533441 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
11:30:27.533441 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
11:30:27.539438 trace git-lfs: filepathfilter: accepting "tmp"
11:30:27.541438 trace git-lfs: filepathfilter: accepting "objects"

I'd like to add that running the command which appears to hang manually is successful:

> C:\Users\REDACTED\scoop\apps\win32-openssh\current\ssh.exe -- git@github.com git-lfs-authenticate myorg/myrepo.git upload
{
  "href": "https://lfs.github.com/myorg/myrepo.git",
  "header": {
    "Authorization": "RemoteAuth REDACTED"
  },
  "expires_at": "2019-06-20T13:56:42Z",
  "expires_in": 21599
}

Versions:

OpenSSH_for_Windows_7.9p1, LibreSSL 2.6.5
git-lfs/2.7.2 (GitHub; windows amd64; go 1.12.2; git 08a08ae0)
git version 2.22.0.windows.1

[davclark]

I have the same issue with the same configuration. I tried with git installed via scoop and also via chocolatey. I'm trying now using the openssh installed from scoop. That works fine... it's a minor annoyance for someone who wants a clean install, since you've got two versions of OpenSSH installed, and the scoop version installs the whole msys2 stack. But having a working git is more important! Looking forward to a cleaner fix.

I think this establishes that it's an issue with the microsoft-packaged OpenSSH?

[yogasantosa]

Latest release (v8.0.0.0p1-Beta) from https://github.com/PowerShell/Win32-OpenSSH/releases works fine.

The one activated through Windows 10 optional feature is as far as i know - is 7.9 and having this hanging issue.

[AnthonyMastrean]

@yogasantosa do you see anything specific that fixes this issue (in the linked milestone, maybe)? Or does this one just happen to work in your testing?

[yogasantosa]

@AnthonyMastrean - found this by accident:

• got 2 machines: 1 running Windows 10, the other windows 7
• in windows 10, i used optional feature to install (v 7.9)
• in windows 7, i installed from the link I shared (v 8.0)
• and i noticed in windows 10 machine it was hanging, but not on windows 7
• then i checked the version, uninstalled the one on windows 10 and installed v8.0 and it works

Looking at the milestone there seemed to be two possible issues related to this one

[maertendMSFT]

This has been reported to be working, please reopen if this repros on the latest release.