Prompting for SSH password when it shouldn't

[er2]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.33.0.windows.2
cpu: x86_64
built from commit: 8735530946cced809cc6cc4c2ca3b078cdb3dfc8
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

^This is the last working version which I rolled back to. Affected version is 2.33.1

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Windows 10.0 19042 64-bit

• What options did you set as part of the installation? Or did you choose the
  defaults?

Default options. Installed via Chocolatey

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash and WIndows Terminal

What did you do before the bug happened? (Steps to reproduce your issue)

Used git as normal, fetch/pushing to a gitlab instance, authenticated via an SSH key, with a blank password

What did you expect to happen? (Expected behavior)

Successfully fetch without a password prompt

What happened instead? (Actual behavior)

It asked me for a password.

What's different between what you expected and what actually happened?

Anything else you want to add:

Reporting from a rolled back version that works correctly. The affected version is 2.33.1 installed on Windows via Chocolatey

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.

[System Info]
git version:
git version 2.33.0.windows.2
cpu: x86_64
built from commit: 8735530
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
uname: Windows 10.0 19042
compiler info: gnuc: 10.3
libc info: no libc information available
$SHELL (typically, interactive shell):

[Enabled Hooks]

[dscho]

Reporting from a rolled back version that works correctly. The affected version is 2.33.1 installed on Windows via Chocolatey

My biggest question is: which ssh.exe is in play here? Does Git for Windows via Chocolatey include the same OpenSSH version as the regular Git for Windows installer? Or the Portable Git? Can you verify that you experience the same issue using the latest snapshot's Portable Git version?

[er2]

I am having the issue with Portable Git 64-bit 6ffa071

[dscho]

I am having the issue with Portable Git 64-bit 6ffa071

Okay, that is the latest snapshot. Clearly, you had the issue with an earlier version, too: your original report indicates that 05d80ad exposes the reported issue, when 8735530 does not. There are quite a couple snapshots between that. Would you mind bisecting which snapshot was the first to prompt when it should not?

[er2]

Date	Version	Good/Bad
2021-10-30-13:38:26	PortableGit-2.34.0-rc0-64-bit.7z.exe	Bad
2021-10-29-14:11:24	PortableGit-prerelease-2.33.1.windows.1-36-gbd465d70df-20211029141659-64-bit.7z.exe
2021-10-28-19:16:10	PortableGit-prerelease-2.33.1.windows.1-27-gf8f57df4d5-20211028224346-64-bit.7z.exe
2021-10-15-13:12:48	PortableGit-prerelease-2.33.1.windows.1-13-g3f28c5f141-20211015131715-64-bit.7z.exe
2021-10-14-12:22:50	PortableGit-prerelease-2.33.1.windows.1-2-gc7d0a86b02-20211014122857-64-bit.7z.exe
2021-10-13-16:23:50	PortableGit-2.33.1-64-bit.7z.exe
2021-10-13-15:40:27	PortableGit-prerelease-2.33.0.windows.2-148-ga98d291bbd-20211013154533-64-bit.7z.exe
2021-10-13-14:47:05	PortableGit-prerelease-2.33.0.windows.2-144-g0055932c3c-20211013145330-64-bit.7z.exe	Bad
2021-10-13-13:08:43	PortableGit-prerelease-2.33.0.windows.2-141-g42f3463621-20211013131513-64-bit.7z.exe
2021-10-13-05:43:47	PortableGit-prerelease-2.33.0.windows.2-135-ge9b4cc4f52-20211013054852-64-bit.7z.exe
2021-10-12-10:06:10	PortableGit-prerelease-2.33.0.windows.2-131-gaaec11d74d-20211012102731-64-bit.7z.exe
2021-10-02-21:32:18	PortableGit-prerelease-2.33.0.windows.2-126-g988794267c-20211004113703-64-bit.7z.exe	Bad
2021-09-08-10:35:44	PortableGit-prerelease-2.33.0.windows.2-7-g2aa9e28d2a-20210908104152-64-bit.7z.exe	Good
2021-09-08-08:50:49	PortableGit-prerelease-2.33.0.windows.2-5-g82a7ae1b54-20210908085653-64-bit.7z.exe	Good
2021-08-31-07:55:26	PortableGit-prerelease-2.33.0.windows.2-2-g0d6923f69e-20210831080057-64-bit.7z.exe
2021-08-23-22:05:45	PortableGit-2.33.0.2-64-bit.7z.exe	Good

[dscho]

2021-10-02-21:32:18 PortableGit-prerelease-2.33.0.windows.2-126-g988794267c-20211004113703-64-bit.7z.exe Bad
2021-09-08-10:35:44 PortableGit-prerelease-2.33.0.windows.2-7-g2aa9e28d2a-20210908104152-64-bit.7z.exe Good

The diff between the included etc/package-versions.txt reads like this:

diff --git a/versions-2.33.0.windows.2-126-g988794267c-20211004113703.txt b/versions-2.33.0.windows.2-7-g2aa9e28d2a-20210908104152.txt
index c688c75..a988b6a 100644
--- a/versions-2.33.0.windows.2-126-g988794267c-20211004113703.txt
+++ b/versions-2.33.0.windows.2-7-g2aa9e28d2a-20210908104152.txt
@@ -4,7 +4,7 @@ bash 4.4.023-1
 bzip2 1.0.8-2
 ca-certificates 20210119-2
 coreutils 8.32-1
-diffutils 3.8-1
+diffutils 3.7-1
 docx2txt 1.4-1
 dos2unix 7.4.2-1
 expat 2.4.1-1
@@ -13,13 +13,13 @@ findutils 4.8.0-1
 gawk 5.0.0-1
 gcc-libs 10.2.0-1
 gettext 0.19.8.1-1
-git-extra 1.1.558.008c122-1
+git-extra 1.1.555.3da707f-1
 git-flow 1.12.3-1
 glib2 2.68.4-1
 gmp 6.2.1-1
 gnupg 2.2.29-1
 grep 3.6-1
-gzip 1.11-1
+gzip 1.10-1
 heimdal-libs 7.5.0-3
 icu 69.1-1
 less 590-1
@@ -28,7 +28,7 @@ libassuan 2.5.5-1
 libbz2 1.0.8-2
 libcbor 0.8.0-1
 libcrypt 2.1-3
-libcurl 7.79.1-1
+libcurl 7.78.0-1
 libedit 20210714_3.1-1
 libexpat 2.4.1-1
 libffi 3.3-1
@@ -45,7 +45,7 @@ libksba 1.6.0-1
 liblz4 1.9.3-1
 liblzma 5.2.5-1
 libnettle 3.7.3-1
-libnghttp2 1.45.0-1
+libnghttp2 1.44.0-1
 libnpth 1.6-1
 libopenssl 1.1.1.l-1
 libp11-kit 0.24.0-1
@@ -56,7 +56,7 @@ libreadline 8.1.001-1
 libsasl 2.1.27-1
 libserf 1.3.9-6
 libsqlite 3.36.0-2
-libssh2 1.10.0-1
+libssh2 1.9.0-1
 libtasn1 4.17.0-2
 libunistring 0.9.10-1
 libutil-linux 2.35.2-1
@@ -66,35 +66,35 @@ mingw-w64-x86_64-antiword 0.37-2
 mingw-w64-x86_64-brotli 1.0.9-3
 mingw-w64-x86_64-bzip2 1.0.8-2
 mingw-w64-x86_64-ca-certificates 20200601-3
-mingw-w64-x86_64-c-ares 1.17.2-1
+mingw-w64-x86_64-c-ares 1.17.1-1
 mingw-w64-x86_64-connect 1.105-2
-mingw-w64-x86_64-curl 7.79.1-1
+mingw-w64-x86_64-curl 7.78.0-1
 mingw-w64-x86_64-expat 2.4.1-1
 mingw-w64-x86_64-gcc-libs 10.3.0-5
 mingw-w64-x86_64-gettext 0.19.8.1-10
-mingw-w64-x86_64-git 2.33.0.windows.2.126.g988794267c.20211004113703-1
+mingw-w64-x86_64-git 2.33.0.windows.2.7.g2aa9e28d2a.20210908104152-1
 mingw-w64-x86_64-git-credential-manager-core 2.0.498.54650-1
-mingw-w64-x86_64-git-doc-html 2.33.0.windows.2.126.g988794267c.20211004113703-1
-mingw-w64-x86_64-git-lfs 3.0.1-1
+mingw-w64-x86_64-git-doc-html 2.33.0.windows.2.7.g2aa9e28d2a.20210908104152-1
+mingw-w64-x86_64-git-lfs 2.13.3-1
 mingw-w64-x86_64-gmp 6.2.1-2
-mingw-w64-x86_64-gnutls 3.7.2-4
+mingw-w64-x86_64-gnutls 3.7.2-3
 mingw-w64-x86_64-jansson 2.13.1-1
 mingw-w64-x86_64-jemalloc 5.2.1-2
 mingw-w64-x86_64-libffi 3.3-4
 mingw-w64-x86_64-libiconv 1.16-2
 mingw-w64-x86_64-libidn2 2.3.1-1
-mingw-w64-x86_64-libssh2 1.10.0-1
+mingw-w64-x86_64-libssh2 1.9.0-5
 mingw-w64-x86_64-libsystre 1.0.1-4
 mingw-w64-x86_64-libtasn1 4.17.0-1
 mingw-w64-x86_64-libtre-git r128.6fb7206-2
 mingw-w64-x86_64-libunistring 0.9.10-4
-mingw-w64-x86_64-libwinpthread-git 9.0.0.6306.586baa17b-1
+mingw-w64-x86_64-libwinpthread-git 9.0.0.6294.f5ac9206e-1
 mingw-w64-x86_64-libxml2 2.9.12-3
-mingw-w64-x86_64-libzip 1.8.0-1
+mingw-w64-x86_64-libzip 1.7.3-3
 mingw-w64-x86_64-mpc 1.2.1-1
 mingw-w64-x86_64-mpfr 4.1.0.p13-1
 mingw-w64-x86_64-nettle 3.7.3-3
-mingw-w64-x86_64-nghttp2 1.45.1-1
+mingw-w64-x86_64-nghttp2 1.43.0-1
 mingw-w64-x86_64-odt2txt 0.5-2
 mingw-w64-x86_64-openssl 1.1.1.l-1
 mingw-w64-x86_64-pcre 8.45-1
@@ -111,7 +111,7 @@ msys2-runtime 3.1.7-5
 nano 5.8-1
 ncurses 6.2-1
 nettle 3.7.3-1
-openssh 8.8p1-1
+openssh 8.7p1-1
 openssl 1.1.1.l-1
 p11-kit 0.24.0-1
 patch 2.7.6-1
@@ -127,13 +127,13 @@ perl-HTML-Tagset 3.20-2
 perl-HTTP-Cookies 6.10-1
 perl-HTTP-Daemon 6.12-1
 perl-HTTP-Date 6.05-1
-perl-HTTP-Message 6.33-1
+perl-HTTP-Message 6.32-1
 perl-HTTP-Negotiate 6.01-2
 perl-IO-HTML 1.004-1
-perl-IO-Socket-SSL 2.072-1
+perl-IO-Socket-SSL 2.071-1
 perl-IO-Stringy 2.113-1
 perl-JSON 4.03-1
-perl-libwww 6.57-1
+perl-libwww 6.55-1
 perl-LWP-MediaTypes 6.04-1
 perl-MailTools 2.21-1
 perl-MIME-tools 5.509-1
@@ -152,11 +152,11 @@ subversion 1.14.1-1
 tar 1.34-1
 tig 2.5.4-1
 unzip 6.0-2
-vim 8.2.3441-1
+vim 8.2.3182-1
 which 2.21-2
 winpty 0.4.3-1
 zlib 1.2.11-1
-filesystem 2021.06-2
+filesystem 2021.06-1
 dash 0.5.11.4-1
 rebase 4.5.0-1
 util-linux 2.35.2-1

The most likely culprit is the upgrade to OpenSSH v8.8. Could you copy usr\bin\ssh.exe from the good to the bad version, to see whether it works around the issue?

[er2]

Bingo. Copying ssh.exe from the first bad version to the last good version presented the bug. And I copied just that file, not any of the other ssh*.exe executables in the same folder.

[er2]

I just realized I did that backwards. I'll try it the way you said.

[er2]

Yep, copying ssh.exe from the last good version to the first bad version fixed the bug.

[dscho]

Hrm. I cannot spot anything in the release notes at https://www.openssh.com/txt/release-8.8 that could explain the reported behavior... can you spot anything suspicious?

[er2]

I added the ssh config mentioned in the release notes,

    Host my-gitlab
        HostkeyAlgorithms +ssh-rsa
	PubkeyAcceptedAlgorithms +ssh-rsa

and now all previously broken installs successfully fetch without prompting for a password.

[dscho]

Excellent!

[er2]

Isn't it still a bug that this resulted in such unclear behavior? Currently there's no amount of -v flags I can add to fetch to have it tell me what's going on in this case.

[er2]

For the record, this problem only appears on an out of date gitlab instance. Without the config and with the newer ssh.exe and the same old key, fetching from gitlab.com works fine.

[dscho]

this problem only appears on an out of date gitlab instance.

I think the problem is that that instance might offer insecure host keys, and that's exactly what the new OpenSSH version wants to prevent you from relying on.

TBH I am somewhat puzzled why you seem to be able to still use it, albeit after typing your password. Maybe there are multiple keys in play, and ssh first uses an inapplicable one, then falls back to a secure one?

[er2]

I think the problem is that that instance might offer insecure host keys, and that's exactly what the new OpenSSH version wants to prevent you from relying on.

The problem is that I had to work with you and do all this work to figure out what was going on. It's fine for this to block me from fetching as long as it tells me "key algorithm deprecated. Please generate a new key, or update your server, or configure an exception." Do you want to go through this 30,000 more times?

TBH I am somewhat puzzled why you seem to be able to still use it, albeit after typing your password.

That's not true. I'm not able to fetch when I'm affected by the "bug" and it prompts for a password. I don't have a password for "git@my-gitlab" nor for the ssh key.

[dscho]

The problem is that I had to work with you

I am sorry 😝

I'm not able to fetch when I'm affected by the "bug"

Oh, I missed that, sorry. And about this:

there's no amount of -v flags I can add to fetch to have it tell me what's going on in this case.

I don't think that the -v flags should go to the fetch command. You will have to either call ssh with tons of -v options directly, or do something like git -c core.sshCommand="ssh -v -v -v -v -v -v" fetch [...] so that it is the ssh command that receives those -v options.