git fetch complains about bad RSA key signature with gitlab.com

[t-b]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$  git --version --build-options
git version 2.31.0.rc2.windows.1
cpu: x86_64
built from commit: 964fdfa4ed1d39cfa898d61abcee4048c71c5a93
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.18363.1440] 64bit

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: ConHost
Git Pull Behavior Option: Rebase
Use Credential Manager: Disabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Nope.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Windows Terminal with bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

$ git clone --depth 1 git@gitlab.com:tango-controls/cppTango
Cloning into 'cppTango'...
client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0
remote: Enumerating objects: 1236, done.
remote: Counting objects: 100% (1236/1236), done.
remote: Compressing objects: 100% (959/959), done.
remote: Total 1236 (delta 354), reused 395 (delta 80), pack-reused 0
Receiving objects: 100% (1236/1236), 2.27 MiB | 4.30 MiB/s, done.
Resolving deltas: 100% (354/354), done.

$ cat ~/.ssh/known_hosts
gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf

My SSH key is also not a RSA key but a ED25519 one.

• What did you expect to occur after running these commands?

no bad signature message combined with a successfull clone. When the signature is bad and I should not be able clone.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

gitlab.com/tango-controls/cppTango

I tried to reproduce it with a portable installation of git version 2.30.2.windows.1 but that always worked.

[t-b]

My hunch is that this might due to openssh 8.5p1 from 2.31.0.rc2.

[dscho]

My hunch is that this might due to openssh 8.5p1 from 2.31.0.rc2.

Quite possible. https://github.com/git-for-windows/build-extra/blob/HEAD/versions/package-versions-2.30.2.txt#L117 says that Git for Windows v2.30.2 came with 8.4p1. Could you drop the usr/bin/ssh.exe from v2.30.2 into your Git installation and verify this hunch?

[dscho]

https://forum.gitlab.com/t/openssh-8-5p1-1-potential-problem-ssh-keys-on-gitlab/49980 might be relevant, too.

[dscho]

And https://www.reddit.com/r/archlinux/comments/lyazre/openssh_update_causes_problems/. This even has the suggestion to work around the issue by putting this into your $HOME/.ssh/config:

Host gitlab.com
  UpdateHostKeys no

[t-b]

Thanks for digging.

Using the ssh.exe in version 8.4p1 from portable 2.30.2 fixes the issue, as does using UpdateHostKeys no.

thomas@Win10-Thomas-PC MINGW64 /e/projekte/cppTango (backport/musl-fixes)
$ GIT_SSH_COMMAND=~/Downloads/PortableGit/usr/bin/ssh.exe git clone --depth 1 git@gitlab.com:tango-controls/cppTango
Cloning into 'cppTango'...
The authenticity of host 'gitlab.com (2606:4700:90:0:f22e:fbec:5bed:a9b9)' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'gitlab.com,2606:4700:90:0:f22e:fbec:5bed:a9b9' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 1236, done.
remote: Counting objects: 100% (1236/1236), done.
remote: Compressing objects: 100% (959/959), done.
remote: Total 1236 (delta 354), reused 395 (delta 80), pack-reused 0
Receiving objects: 100% (1236/1236), 2.27 MiB | 4.14 MiB/s, done.
Resolving deltas: 100% (354/354), done.
(base)
thomas@Win10-Thomas-PC MINGW64 /e/projekte/cppTango (backport/musl-fixes)
$ rm ~/.ssh/known_hosts
(base)
thomas@Win10-Thomas-PC MINGW64 /e/projekte/cppTango (backport/musl-fixes)
$ rm -rf cppTango/
(base)
thomas@Win10-Thomas-PC MINGW64 /e/projekte/cppTango (backport/musl-fixes)
$ git clone --depth 1 git@gitlab.com:tango-controls/cppTango
Cloning into 'cppTango'...
The authenticity of host 'gitlab.com (2606:4700:90:0:f22e:fbec:5bed:a9b9)' can't be established.
ED25519 key fingerprint is SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'gitlab.com' (ED25519) to the list of known hosts.
client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0
remote: Enumerating objects: 1236, done.
remote: Counting objects: 100% (1236/1236), done.
remote: Compressing objects: 100% (959/959), done.
remote: Total 1236 (delta 354), reused 395 (delta 80), pack-reused 0
Receiving objects: 100% (1236/1236), 2.27 MiB | 2.75 MiB/s, done.
Resolving deltas: 100% (354/354), done.
(base)

[dscho]

FWIW https://bugzilla.mindrot.org/buglist.cgi?quicksearch=server%20gave%20bad%20signature%20for%20RSA%20key came up empty. You might want to dig in a bit more yourself, but my hunch is that GitLab uses an ssh-rsa host key and OpenSSH v8.5p1 might have unintentionally disallowed that (I don't find anything in the notes about potentially backwards-incompatible changes in https://www.openssh.com/txt/release-8.5).

[t-b]

Thanks for your time and help. I've opened https://bugzilla.mindrot.org/show_bug.cgi?id=3279.

[t-b]

You might want to dig in a bit more yourself, but my hunch is that GitLab uses an ssh-rsa host key and OpenSSH v8.5p

From my understanding the host key is of ed25519 type as posted in #3108 (comment).

[dscho]

I've opened https://bugzilla.mindrot.org/show_bug.cgi?id=3279.

Excellent. Could you keep us posted? If no patch for this manifests before Git for Windows v2.31.0, we might have to hack /etc/ssh/ssh_config.