ssh config with pubkeys does not load keys from ssh-agent

[shadow00]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.36.0.windows.1
cpu: x86_64
built from commit: ea1e13f73339d57cbe81a0bae6fba669aaccf656
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.16299.19]

• What options did you set as part of the installation? Or did you choose the
  defaults?

$ cat /etc/install-options.txt

Editor Option: VisualStudioCode
Custom Editor Path:
Default Branch Option: master
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: LFOnly
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Rebase
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Enabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

I use KeePass2 with the KeeAgent plugin to keep my private keys inside the password manager. In my .ssh folder I only keep the corresponding pubkey files, and an ssh config with hosts that look like this:

Host github.com
    User git
    AddressFamily = inet
    HostName github.com
    Port 22
    IdentityFile ~/.ssh/github_shadow00.pub
    # Note the pubkey! Not a private key ^
    ForwardAgent no

The public keys are in the following format:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBsQj71qX9dbbCIDjfOPwZ/z/u7US0zWqcSavqUP+2O3 foo@bar
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBsQj71qX9dbbCIDjfOPwZ/z/u7US0zWqcSavqUP+2O3 [MD5 fingerprint] foo@bar
ssh-rsa [4096 bit rsa key] foo@bar
ssh-rsa [4096 bit rsa key] [MD5 fingerprint] foo@bar

Then, in my .bash_profile, I have the following line:

# Use KeeAgent as SSH agent
export SSH_AUTH_SOCK="C:\Users\endor\.ssh\agent.sock"

In case they might be relevant:

$ where ssh
C:\Program Files\Git\usr\bin\ssh.exe

$ where ssh-agent
C:\Program Files\Git\usr\bin\ssh-agent.exe

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

ssh github.com

• What did you expect to occur after running these commands?

ssh looks up the settings for github.com in the .ssh/config file, finds the pubkey, requests the corresponding private key to ssh-agent, and then connects.

• What actually happened instead?

$ ssh github.com
Load key "/c/Users/shadow00/.ssh/github_shadow00.pub": invalid format
git@github.com: Permission denied (publickey).

$ ssh github.com -vvv
OpenSSH_8.9p1, OpenSSL 1.1.1n  15 Mar 2022
debug1: Reading configuration data /c/Users/shadow00/.ssh/config
debug1: /c/Users/shadow00/.ssh/config line 1: Applying options for *
debug3: kex names ok: [curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group18-sha512,diffie-hellman-group16-sha512,diffie-hellman-group14-sha256]
debug1: /c/Users/shadow00/.ssh/config line 178: Applying options for github.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/c/Users/shadow00/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/c/Users/shadow00/.ssh/known_hosts2'
debug2: resolving "github.com" port 22
debug3: resolve_host: lookup github.com:22
debug3: ssh_connect_direct: entering
debug1: Connecting to github.com [140.82.121.4] port 22.
debug3: set_sock_tos: set socket 4 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /c/Users/shadow00/.ssh/github_shadow00.pub type 3
debug1: identity file /c/Users/shadow00/.ssh/github_shadow00.pub-cert type -1
[... algo negotiation stuff, I doubt it's relevant]
debug1: Authenticating to github.com:22 as 'git'
[...]
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
debug3: record_hostkey: found key type ED25519 in file /c/Users/shadow00/.ssh/known_hosts:79
debug3: load_hostkeys_file: loaded 1 keys from github.com
debug1: Host 'github.com' is known and matches the ED25519 host key.
debug1: Found key in /c/Users/shadow00/.ssh/known_hosts:79
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation
debug1: get_agent_identities: ssh_fetch_identitylist: agent refused operation
debug1: Will attempt key: /c/Users/shadow00/.ssh/github_shadow00.pub ED25519 SHA256:[fingerprint] explicit
debug2: pubkey_prepare: done
[...]
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /c/Users/shadow00/.ssh/github_shadow00.pub ED25519 SHA256:[fingerprint] explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /c/Users/shadow00/.ssh/github_shadow00.pub ED25519 SHA256:[fingerprint] explicit
debug3: sign_and_send_pubkey: using publickey with ED25519 SHA256:[fingerprint]
debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:[fingerprint]
Load key "/c/Users/shadow00/.ssh/github_shadow00.pub": invalid format
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@github.com: Permission denied (publickey).

I've tried looking into #3468 and Openssh's release notes, but I can't seem to find anything relevant.
This setup used to work fine up to the previous upate (version 2.35.2.windows.2 I believe? The one with the security update). It seems however that version 2.36.0 broke something, because ssh does not fetch the private keys from the agent anymore.
Note that the agent still seems to have all the keys loaded: ssh-add -l returns the expected list of keys loaded from KeePass.

However, if I point my ssh config to an actual private key file and manually provide the passphrase to unlock it, I am able to connect to my hosts as usual. So I'm guessing something broke down in the communication between ssh and ssh-agent?

[dscho]

In particular on Windows, where Unix sockets are not generally available and therefore have to be emulated in one way or another, you cannot mix and match ssh from one OpenSSH package and ssh-agent from another one. You will most likely need to set core.sshCommand.

[shadow00]

Note that I'm still on Win10 build 1709, so the only version of ssh and ssh-agent I have are the ones installed by git-for-windows. Windows' version of ssh was only included in 2018.

Also, the setup that I described had been working as intended for years until the last git-for-windows update (which I performed last week). Hence my question: what changed? The only clue I got from the release notes is the bump from OpenSSH v8.8p1 to OpenSSH v8.9p1, but In couldn't find anything in their release notes.

[carenas]

Also, the setup that I described had been working as intended for years until the last git-for-windows update (which I performed last week). Hence my question: what changed?

Assuming that you didn't kill your ssh-agent process after that update, the comment from above about running two different versions of openssh still applies IMHO. The change of behaviour might had been introduced with that new OpenSSH version, so it would be useful to know if still works after using a clean setup (ex: after a reboot)

[shadow00]

it would be useful to know if still works after using a clean setup (ex: after a reboot)

Do you mean just rebooting after the git-for-windows update and seeing if the problem persists? Yes, I've rebooted several times already. I haven't messed with the openssh installation, so afaik the only version of ssh and ssh-agent on my machine are the ones that came with the git-for-windows installation/update.

[shadow00]

It appears that the $SSH_AGENT_PID env variable is not getting set, even though $SSH_AUTH_SOCK is picked up correctly from my .bash_profile. My Keepass/Keeagent settings nor versions haven't changed, so I would exclude that. I don't have a reference from before the update, but I would assume that they were getting set correctly since everything was working - right?

What's weird is that ssh-add -l picks up Keeagent just fine though. Were there any changes in the last update in the way sockets work?

[shadow00]

Just tried downgrading back to git version 2.35.3.windows.1 and the agent works again. $SSH_AGENT_PID is still not set.

[kamrup]

I'm having the same problem. Downgrading to 2.35.3.windows.1 makes the agent work again.

[kamrup]

KeeAgent 0.12.1 is apparently incompatible with OpenSSH 8.9, see: dlech/KeeAgent#341
It looks like there's a pre-release which should fix the issue though.

[shugaoye]

I have exactly the same issue. I tested the current version 2.39.1 and it still cannot work. I downgraded to 2.34.1 and this version works fine.

[dscho]

you cannot mix and match ssh from one OpenSSH package and ssh-agent from another one.

This is still true. I simply forgot to close the ticket, sorry.