Yubikey FIDO2 ED25519-SK -- Failed to authenticate key 0: SSH-Agent reports failure

[MaxwellDPS]

Hey @z4yx!

Super neat library, seeing some issues when trying to use the pam module, it seems to be an issue with the ssh agent, but I am struggling to track it down. For some context of the environment: SSH is established using a ECDSA-SK that has a seperate CA. I am attempting then to use the 2nd agent key that is loaded as a resident ED25519-SK key on a yubikey for sudo auth

Please let me know if I can do anything to troubleshoot here! Thanks in advance!

OS: Ubuntu 22.04 Linux sea-z1-ctu-1 5.15.0-70-generic #77-Ubuntu SMP Tue Mar 21 14:02:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

pam.d/sudo

#%PAM-1.0

# Set up user limits from /etc/security/limits.conf.
session    required   pam_limits.so

session    required   pam_env.so readenv=1 user_readenv=0
session    required   pam_env.so readenv=1 envfile=/etc/default/locale user_readenv=0

auth sufficient /usr/local/lib/libpam_rssh.so auth_key_file=/opt/chaos.corp/auth_sudo loglevel=trace

@include common-auth
@include common-account
@include common-session-noninteractive

auth.log

Jun  6 13:35:01 sea-z1-ctu-1 sshd[2814341]: Starting session: shell on pts/3 for max from 172.30.50.10 port 50224 id 0
Jun  6 13:35:01 sea-z1-ctu-1 sshd[2814342]: debug1: Setting controlling tty using TIOCSCTTY.
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: SSH-Agent address: /tmp/ssh-XXXXt2jJ66/agent.2814341
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Reading configured authorized_keys file: /opt/chaos.corp/auth_sudo
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: parse_authorized_keys: sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIEcmWXEXk0+TuL5BLcMmWygLYxDd8dhTbbM7yU2sI4PFAAAADHNzaDpzdWRvLXBhbQ==
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Got 1 entries from authorized_keys
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Connected to Unix("/tmp/ssh-XXXXt2jJ66/agent.2814341")
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Written 5 bytes: [00, 00, 00, 01, 0B]
Jun  6 13:37:09 sea-z1-ctu-1 sshd[2814341]: debug1: channel 2: new [accepted auth socket]
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: read_message len=725
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Read 725 bytes: [<REMOVED BYTES>]
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: list_identities: [<REMOVED BYTES>] ()
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: list_identities: [<REMOVED BYTES>] ()
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: SSH-Agent reports 1 keys
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Key 0 is authorized
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Written 131 bytes: [<REMOVED BYTES>]
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: read_message len=1
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Read 1 bytes: [05]
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: Failed to authenticate key 0: SSH-Agent reports failure
Jun  6 13:37:09 sea-z1-ctu-1 pam_rssh[2815745]: None of these keys passed authentication
Jun  6 13:37:09 sea-z1-ctu-1 sshd[2814341]: debug1: channel 2: free: accepted auth socket, nchannels 3

[z4yx]

It seems that ssh-agent reported an error, but doesn't give any detailed info. You may run ssh-agent with -d option, then check if there is any useful information on that error.

[MaxwellDPS]

Hey @z4yx I am having some issues on MacOS getting ssh-agent to load the key when the agent is spawned with -d

On the remote side, running with -d also gives no output.

Do you mind providing an example of how I should spawn the agent?

Thanks!

[z4yx]

• On the machine where Yubikey presents, start ssh-agent with ssh-agent -d.
• Set environment vars as ssh-agent prints.
• Then add a pkcs plugin as per Yubikey user guide: ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
• Now ssh to a remote machine, and try sudo with pam_rssh.

ssh-agent should give messages like:

debug1: new_socket: type = SOCKET
debug2: fd 3 setting O_NONBLOCK
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 20
debug2: process_add_smartcard_key: entering
debug1: process_add_smartcard_key: add /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
debug1: pkcs11_start_helper: starting /usr/local/libexec/ssh-pkcs11-helper -vvv
debug1: process_add
debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.19
debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so slot 0: label <XXX> manufacturerID <piv_II> model <PKCS#15 emulate> serial <XXX> flags 0x40d
debug2: pkcs11_fetch_keys: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so slot 0: ECDSA SHA256:XXXXXXXXXXXX
debug1: have 1 keys
debug2: pkcs11_fetch_certs: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so slot 0: ECDSA SHA256:XXXXXXXXXXXX
debug2: pkcs11_fetch_certs: key already included
debug1: pkcs11_k11_free: parent 0x55e773eb1e00 ptr 0x55e773f134e0 idx 1
debug1: pkcs11_provider_unref: provider "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so" refcount 2
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug1: process_message: socket 1 (fd=4) type 13
debug1: process_sign_request2: entering
debug1: process_sign
debug1: check ECDSA /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so CARD AUTH pubkey
debug1: pkcs11_check_obj_bool_attrib: provider "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so" slot 0 object 94452570984256: attrib 514 = 0
debug1: pkcs11_k11_free: parent 0x55e773eb1e00 ptr (nil) idx 1

[z4yx]

Sorry, I didn't realize you are using ssh-agent with FIDO2. The steps I gave is for PIV.

Just ignore the "pkcs plugin" step.

[MaxwellDPS]

Hey @z4yx Thanks! Apologies on the delayed response... Looking at this I think its due to the interactive (touch) requirement set on the key.

Logs

new_socket: type = CONNECTION
fd 4 is O_NONBLOCK
process_message: socket 1 (fd=4) type 27
process_extension: entering
process_ext_session_bind: entering
process_ext_session_bind: recorded ECDSA-CERT SHA256:vTYVqGtIPBZGO/R9uoeRo2N0TNVStFTQQ6jhb6WF82Y (slot 0 of 16)
process_message: socket 1 (fd=4) type 11
process_request_identities: entering
identity_permitted: entering: key ED25519-SK comment "", 1 socket bindings, 0 constraints
identity_permitted: entering: key ECDSA-CERT comment "", 1 socket bindings, 0 constraints
process_request_identities: replying with 2 allowed of 2 available keys
process_message: socket 1 (fd=4) type 13
process_sign_request2: entering
Confirm user presence for key ED25519-SK SHA256:uSJyeEuJ1TxI5uThrmBBhVe245aa05jn9UNR11j6kTk
start_helper: started pid=4018
ssh_msg_send: type 5
ssh_msg_recv entering
start_helper: starting /opt/homebrew/Cellar/openssh/9.3p1/libexec/ssh-sk-helper
process_sign: ready to sign with key ED25519-SK, provider internal: msg len 32, compat 0x0
sshsk_sign: provider "internal", key ED25519-SK, flags 0x25
sk_probe: 1 device(s) detected
sk_probe: selecting sk by touch
check_sk_options: option uv is unknown
ssh_sk_sign: check_sk_options uv
sshsk_sign: sk_sign failed with code -3
ssh-sk-helper: Signing failed: incorrect passphrase supplied to decrypt private key
main: reply len 8
ssh_msg_send: type 5
client_converse: helper returned error -43
reap_helper: pid=4018
process_sign_request2: sshkey_sign: incorrect passphrase supplied to decrypt private key
start_helper: started pid=4019
ssh_msg_send: type 5
ssh_msg_recv entering
start_helper: starting /opt/homebrew/Cellar/openssh/9.3p1/libexec/ssh-sk-helper
process_sign: ready to sign with key ED25519-SK, provider internal: msg len 32, compat 0x0
sshsk_sign: provider "internal", key ED25519-SK, flags 0x25
sk_probe: 1 device(s) detected
sk_probe: selecting sk by touch
check_sk_options: option uv is unknown
ssh_sk_sign: check_sk_options uv
sshsk_sign: sk_sign failed with code -3
ssh-sk-helper: Signing failed: incorrect passphrase supplied to decrypt private key
main: reply len 8
ssh_msg_send: type 5
client_converse: helper returned error -43
reap_helper: pid=4019
process_sign_request2: sshkey_sign: incorrect passphrase supplied to decrypt private key
process_sign_request2: sshkey_sign: incorrect passphrase supplied to decrypt private key
process_sign_request2: good signature

[z4yx]

Have you tested normal ssh login with this resident key? Or ssh with forwarded ssh-agent?
From this log, it seems like Yubikey refused to sign because of PIN or touch. Did you set the PIN of FIDO2?

[MaxwellDPS]

Yep! So ssh auth is good, and pin is set. I added a touch requirement to the resident key, it seems its not waiting for that on sudo. Instead of a pause for the key to flash for touch, it just fails immediately

[z4yx]

After reading the code, I found that the ssh-agent fails to get the pin from user, at this line:

https://github.com/openssh/openssh-portable/blob/2709809fd616a0991dc18e3a58dea10fb383c3f0/ssh-agent.c#L837

Then it calls sshkey_sign with pin=NULL. Log sshsk_sign: provider "internal", key ED25519-SK, flags 0x25 confirms this. If the pin is not NULL, it should print with-pin at the end.

https://github.com/openssh/openssh-portable/blob/2709809fd616a0991dc18e3a58dea10fb383c3f0/ssh-agent.c#L824

https://github.com/openssh/openssh-portable/blob/2709809fd616a0991dc18e3a58dea10fb383c3f0/ssh-sk.c#L646-L648

So I guess there is something wrong in your OS that prevents ssh-agent from showing the passphrase dialog.

[MaxwellDPS]

Hmm, any ideas where to start? Im running Mac OS with the brew version of openssh. I dont see any other errors, I can have someone else with a Mac test tonight to see if this is unique. Ill update in a bit!

MacOS: 13.4
SSH version

OpenSSH_9.3p1, OpenSSL 1.1.1t  7 Feb 2023

[z4yx]

I don't have macOS to reproduce the problem. If you could build ssh-agent yourself, you may add some prints in the read_passphrase and find out which branch returns the empty string or NULL.
https://github.com/openssh/openssh-portable/blob/2709809fd616a0991dc18e3a58dea10fb383c3f0/readpass.c#L123

[z4yx]

@MaxwellDPS I've figured out that the key factor is an environment variable named SSH_ASKPASS. Set it to the path of the ssh-askpass program, which shows a passphrase input dialog. Then ssh-agent can work well with resident keys.

The ssh-askpass can be installed with package managers on Linux distributions. But I don't know how to get that on macOS. Maybe this project can help: https://github.com/theseal/ssh-askpass

[z4yx]

Here is the log of a successful authentication

debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key ED25519-SK comment "", 0 socket bindings, 0 constraints
debug3: identity_permitted: entering: key ED25519-SK comment "", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 2 allowed of 2 available keys
debug1: process_message: socket 1 (fd=4) type 13
debug1: process_sign_request2: entering
Confirm user presence for key ED25519-SK SHA256:2gKjOHfcuEzllcLfv+v2Fu8YgoAj9ym3aGBmYmjD0CE
debug3: start_helper: started pid=33040
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/local/libexec/ssh-sk-helper 
debug1: process_sign: ready to sign with key ED25519-SK, provider internal: msg len 32, compat 0x0
debug1: sshsk_sign: provider "internal", key ED25519-SK, flags 0x25
debug1: sk_probe: 1 device(s) detected
debug1: sk_probe: selecting sk by touch
debug1: check_sk_options: option uv is unknown
debug1: ssh_sk_sign: check_sk_options uv
debug1: sshsk_sign: sk_sign failed with code -3
debug1: ssh-sk-helper: Signing failed: incorrect passphrase supplied to decrypt private key
debug1: main: reply len 8
debug3: ssh_msg_send: type 5
debug1: client_converse: helper returned error -43
debug3: reap_helper: pid=33040
debug1: process_sign_request2: sshkey_sign: incorrect passphrase supplied to decrypt private key
debug3: start_helper: started pid=33049
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/local/libexec/ssh-sk-helper 
debug1: process_sign: ready to sign with key ED25519-SK, provider internal: msg len 32, compat 0x0
debug1: sshsk_sign: provider "internal", key ED25519-SK, flags 0x25 with-pin
debug1: sk_probe: 1 device(s) detected
debug1: sk_probe: selecting sk by touch
debug1: main: reply len 111
debug3: ssh_msg_send: type 5
debug3: reap_helper: pid=33049
debug1: process_sign_request2: good signature

[MaxwellDPS]

@z4yx Thank you for all the work you put in to get this resolved! Was able to get that working with ssh-ask pass! Seems the plist is broken for 13.4 but I'll get that sorted!