When using the config generated by tsh config, ssh to agentless nodes on a remote cluster fails known hosts check

[timothyb89]

Expected behavior:

When a node on a remote (leaf) cluster is joined per our Agentless OpenSSH guide, connection attempts using the regular ssh client should work without additional TOFU prompts to trust the remote host.

Current behavior:

SSH to Teleport nodes on the root cluster, agentless nodes on the root cluster, and Teleport nodes on the leaf cluster all work as expected.

However, agentless nodes on the leaf cluster do not match any CAs listed in the generated ~/.tsh/known_hosts and trigger a TOFU check. You can still connect, however the RSA fingerprint changes on every connection, so subsequent connection attempts will fail until the entry is removed:

~ 3s ❯ ssh -p 22 -A -F ~/.ssh/tsh_config tim@teleport-leaf-openssh.teleport-leaf.ethernet.fyi
The authenticity of host 'teleport-leaf-openssh.teleport-leaf.ethernet.fyi (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:wjXsOP7NrG2KCak3cfzBZY3nWbkJTB/KiFxo5MjbVbw.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^CERROR: context canceled

~ ❯ ssh -p 22 -A -F ~/.ssh/tsh_config tim@teleport-leaf-openssh.teleport-leaf.ethernet.fyi
The authenticity of host 'teleport-leaf-openssh.teleport-leaf.ethernet.fyi (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:eH1kIIo5fpJ4BtpEARi103kki43EmU8j1FwFy8M0H68.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^CERROR: context canceled

~ ❯ ssh -p 22 -A -F ~/.ssh/tsh_config tim@teleport-leaf-openssh.teleport-leaf.ethernet.fyi
The authenticity of host 'teleport-leaf-openssh.teleport-leaf.ethernet.fyi (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:cIqMdfoF3pRBnO3qVjQ3k7GYLIJ5EHo1h/CCTyZSozk.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'teleport-leaf-openssh.teleport-leaf.ethernet.fyi' (RSA) to the list of known hosts.
Linux teleport-leaf-openssh 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri May 31 17:00:28 2024 from 192.168.10.91
tim@teleport-leaf-openssh:~$ 
logout
Connection to teleport-leaf-openssh.teleport-leaf.ethernet.fyi closed.

~ 3s ❯ ssh -p 22 -A -F ~/.ssh/tsh_config tim@teleport-leaf-openssh.teleport-leaf.ethernet.fyi
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:O/5jMTzuyZr0laPEed0x+VNaayb4nkdix2R718s+VFA.
Please contact your system administrator.
Add correct host key in /home/tim/.tsh/known_hosts to get rid of this message.
Offending RSA key in /home/tim/.tsh/known_hosts:3
You can use following command to remove the offending key:
ssh-keygen -R teleport-leaf-openssh.teleport-leaf.ethernet.fyi -f /home/tim/.tsh/known_hosts
Host key for teleport-leaf-openssh.teleport-leaf.ethernet.fyi has changed and you have requested strict checking.
Host key verification failed.

Connecting directly to the leaf cluster with tsh login --proxy=teleport-leaf.ethernet.fyi:443 and regenerating the config with tsh config > ~/.ssh/tsh_config works as expected, so this only impacts remote clusters.

The ~/.tsh/known_hosts contains the following:

@cert-authority teleport.ethernet.fyi,teleport.ethernet.fyi,*.teleport.ethernet.fyi ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEYMFJ8AQtutkV+4bS2hu68xk4WptMLIWeYzibUOINQPzOXIp4Z5W4lDSmkZsixOFE9XNbkS5emHH5B0frCMzTgSnr8MRS5HzUyGaovwp7GpBFgDdFj+bQ34IKAYMBOE0QKI+HW5z2eFXEI8KgzCwCUO9h4v78FDYOURmNRq8XH4cWjBFL9ly48GzHn33z8046F41Rs8t+DFItxkPiT1z4aq4B5HpvBYBlDN8PBo1N7LhbHYwxI+u8SWP5ICQUFoRzA9Za0AFAZWb7C2PFi9ZcGsblk2bfh0J1WDvnA+dcck62oIeRL8vAb2JNU/3krAU0WDp5Ua4eriU9LVCgnpzd type=host
@cert-authority teleport.ethernet.fyi,teleport-leaf.ethernet.fyi,*.teleport-leaf.ethernet.fyi ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQMx0wb56Nkz69QnAzCwzZweSH29g17S4gPCWgYASkmIXShtr0tnXgYXMUWfVS6vlG7eaatByFljFc4ToGD5YtNkb7jQVqb9EEMdMzZSjbvBUsR8Rg9pryKD8CDX/jjZ8U6P9pU8UM18+OUeVLFaoVcszM+pZHuWblLmmVts4ixRTYjSR4RZShWP1qzuKy5qM3SO+HdLRpURn0wcdA/g8hhGX3dTiWI7nDrmr4078eWfxrtX9ut2tWB95vtM8loh60jBPGmxMOLDp0JS1jPvTeOh4pEbBj9b+cQ1umYua7bOfG8NNBWTy2xiDMN+0Rot7BXi1I/YictkK4ONk9gPHp type=host

SSH client debug log

ssh -vvvvv -p 22 -A -F ~/.ssh/tsh_config tim@teleport-leaf-openssh.teleport-leaf.ethernet.fyi
OpenSSH_9.6p1, OpenSSL 3.1.4 24 Oct 2023
debug1: Reading configuration data /home/tim/.ssh/tsh_config
debug1: /home/tim/.ssh/tsh_config line 15: Applying options for *.teleport-leaf.ethernet.fyi
debug1: /home/tim/.ssh/tsh_config line 22: Applying options for *.teleport-leaf.ethernet.fyi
debug3: channel_clear_timeouts: clearing
debug1: Executing proxy command: exec "/home/tim/projects/teleport/build/tsh" proxy ssh --cluster=teleport-leaf.ethernet.fyi --proxy=teleport.ethernet.fyi:443 tim@teleport-leaf-openssh.teleport-leaf.ethernet.fyi:22
debug1: identity file /home/tim/.tsh/keys/teleport.ethernet.fyi/tim type 0
debug1: certificate file /home/tim/.tsh/keys/teleport.ethernet.fyi/tim-ssh/teleport.ethernet.fyi-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_9.6
debug1: Remote protocol version 2.0, remote software version Go
debug1: compat_banner: no match: Go
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to teleport-leaf-openssh.teleport-leaf.ethernet.fyi:22 as 'tim'
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com
debug2: host key algorithms: rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com
debug2: ciphers ctos: aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug3: kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:PpDEuAKU857l7DKKWhCP3zZ4CbyttfnYIM8sFy0REs8, serial 0 ID "" CA ssh-rsa SHA256:lJdrXuMUDf69QrlpeDV5IOXaO4QO8jTxbfw0Ifu5/VQ valid after 2024-05-31T17:24:34
debug2: Server host certificate hostname: teleport-leaf-openssh.teleport.ethernet.fyi
debug2: Server host certificate hostname: teleport-leaf-openssh
debug2: Server host certificate hostname: localhost
debug2: Server host certificate hostname: 127.0.0.1
debug2: Server host certificate hostname: ::1
debug2: Server host certificate hostname: 1bb60005-f20e-4fe3-bfd8-13da551fc2d2.teleport-leaf.ethernet.fyi
debug2: Server host certificate hostname: 192.168.10.93
debug3: record_hostkey: found ca key type RSA in file /home/tim/.tsh/known_hosts:2
debug3: record_hostkey: found ca key type RSA in file /home/tim/.tsh/known_hosts:3
debug3: load_hostkeys_file: loaded 2 keys from teleport-leaf-openssh.teleport-leaf.ethernet.fyi
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: No matching CA found. Retry with plain key
debug3: hostkeys_find_by_key_hostfile: trying user hostfile "/home/tim/.tsh/known_hosts"
debug3: hostkeys_foreach: reading file "/home/tim/.tsh/known_hosts"
debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts does not exist
debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts2"
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts2 does not exist
The authenticity of host 'teleport-leaf-openssh.teleport-leaf.ethernet.fyi (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:PpDEuAKU857l7DKKWhCP3zZ4CbyttfnYIM8sFy0REs8.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

Bug details:

• Teleport version: v16.0.0-alpha.3. Agentless nodes running plain Debian 12, openssh 9.2p1.

[strideynet]

This came up during last test plan as well :/

#36801