Confusing error message when trying to verify unknown host key types

[CiaranG]

Paramiko only recognises ssh-rsa and ssh-dss keys when reading known_hosts. However, newer versions of openssh use additional key types which Paramiko silently ignores, resulting in very confusing missing host key errors which occur when attempting to connect with Paramiko, even though the same connection via the command line ssh works perfectly.

From the openssh documentation:

For protocol version 2 the keytype is “ecdsa-sha2-nistp256”, “ecdsa-sha2-nistp384”, “ecdsa-sha2-nistp521”, “ssh-dss” or “ssh-rsa”

[guettli]

I had this error: I can connect from the shell, but paramiko says "Unknown server workdevel114".

There were two similar entries in known_hosts:

user@host> grep workdevel114 ~/.ssh/known_hosts
workdevel114 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC8qGbuI
user@host> grep I1BaBodi7sKWLfV8Eh+De80Th7HFLD4WiJWo57THl0Q+QcopUaU3pF ~/.ssh/known_hosts
workdevel114 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC8qGbuI1BaBodi7sK...
|1|f/auQ9nY5dFbVtOdY3ocjtVO9dM=|esvazUDTT3VIcLk9DxmPI6FZt1s= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC8qGbuI1BaBodi7sKWLfV8Eh+De80Th7HFLD4...

The seconds entry (|1|....) seems to confuse paramiko. I get: Unknown server workdevel114

Version: paraiko-version: 1.7.7.1-1ubuntu1

[bitprophet]

Renamed ticket to be more obvious. There's two problems here:

1. Paramiko doesn't understand ECDSA host keys. There's another ticket for that: Add ECDSA support to paramiko #88.
2. Paramiko silently ignores unknown key types instead of saying "I have no idea what this key is" or "I and the server do not agree on available key ciphers". That's this ticket now.

I haven't traced this all the way yet but I think this section of Transport is linked.

My untested guess so far is:

• The linked lines above effectively say "We're assuming that the server key type is the first key type in [list of types Paramiko prefers that the server also supports]."
• I see nothing backing up that assumption: if a host sshd supports both RSA (known to Paramiko) and ECDSA (unknown), that list will end up just being [RSA,], Paramiko then grabs the first (only) item in that list (RSA) and proceeds as if the server has given it an RSA host key.
• However, AFAIK, the server may well have handed us an ECDSA host key, which I assume is what happened for @CiaranG and others.
• Elsewhere in Transport, that key type is used to obtain a key generation class, into which is passed the original host key data.
• If my theory is correct, this basically means Paramiko is calling e.g. RSAKey(<something that is not RSA here>).
• However, RSAKey looks like it should bail out in this instance with Invalid key, which nobody has reported -- so I need to double check the above.

---

@CiaranG if you can provide a foolproof way to recreate this, that will help me test any fixes. My guess is you've got ECDSA host keys from an Ubuntu 12.04 (or 12.10?) server running a newer OpenSSH, stored into your known_hosts via using the normal ssh program, and are then trying to connect to the same server via Paramiko. Is that right?

[bitprophet]

This may also be related to #87 since it also deals with the |1|... style host keys. Unclear to me now whether we're dealing with actual brand new key types, or if this is "just" the (older, afaik) plaintext vs hashed known_host file/line formats.

[guettli]

@bitprophet thank you for looking at this. Tell me I can help.

[glasserc]

I can't say I know exactly the situation that @CiaranG was facing, but I can say that I encountered a situation that sounds like this one, and it doesn't match the description that @bitprophet describes above. I'm running Ubuntu 12.10 and I just installed paramiko 1.10.0 using pip. I'm trying to connect to localhost using paramiko. Here's the output when I set logging to DEBUG.

starting thread (client mode): 0x12f4390L
Connected (version 2.0, client OpenSSH_6.0p1)
kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss', 'ecdsa-sha2-nistp256'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
Switch to new keys ...
Rejecting ssh-rsa host key for localhost: f8ed063255503f72e1f49195498117c7

In fact, it seems like it's negotiating the host key process correctly, because that is the correct fingerprint for my host RSA key:

$ sudo ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key
2048 f8:ed:06:32:55:50:3f:72:e1:f4:91:95:49:81:17:c7  root@silver (RSA)

It was still pretty surprising to me, though, because I told paramiko to load my ~/.ssh/known_hosts file. Here's the output when I print sshclient.get_host_keys() [Editor's note: reformatted for key readability]:

|1|5b8M50I0fyAkX13+ms33GQikegw=|NGPM76aQrMzBSmFfNuXoLVfOR4M=
|1|CdEHEiCkdem8vcQNJrqcgS9/fvQ=|NCnNW9onF6Q6Bu+IIc0NX7Uh9AE=
|1|M1a64VqwOHRhEimc/XSrO7ZEco4=|Yb6MkB5XIlRVltZiiA366XIxkiM=
|1|OJxCv4eRXp0lYMJDJ8WztGUf5qY=|BLXNyY+u98L66YrI+UyW0TvchkQ=
|1|PPJUhG109txjO6vpta+2ZQrcX/I=|4g5CQihv7TVGcEUpPWzJBuWEwHQ=
|1|QRZnn/NKe9arYzyUcteuxSxDgyc=|qWQVgCGEZu2PpugfJ+t3eY61Z88=
|1|RcI4hyYPH0cj96GtAL8P4tL8oFI=|finEKljDR2iDNGTfMKU1Hr+4dT8=
|1|RgkJEYtbSiJJBs4JoGQiYq2XO3g=|pBUS90JD5ZMQYIvb6TLPh5ZwkU8=
|1|aaUUf5MJg/o6Hv0fhUv5c7VSwnk=|4SksewVI6xGGkYSsYAAzPiyNHUE=
|1|cCCjsEccaY9g8Ye1Ejf00AEoBcg=|tZIVKOjLpU0Q9BUaraUQbqxeRcI=
|1|eL9wGQohWPWecvGVH9+l3QskoUk=|dTcIWjdEr8dT1zEbjfcsUkEv6Ng=
|1|emMD/lK1PpFkv85nN/qJUF7D3bg=|z3MiH1aWTpYCFM69hkt0/KQO9Xs=
|1|kv6/a6UQ8lyjhb9bZkFo63kHBz8=|2/tFbR4jS09eYabP4Sbs3jPkoPw=
|1|nd28tsxS0jTpPcCO6neiA3anaKc=|zQvF2VSnL+THXJjK/qImvQxChFw=
|1|o23ok2ExjH0GvEqOZlIEenfLPdM=|8QxQH8LwQJBIP4fmnIypBziKGJg=
|1|p04jt+IFMoBRWDT1pM8kPLMt048=|3mfzbSWtTR2tuLf+fS5llCXVWzU=
|1|rijVcGrGWbD+jgaa5ZX4ECjl648=|XeuaNPqKzWAzo3ssz+OErLZCyho=
|1|uF5qqcYKGE62vX4Jsg8rZsJDTak=|It2244JvvTLbYjMI1BkAnj1G6qQ=

So it definitely seems to be loading the keys in my known_hosts file. And those keys seem to be correct because I don't have any trouble SSH'ing in normally:

$ ssh localhost
ethan@localhost's password: 

Fortunately, though, I know that the key for localhost is the last one added to my known_hosts file, since I just accepted the key earlier today. The entry says:

|1|xnahrF8oXFAlTlmGjzsuzsy/5gs=|VhZf/UzvPmC98V4BDlQE/3clMjs= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNlkxlK97HWjsBai2WzY9vKS4xJD+TOlqHaH4GQdDjSqwaERFIkQCWeHV8JwbQnIYOpGFPOTq09gW17x9Zc1sU8=

Indeed, although there are a bunch of ecdsa-sha2-nistp256 keys in this file, none seem to show up in the HostKeys output above. So it seems like paramiko is silently ignoring ECDSA keys when they're being read in the known_hosts file, and getting the RSA key from the server, which isn't in my known_hosts file, and therefore complaining that the key is unknown.

Based on this analysis, I don't think this bug merits any fix more complicated than emitting a message of level WARNING when we encounter a key of a type that we're not familiar with (line

paramiko/paramiko/hostkeys.py

Line 81 in 01aaf70

return None

). Fixing paramiko to accept ECDSA keys is, of course, a good idea too.

[bitprophet]

@glasserc Thanks for the detailed notes & the related pull requests! Your situation does still feel strongly related, though I do wonder why you and the OP appear to differ re: the host keys offered by the target host and/or stored in known_hosts.

[bitprophet]

Between the newer PR from @glasserc and the existing PRs for ECDSA support & other fixes, I think this guy is now redundant; closing.

[zeugmatis]

Hello, I encountered this issue and wanted to post a workaround here. The issue was indeed the ssh server sending ecdsa keys. On my debian Wheezy system I disabled ecdsa by commenting out a single line in /etc/ssh/sshd_config:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key

HostKey /etc/ssh/ssh_host_ecdsa_key

Restarted sshd, and it was back to using RSA. There were some ecdsa keys in my known_hosts file so I just deleted it to reset, and logged in manually to recreate the keys. From there, paramiko worked perfectly as expected, with RSA host key checking.

[mpdehaan]

I've encountered a user report of paramiko's load and save methods not being able to write a ecdsa-sha2-nistp256 back to a known host file in python-paramiko 1.7.7.1-3

Can you let me know if you think that version should be able to load/use and save that key back successfully?

[Tatsh]

I upgraded a machine and the machine changed keys as well as enabled ECDSA. I copied the machine's /etc/ssh/ssh_host_rsa_key.pub contents into ~/.ssh/config:

hostname ssh-rsa ..

Yes, paramiko first fails to negotiate the ECDSA connection, but then falls back to RSA. I do not mind doing this manually as I would much rather get the key manually than auto-accept.

2014-07-01 22:37:18,509 - paramiko.hostkeys - INFO - Unable to handle key of type ssh-ed25519
2014-07-01 22:37:18,509 - paramiko.hostkeys - INFO - Unable to handle key of type ssh-ed25519
2014-07-01 22:37:18,509 - paramiko.hostkeys - INFO - Unable to handle key of type ssh-ed25519
2014-07-01 22:37:18,509 - paramiko.hostkeys - INFO - Unable to handle key of type ssh-ed25519
2014-07-01 22:37:18,509 - paramiko.hostkeys - INFO - Unable to handle key of type ssh-ed25519
2014-07-01 22:37:18,790 - paramiko.transport - DEBUG - starting thread (client mode): 0x64b17d90
2014-07-01 22:37:18,841 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_6.6)
2014-07-01 22:37:18,842 - paramiko.transport - DEBUG - kex algos:['curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman
-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss', 'ecdsa-sha2-nistp256'
, 'ssh-ed25519'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'chacha20-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', '
cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'chacha20
-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5-etm@op
enssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-e
tm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2
-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com
', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-etm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm
@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1
-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
2014-07-01 22:37:18,843 - paramiko.transport - DEBUG - Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
2014-07-01 22:37:18,843 - paramiko.transport - DEBUG - using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-
sha1, remote hmac-sha1; compression: local zlib@openssh.com, remote zlib@openssh.com
2014-07-01 22:37:18,896 - paramiko.transport - DEBUG - Switch to new keys ...
2014-07-01 22:37:18,897 - paramiko.transport - DEBUG - Trying SSH agent key b'7a2e2ea2c7a291cf199bd42d57019545'
2014-07-01 22:37:18,900 - paramiko.transport - DEBUG - userauth is OK
2014-07-01 22:37:18,905 - paramiko.transport - INFO - Authentication (publickey) failed.
2014-07-01 22:37:18,905 - paramiko.transport - DEBUG - Trying SSH agent key b'ae8b03ba673f117e076bacfeb06a4542'
2014-07-01 22:37:18,907 - paramiko.transport - DEBUG - userauth is OK
2014-07-01 22:37:18,917 - paramiko.transport - INFO - Authentication (publickey) successful!

[bitprophet]

@mpdehaan - yes, 1.7.7 is too old and unsupported now, you'll need 1.13+ for ecdsa support to function, sorry!

@Tatsh there's a few open tickets about interplay between multiple key types, I think your issue probably falls under one of them. Please keep an eye out for upcoming bugfix/feature releases and see if they help. If not, please open a new ticket & link to your comment in this one. Thanks!

[rbjorklin]

Just wanted to chime in and say that I'm seeing the issue with ECDSA keys as well and that @zeugmatis workaround is still valid.

[zeugmatis]

Oh good makes me happy to have helped someone - thanks for that! :-)

Mark

On Dec 16, 2014, at 3:35 AM, Robin notifications@github.com wrote:

Just wanted to chime in and say that I'm seeing the issue with ECDSA keys as well and that @zeugmatis solution worked for me too.

—
Reply to this email directly or view it on GitHub.

[benson-basis]

I'm trying to connect to a brand-new EC2 instance with a ECDSA key, and the connection fails after the messages about the ECDSA keys.

INFO:paramiko.hostkeys:Unable to handle key of type ssh-ed25519
INFO:paramiko.hostkeys:Unable to handle key of type ssh-ed25519
Traceback (most recent call last):
File "/Users/benson/Library/Application Support/IntelliJIdea14/python/helpers/pydev/pydevd.py", line 2358, in
globals = debugger.run(setup['file'], None, None, is_module)
File "/Users/benson/Library/Application Support/IntelliJIdea14/python/helpers/pydev/pydevd.py", line 1778, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Users/benson/Library/Application Support/IntelliJIdea14/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/benson/x/rosapi-boot/images/create_image.py", line 136, in
manager.process()
File "/Users/benson/x/rosapi-boot/images/create_image.py", line 122, in process
self.__connect_to_instance()
File "/Users/benson/x/rosapi-boot/images/create_image.py", line 76, in __connect_to_instance
hostname=self.instance.private_ip_address
File "/usr/local/lib/python3.4/site-packages/paramiko/client.py", line 251, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/usr/local/lib/python3.4/site-packages/paramiko/util.py", line 270, in retry_on_signal
return function()
File "/usr/local/lib/python3.4/site-packages/paramiko/client.py", line 251, in
retry_on_signal(lambda: sock.connect(addr))
ConnectionRefusedError: [Errno 61] Connection refused

[bitprophet]

Those seem unrelated to me, @benson-basis - the hostkeys errors are Paramiko trying to load your local host-keys database and being unable to parse some of the host keys; the 'Connection refused' error should be a straight up "that port is literally not talking to me" error.

If the problem was in the actual host-key exchange, you'd be getting SSHErrors or similar from Paramiko itself, not ConnectionRefusedError. My guess is your code is trying to talk to the EC2 instance before its sshd is operational. (If this specific host is one of the ones with the ed25519 keys, though, you would then receive actual SSH level errors...)

[benson-basis]

Yea, I'm sorry. I got turned around in my early debugging process. All is well @bitprophet .