Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can't connect anymore with known_hosts=None after fe757eae9fb1df5002 #263

Closed
sanga opened this issue Mar 4, 2020 · 8 comments
Closed

can't connect anymore with known_hosts=None after fe757eae9fb1df5002 #263

sanga opened this issue Mar 4, 2020 · 8 comments

Comments

@sanga
Copy link

sanga commented Mar 4, 2020

minimum failing example I could come up with is opening ipython and running the following:

In [24]: async def go():
    ...:     async with connect('192.168.1.1', username='root', password='root') as c:
    ...:         result = await c.run('echo "Hello!"', check=True)
    ...:         print(result.stdout, end='')
    ...: await go()
    ...:
    ...:
    ...:
Hello!

In [25]: async def go():
    ...:     async with connect('192.168.1.1', username='root', password='root', known_hosts=None) as c:
    ...:         result = await c.run('echo "Hello!"', check=True)
    ...:         print(result.stdout, end='')
    ...: await go()
    ...:
    ...:
    ...:
    ...:
---------------------------------------------------------------------------
ConnectionLost                            Traceback (most recent call last)
<ipython-input-25-d6dae9cb6db8> in async-def-wrapper()
      6
      7
----> 8
      9

<ipython-input-25-d6dae9cb6db8> in go()
      3         result = await c.run('echo "Hello!"', check=True)
      4         print(result.stdout, end='')
----> 5 await go()
      6
      7

/usr/local/lib/python3.7/site-packages/asyncssh/misc.py in __aenter__(self)
    162
    163         async def __aenter__(self):
--> 164             self._result = await self._coro
    165             return await self._result.__aenter__()
    166

/usr/local/lib/python3.7/site-packages/asyncssh/connection.py in connect(host, port, tunnel, family, flags, local_addr, options, **kwargs)
   5645
   5646     return await _connect(host, port, loop, tunnel, family, flags, local_addr,
-> 5647                           conn_factory, 'Opening SSH connection to')
   5648
   5649

/usr/local/lib/python3.7/site-packages/asyncssh/connection.py in _connect(host, port, loop, tunnel, family, flags, local_addr, conn_factory, msg)
    168     # pylint: disable=broad-except
    169     try:
--> 170         await conn.wait_established()
    171     except Exception:
    172         conn.abort()

/usr/local/lib/python3.7/site-packages/asyncssh/connection.py in wait_established(self)
   1880         """Wait for connection to be established"""
   1881
-> 1882         await self._waiter
   1883
   1884     async def wait_closed(self):

ConnectionLost: Connection lost

I bisected this down to fe757ea So apparently something in that commit introduced this.

Not sure it's relevant but this is against Dropbear server v2017.75

@sanga sanga changed the title can't connect anymore with known_hosts=None after can't connect anymore with known_hosts=None after fe757eae9fb1df5002 Mar 4, 2020
@ronf
Copy link
Owner

ronf commented Mar 4, 2020

Can you get any logs from the Dropbear server to see if it is reporting an error? Also, can you increase the debug level to 2 and grab the logs from the client? The commands for that would be something like:

import logging
logging.basicConfig(level='DEBUG')
asyncssh.set_debug_level(2)

@sanga
Copy link
Author

sanga commented Mar 4, 2020

asyncssh logs:

against v2.1.0

In [8]: async def go():
   ...:     ...:     async with connect('192.168.1.1', username='root', password='root', known_hosts=None) as c:
   ...:     ...:         result = await c.run('echo "Hello!"', check=True)
   ...:     ...:         print(result.stdout, end='')
   ...:     ...: await go()
INFO:asyncssh:Opening SSH connection to 192.168.1.1, port 22
INFO:asyncssh:[conn=3] Connection to 192.168.1.1, port 22 succeeded
INFO:asyncssh:[conn=3]   Local address: 192.168.1.2, port 41662
DEBUG:asyncssh:[conn=3] Requesting key exchange
DEBUG:asyncssh:[conn=3]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,rsa2048-sha256,rsa1024-sha1
DEBUG:asyncssh:[conn=3]   Host key algs: ssh-ed25519-cert-v01@openssh.com,ssh-ed448-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,ssh-ed448,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ecdsa-sha2-1.3.132.0.10,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss
DEBUG:asyncssh:[conn=3]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour256,arcfour128,arcfour
DEBUG:asyncssh:[conn=3]   MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-md5-etm@openssh.com,hmac-sha2-256-96-etm@openssh.com,hmac-sha2-512-96-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha2-256-96,hmac-sha2-512-96,hmac-sha1-96,hmac-md5-96
DEBUG:asyncssh:[conn=3]   Compression algs: zlib@openssh.com,zlib,none
DEBUG:asyncssh:[conn=3] Received key exchange request
DEBUG:asyncssh:[conn=3]   Key exchange algs: curve25519-sha256@libssh.org,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,kexguess2@matt.ucc.asn.au
DEBUG:asyncssh:[conn=3]   Host key algs: ssh-rsa
DEBUG:asyncssh:[conn=3]   Client to server:
DEBUG:asyncssh:[conn=3]     Encryption algs: aes128-ctr,aes256-ctr
DEBUG:asyncssh:[conn=3]     MAC algs: hmac-sha1,hmac-sha2-256
DEBUG:asyncssh:[conn=3]     Compression algs: none
DEBUG:asyncssh:[conn=3]   Server to client:
DEBUG:asyncssh:[conn=3]     Encryption algs: aes128-ctr,aes256-ctr
DEBUG:asyncssh:[conn=3]     MAC algs: hmac-sha1,hmac-sha2-256
DEBUG:asyncssh:[conn=3]     Compression algs: none
DEBUG:asyncssh:[conn=3] Beginning key exchange
DEBUG:asyncssh:[conn=3]   Key exchange alg: curve25519-sha256@libssh.org
DEBUG:asyncssh:[conn=3]   Client to server:
DEBUG:asyncssh:[conn=3]     Encryption alg: aes256-ctr
DEBUG:asyncssh:[conn=3]     MAC alg: hmac-sha2-256
DEBUG:asyncssh:[conn=3]     Compression alg: none
DEBUG:asyncssh:[conn=3]   Server to client:
DEBUG:asyncssh:[conn=3]     Encryption alg: aes256-ctr
DEBUG:asyncssh:[conn=3]     MAC alg: hmac-sha2-256
DEBUG:asyncssh:[conn=3]     Compression alg: none
DEBUG:asyncssh:[conn=3] Requesting service ssh-userauth
DEBUG:asyncssh:[conn=3] Completed key exchange
DEBUG:asyncssh:[conn=3] Request for service ssh-userauth accepted
INFO:asyncssh:[conn=3] Beginning auth for user root
DEBUG:asyncssh:[conn=3] Remaining auth methods: publickey,password
DEBUG:asyncssh:[conn=3] Trying password auth
INFO:asyncssh:[conn=3] Auth for user root succeeded
DEBUG:asyncssh:[conn=3, chan=0] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=3, chan=0] Requesting new SSH session
DEBUG:asyncssh:[conn=3, chan=0]   Initial recv window 2097152, packet size 32768
DEBUG:asyncssh:[conn=3, chan=0]   Initial send window 24576, packet size 32759
INFO:asyncssh:[conn=3, chan=0]   Command: echo "Hello!"
DEBUG:asyncssh:[conn=3, chan=0] Reading from channel started
DEBUG:asyncssh:[conn=3, chan=0] Received 7 data bytes
DEBUG:asyncssh:[conn=3, chan=0] Received EOF
INFO:asyncssh:[conn=3, chan=0] Received exit status 0
INFO:asyncssh:[conn=3, chan=0] Received channel close
INFO:asyncssh:[conn=3, chan=0] Channel closed
Hello!
INFO:asyncssh:[conn=3] Closing connection
INFO:asyncssh:[conn=3] Sending disconnect: Disconnected by application (11)
INFO:asyncssh:[conn=3] Connection closed

against v2.2.0

In [5]: async def go():
   ...:     ...:     async with connect('192.168.1.1', username='root', password='root', known_hosts=None) as c:
   ...:     ...:         result = await c.run('echo "Hello!"', check=True)
   ...:     ...:         print(result.stdout, end='')
   ...:     ...: await go()
INFO:asyncssh:Opening SSH connection to 192.168.1.1, port 22
INFO:asyncssh:[conn=0] Connection to 192.168.1.1, port 22 succeeded
INFO:asyncssh:[conn=0]   Local address: 192.168.1.2, port 41670
DEBUG:asyncssh:[conn=0] Requesting key exchange
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,rsa2048-sha256,rsa1024-sha1
DEBUG:asyncssh:[conn=0]   Host key algs: sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-ed448-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ssh-ed448,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ecdsa-sha2-1.3.132.0.10,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss
DEBUG:asyncssh:[conn=0]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour256,arcfour128,arcfour
DEBUG:asyncssh:[conn=0]   MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-md5-etm@openssh.com,hmac-sha2-256-96-etm@openssh.com,hmac-sha2-512-96-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha2-256-96,hmac-sha2-512-96,hmac-sha1-96,hmac-md5-96
DEBUG:asyncssh:[conn=0]   Compression algs: zlib@openssh.com,zlib,none
DEBUG:asyncssh:[conn=0] Received key exchange request
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256@libssh.org,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,kexguess2@matt.ucc.asn.au
DEBUG:asyncssh:[conn=0]   Host key algs: ssh-rsa
DEBUG:asyncssh:[conn=0]   Client to server:
DEBUG:asyncssh:[conn=0]     Encryption algs: aes128-ctr,aes256-ctr
DEBUG:asyncssh:[conn=0]     MAC algs: hmac-sha1,hmac-sha2-256
DEBUG:asyncssh:[conn=0]     Compression algs: none
DEBUG:asyncssh:[conn=0]   Server to client:
DEBUG:asyncssh:[conn=0]     Encryption algs: aes128-ctr,aes256-ctr
DEBUG:asyncssh:[conn=0]     MAC algs: hmac-sha1,hmac-sha2-256
DEBUG:asyncssh:[conn=0]     Compression algs: none
DEBUG:asyncssh:[conn=0] Beginning key exchange
DEBUG:asyncssh:[conn=0]   Key exchange alg: curve25519-sha256@libssh.org
INFO:asyncssh:[conn=0] Connection lost
INFO:asyncssh:[conn=0] Aborting connection

So it looks like it bails out during key-exchange. At a glance I didn't spot differences there...

@sanga
Copy link
Author

sanga commented Mar 4, 2020

dropbear logs thusly:

v2.2.0
Wed Mar  4 10:03:53 2020 authpriv.info dropbear[2529]: Child connection from 192.168.1.2:41756
Wed Mar  4 10:03:53 2020 authpriv.info dropbear[2529]: Exit before auth: No matching algo hostkey

v2.1.0
Wed Mar  4 10:04:20 2020 authpriv.info dropbear[2530]: Child connection from 192.168.1.2:41770
Wed Mar  4 10:04:20 2020 authpriv.notice dropbear[2530]: Password auth succeeded for 'root' from 192.168.1.2:41770
Wed Mar  4 10:04:20 2020 authpriv.info dropbear[2530]: Exit (root): Disconnect received

So Exit before auth: No matching algo hostkey piques my interest

@ronf
Copy link
Owner

ronf commented Mar 5, 2020

Thank you for the additional info!

The "No matching algo hostkey" is definitely interesting. Looking at the two host key lists, v2.2.0 is a strict superset of 2.1.0, adding the following algorithms:

sk-ssh-ed25519-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ecdsa-sha2-nistp256@openssh.com

So, if the Dropbear server found a match in v2.1.0, it should also be able to find a match in the v2.2.0 case. However, I wonder if perhaps the problem could be related to the length of the host key algorithm list. With the added algorithms, the length of the list is 579 bytes in v.2.2.0 vs. 437 bytes in v2.1.0. Perhaps DropBear has a length limit on that.

In the connect() call, could you try setting server_host_key_algs=['ssh-rsa'] and see if that still has the error? If that works, try using a longer list, like

server_host_key_algs=['ssh-ed25519-cert-v01@openssh.com', 'ssh-ed448-cert-v01@openssh.com', 'ecdsa-sha2-nistp521-cert-v01@openssh.com', 'ecdsa-sha2-nistp384-cert-v01@openssh.com', 'ecdsa-sha2-nistp256-cert-v01@openssh.com', 'ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com', 'ssh-rsa-cert-v01@openssh.com', 'ssh-dss-cert-v01@openssh.com', 'ssh-ed25519', 'ssh-ed448', 'ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-1.3.132.0.10', 'rsa-sha2-256', 'rsa-sha2-512', 'ssh-rsa', 'ssh-dss']

This is the list from v2.1.0. If that works, it might also be interesting to try:

server_host_key_algs=['ssh-rsa',' sk-ssh-ed25519-cert-v01@openssh.com', 'sk-ecdsa-sha2-nistp256-cert-v01@openssh.com', 'ssh-ed25519-cert-v01@openssh.com', 'ssh-ed448-cert-v01@openssh.com', 'ecdsa-sha2-nistp521-cert-v01@openssh.com', 'ecdsa-sha2-nistp384-cert-v01@openssh.com', 'ecdsa-sha2-nistp256-cert-v01@openssh.com', 'ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com', 'ssh-rsa-cert-v01@openssh.com', 'ssh-dss-cert-v01@openssh.com', 'sk-ssh-ed25519@openssh.com', 'sk-ecdsa-sha2-nistp256@openssh.com', 'ssh-ed25519', 'ssh-ed448', 'ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-1.3.132.0.10', 'rsa-sha2-256', 'rsa-sha2-512', 'ssh-dss']

This is the full list from v2.2.0, but with 'ssh-rsa' moved to the front. I don't know if this would help or not if it is a buffer length issue, but it'd be an interesting data point.

@ronf
Copy link
Owner

ronf commented Mar 5, 2020

I dug around a bit in the Dropbear source, and I think I may have found the issue. The buf_match_algo() function has some limits on both the number of proposed algorithms and the maximum name length of each algorithm. While the name length is ok (up to 64 bytes per algo name), the limit on the maximum proposed algorithms is 20 in sysoptions.h:

#define MAX_PROPOSED_ALGO 20

This is later used in but_match_algo() in common-also.c and it looks like anything beyond the first 20 algorithms might not be matched, as they don't fit in the fixed-size array it allocates:

algo_type * buf_match_algo(buffer* buf, algo_type localalgos[],
		enum kexguess2_used *kexguess2, int *goodguess)
{
	char * algolist = NULL;
	const char *remotenames[MAX_PROPOSED_ALGO], *localnames[MAX_PROPOSED_ALGO];
	unsigned int len;
	unsigned int remotecount, localcount, clicount, servcount, i, j;
	algo_type * ret = NULL;
	const char **clinames, **servnames;

	if (goodguess) {
		*goodguess = 0;
	}

	/* get the comma-separated list from the buffer ie "algo1,algo2,algo3" */
	algolist = buf_getstring(buf, &len);
	TRACE(("buf_match_algo: %s", algolist))
	if (len > MAX_PROPOSED_ALGO*(MAX_NAME_LEN+1)) {
		goto out;
	}

	/* remotenames will contain a list of the strings parsed out */
	/* We will have at least one string (even if it's just "") */
	remotenames[0] = algolist;
	remotecount = 1;
	for (i = 0; i < len; i++) {
		if (algolist[i] == '\0') {
			/* someone is trying something strange */
			goto out;
		}
		if (algolist[i] == ',') {
			algolist[i] = '\0';
			remotenames[remotecount] = &algolist[i+1];
			remotecount++;
		}
		if (remotecount >= MAX_PROPOSED_ALGO) {
			break;
		}
	}

        ...
}

The overall buffer length check is ok, because 20*(64+1) is 1300 bytes, which is bigger than what AsyncSSH is sending. However, the full list of algorithms in v2.2.0 is now 22, meaning the 'ssh-rsa' and 'ssh-dss' are beyond the 20 algorithms that Dropbear is willing to match against.

I'm guessing that moving ssh-rsa earlier in the list will help, as would reducing the list of algorithms that AsyncSSH sends.

On the Dropbear side, increasing MAX_PROPOSED_ALGO would also be a pretty simple fix for this.

@sanga
Copy link
Author

sanga commented Mar 8, 2020

Thanks for the thorough investigation! Certainly looks like that's the issue.

For the record against Dropbear all of the following work:

server_host_key_algs=['ssh-rsa']

server_host_key_algs=['ssh-ed25519-cert-v01@openssh.com', 'ssh-ed448-cert-v01@openssh.com', 'ecdsa-sha2-nistp521-cert-v01@openssh.com', 'ecdsa-sha2-nistp384-cert-v01@openssh.com', 'ecdsa-sha2-nistp256-cert-v01@openssh.com', 'ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com', 'ssh-rsa-cert-v01@openssh.com', 'ssh-dss-cert-v01@openssh.com', 'ssh-ed25519', 'ssh-ed448', 'ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-1.3.132.0.10', 'rsa-sha2-256', 'rsa-sha2-512', 'ssh-rsa', 'ssh-dss']

server_host_key_algs=['ssh-rsa', 'sk-ssh-ed25519-cert-v01@openssh.com', 'sk-ecdsa-sha2-nistp256-cert-v01@openssh.com', 'ssh-ed25519-cert-v01@openssh.com', 'ssh-ed448-cert-v01@openssh.com', 'ecdsa-sha2-nistp521-cert-v01@openssh.com', 'ecdsa-sha2-nistp384-cert-v01@openssh.com', 'ecdsa-sha2-nistp256-cert-v01@openssh.com', 'ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com', 'ssh-rsa-cert-v01@openssh.com', 'ssh-dss-cert-v01@openssh.com', 'sk-ssh-ed25519@openssh.com', 'sk-ecdsa-sha2-nistp256@openssh.com', 'ssh-ed25519', 'ssh-ed448', 'ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-1.3.132.0.10', 'rsa-sha2-256', 'rsa-sha2-512', 'ssh-dss']

So apparently just moving ssh-rsa up the list would be enough.

@ronf
Copy link
Owner

ronf commented Mar 8, 2020

Great - thanks for the confirmation!

Unfortunately, unless I introduced code which checks the peer's version string to see if it is a Dropbear server and re-orders the host key algs, it'll be difficult to automatically work around this AsyncSSH side. I think explicitly specifying server_host_key_algs might be your best bet in this case.

Alternately, it looks like Dropbear understands ECDSA algorithms and those are earlier in the list. So, you might not see this issue when the Dropbear server is using an EC host key instead of an RSA one.

@sanga
Copy link
Author

sanga commented Mar 9, 2020

Thanks, I wound up solving this by setting server_host_key_algs manually as you suggest. I guess there is not much that can reasonably be done from the asyncssh side on this one so => closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants