"Connection lost" with 2.2.1, works with 1.12.2 (logs included)

[eLvErDe]

Hello,

I wanted to upgrade to the newer version but my code isn't connecting anymore.
Here is a narrowed down script reproducing the issue:

import logging
import asyncio

import asyncssh

if __name__ == "__main__":

    async def test_coro() -> None:
        """ Test coroutine showing the Connection reset by peer bug """

        async with asyncssh.connect("192.168.2.2", port=22, username="root", known_hosts=None) as conn:
            result = await conn.run("ls /")
            logging.info(" --- ".join(result.stdout.splitlines()))

    logging.basicConfig(level=logging.DEBUG, format="%(asctime)s %(levelname)-8s [%(name)s] %(message)s")

    asyncio.run(test_coro())

With 1.12.2 it works as expected:

2020-06-20 20:22:42,082 DEBUG    [asyncio] Using selector: EpollSelector
2020-06-20 20:22:42,084 INFO     [asyncssh] Opening SSH connection to 192.168.2.2, port 22
2020-06-20 20:22:42,089 INFO     [asyncssh] [conn=0] Connection to 192.168.2.2, port 22 succeeded
2020-06-20 20:22:42,090 INFO     [asyncssh] [conn=0]   Local address: 192.168.2.3, port 43396
2020-06-20 20:22:42,099 DEBUG    [asyncssh] [conn=0] Requesting key exchange
/usr/lib/python3/dist-packages/asyncssh/crypto/pyca/ec.py:164: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  return pub.encode_point()
2020-06-20 20:22:42,144 DEBUG    [asyncssh] [conn=0] Beginning key exchange
/usr/lib/python3/dist-packages/asyncssh/crypto/pyca/ec.py:170: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self._priv_key.curve, peer_public).public_key(backend)
2020-06-20 20:22:42,610 DEBUG    [asyncssh] [conn=0] Completed key exchange
2020-06-20 20:22:42,616 INFO     [asyncssh] [conn=0] Beginning auth for user root
2020-06-20 20:22:42,625 INFO     [asyncssh] [conn=0] Auth for user root succeeded
2020-06-20 20:22:42,626 DEBUG    [asyncssh] [conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536
2020-06-20 20:22:42,627 INFO     [asyncssh] [conn=0, chan=0] Requesting new SSH session
2020-06-20 20:22:42,631 INFO     [asyncssh] [conn=0, chan=0]   Command: ls /
2020-06-20 20:22:42,667 INFO     [asyncssh] [conn=0, chan=0] Received exit status 0
2020-06-20 20:22:42,668 INFO     [asyncssh] [conn=0, chan=0] Received channel close
2020-06-20 20:22:42,670 INFO     [asyncssh] [conn=0, chan=0] Channel closed
2020-06-20 20:22:42,671 INFO     [root] WEBSERVER --- bin --- boot --- data --- dev --- etc --- firmware --- home --- legato --- lib --- linuxrc --- media --- mnt --- opt --- proc --- run --- sbin --- sys --- tmp --- usr --- var
2020-06-20 20:22:42,671 INFO     [asyncssh] [conn=0] Closing connection
2020-06-20 20:22:42,672 INFO     [asyncssh] [conn=0] Sending disconnect: Disconnected by application (11)
2020-06-20 20:22:42,674 INFO     [asyncssh] [conn=0] Connection closed

With 2.2.1 it fails:

2020-06-20 20:24:07,118 DEBUG    [asyncio] Using selector: EpollSelector
2020-06-20 20:24:07,121 INFO     [asyncssh] Opening SSH connection to 192.168.2.2, port 22
2020-06-20 20:24:07,126 INFO     [asyncssh] [conn=0] Connection to 192.168.2.2, port 22 succeeded
2020-06-20 20:24:07,127 INFO     [asyncssh] [conn=0]   Local address: 192.168.2.3, port 43402
2020-06-20 20:24:07,141 DEBUG    [asyncssh] [conn=0] Requesting key exchange
2020-06-20 20:24:07,143 DEBUG    [asyncssh] [conn=0] Received key exchange request
2020-06-20 20:24:07,145 DEBUG    [asyncssh] [conn=0] Beginning key exchange
2020-06-20 20:24:07,146 INFO     [asyncssh] [conn=0] Connection lost
2020-06-20 20:24:07,147 INFO     [asyncssh] [conn=0] Aborting connection
Traceback (most recent call last):
  File "test_asyncssh.py", line 17, in <module>
    asyncio.run(test_coro())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "test_asyncssh.py", line 11, in test_coro
    async with asyncssh.connect("192.168.2.2", port=22, username="root", known_hosts=None) as conn:
  File "/usr/lib/python3/dist-packages/asyncssh/misc.py", line 164, in __aenter__
    self._result = await self._coro
  File "/usr/lib/python3/dist-packages/asyncssh/connection.py", line 5696, in connect
    conn_factory, 'Opening SSH connection to')
  File "/usr/lib/python3/dist-packages/asyncssh/connection.py", line 175, in _connect
    await conn.wait_established()
  File "/usr/lib/python3/dist-packages/asyncssh/connection.py", line 1927, in wait_established
    await self._waiter
asyncssh.misc.ConnectionLost: Connection lost

The target device is a LTE modem running dropbear, so it might be a little "non standard" but still it worked before ;-)
I initially though this could be a cypher issue, because I had to also upgrade "cryptography" module but no, old asyncssh with new cryptography work as expected, so I really think this is a regression...

Thanks in advance,

Adam.

PS: Don't hesitate to ask me for more logs or tests, I'll provide asap.

[ronf]

You may be running into #276, caused by Dropbear failing if it doesn't find one of its supported host key algorithms in the first 20 entries in the client's list. AsyncSSH supports more than 20 public key algorithms now, and the Dropbear-compatible args are way down at the end of the list, as they are the weakest.

Try adding server_host_key_algs=['ssh-rsa'] in your connect() call and see if that makes a difference. Normally, this won't be seen if you are doing known_hosts checking, as AsyncSSH will default to only sending algorithms it has a known hosts entry for. However, when you disable that check, it sends its full list, and that can trigger this issue.

[eLvErDe]

Right, adding this param make the connection work again.

Pretty hard to debug IMHO, would it be possible to detect dropbear while greeting and apply workaround ?

Thanks

[ronf]

In the next release of AsyncSSH, I'll be disabling DSA keys by default, and I think that will be enough to keep the RSA algorithm in the default list within Dropbear's limit, at least for now.

As for applying the workaround automatically, I do have some workarounds for bugs I've run into in other clients/servers, but I've tried to write such code in a way that doesn't depend on the version string sent in the handshake. That'll be more difficult to do here, though, since the decision needs to be made before sending the kexinit, which is the first packet after the version strings are exchanged.

If I add more public key algorithms in the future that trigger this, I'll revisit this. Perhaps I could do something to move ssh-rsa up in the list of allowed public keys, but only for dropbear and when known hosts checking is disabled. At that point, we're really not validating the server's host key anyway, so the ordering of allowed public key algorithms probably doesn't matter much.

[eLvErDe]

Fair enough, thanks a lot for the full explanation !

[ronf]

This issue should no longer occur with AsyncSSH 2.3.0, unless you enable all algorithms and leave ssh-rsa toward the very end of the list (beyond where Dropbear will look).