Connection Lost only with asyncssh 2.2.0

[newAM]

I have an ancient KVM that I connect to over SSH.

The following code always works in aynsssh 2.1.0, and always fails in 2.2.0

async with asyncssh.connect(
    host="10.0.0.3",
    username="sysadmin",
    password="PASS",
    port=22,
    known_hosts=None,
    kex_algs=["diffe-hellman-group1-sha1"],
    encryption_algs=["aes128-cbc", "aes128-ctr"],
    mac_algs=["hmac-sha1"],
    compression_algs=None,
) as connection:
    pass

I checked the changelog, but couldn't find any reason for this to occur, any ideas?

[newAM]

I did a wireshark capture, in asyncssh 2.2.0 the key exchange multi-precision integer length is 129 vs 128 on 2.1.0 and putty. This was the only significant difference I could find.

[ronf]

It looks like there is a typo above in the key_algs argument ("diffe" vs. "diffie"), but I'm guessing that's a transcription here in your post here and not the actual code you tried, as that would have caused things to fail completely.

As you said, there were really no differences in the Diffie Hellman KEX implementation between 2.1.0 and 2.2.0, so I'm not really sure what would cause that. Are you using the same version of Python and other third-party libraries in both cases?

In general, an MPInt would always want its high-order bit to be 0. So. if a value was larger than 2^127-1 (meaning the high-order bit of a 128-bit value was 1), the encoding of that into an MPInt would be an extra byte long (starting with a leading 0x00 for the integer data, and with the 4-byte length up front being 0x00000011 instead of 0x00000010. For instance:

>>> MPInt(2**127)
b'\x00\x00\x00\x11\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
>>> MPInt(2**127-1)
b'\x00\x00\x00\x10\x7f\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'

So, it wouldn't be unusual for the length of MPInts to vary depending on what random value was selected and what value came out of doing the exponentiation.

Are there any logs you can look at on the KVM? Alternately, do you know which side closes the connection and at what point in the handshake?

[newAM]

It looks like there is a typo above in the key_algs argument ("diffe" vs. "diffie"), but I'm guessing that's a transcription here in your post here and not the actual code you tried, as that would have caused things to fail completely.

I hand copied this from the PC attached to the aforementioned KVM (it did not have a network connection), that is indeed a typo, good catch!

As you said, there were really no differences in the Diffie Hellman KEX implementation between 2.1.0 and 2.2.0, so I'm not really sure what would cause that. Are you using the same version of Python and other third-party libraries in both cases?

Yup! Here's the exact commands I use to switch between them.

sudo -H python3.8 -m pip install --upgrade asyncssh==2.1.0
Collecting asyncssh==2.1.0
  Using cached asyncssh-2.1.0-py3-none-any.whl (290 kB)
Requirement already satisfied, skipping upgrade: cryptography>=2.8 in /usr/local/lib/python3.8/dist-packages (from asyncssh==2.1.0) (2.8)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.8/dist-packages (from cryptography>=2.8->asyncssh==2.1.0) (1.13.2)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /usr/local/lib/python3.8/dist-packages (from cryptography>=2.8->asyncssh==2.1.0) (1.14.0)
Requirement already satisfied, skipping upgrade: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.8->asyncssh==2.1.0) (2.19)
Installing collected packages: asyncssh
  Attempting uninstall: asyncssh
    Found existing installation: asyncssh 2.2.0
    Uninstalling asyncssh-2.2.0:
      Successfully uninstalled asyncssh-2.2.0
Successfully installed asyncssh-2.1.0

sudo -H python3.8 -m pip install --upgrade asyncssh==2.2.0
Collecting asyncssh==2.2.0
  Using cached asyncssh-2.2.0-py3-none-any.whl (263 kB)
Requirement already satisfied, skipping upgrade: cryptography>=2.8 in /usr/local/lib/python3.8/dist-packages (from asyncssh==2.2.0) (2.8)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.8/dist-packages (from cryptography>=2.8->asyncssh==2.2.0) (1.13.2)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /usr/local/lib/python3.8/dist-packages (from cryptography>=2.8->asyncssh==2.2.0) (1.14.0)
Requirement already satisfied, skipping upgrade: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.8->asyncssh==2.2.0) (2.19)
Installing collected packages: asyncssh
  Attempting uninstall: asyncssh
    Found existing installation: asyncssh 2.1.0
    Uninstalling asyncssh-2.1.0:
      Successfully uninstalled asyncssh-2.1.0
Successfully installed asyncssh-2.2.0

Are there any logs you can look at on the KVM? Alternately, do you know which side closes the connection and at what point in the handshake?

There are unfortunately no logs on the KVM.

The KVM closes the connection directly after the asyncssh client sends the "Diffie-Hellman Key Exchange Init" with the multi precision integer length of 129. I would not be surprised if this was non-compliance with the KVM SSH server, I'll do some more reading and investigation and see what I come up with!

[ronf]

I took a closer look at this today, and confirmed that forcing a KEX alg of "diffie-hellman-group1-sha1" produces public keys which alternate between 128 and 129 bytes, depending on whether the high-order bit is set or not for these public keys. This is true for both AsyncSSH 2.1.0 and AsyncSSH 2.2.0 (and 2.2.1). So, I don't think that difference would explain things working reliably on AsyncSSH 2.1.0 but not on later versions.

Are there any other differences you've been able to spot between the two cases? Do you know which SSH implementation is being used on the KVM? What does it report in its SSH version string? I don't know if it'll tell us much, but getting detailed (level 3) log output from AsyncSSH might be worth looking over. Just be careful to strip out any password or other sensitive information -- it should be fine in this case to just send the log up to the point where the key exchange ends (whether it succeeds or fails), before any authentication happens.

[newAM]

I went back to get the capture but it no longer fails with asyncssh 2.2.0 or 2.2.1, the only thing that I changed in the last two weeks was a KVM firmware update :/

I think it is safe to say that this was a device error. Thank you for your help debugging, and sorry that this one turned out to be my own problem!

[ronf]

No problem - thanks for letting me know. I'm glad you were able to solve the issue!