balena ssh fails with error: "no mutual signature algorithm"

[pdcastro]

As originally reported in: (restricted access)

• https://www.flowdock.com/app/rulemotion/i-cli/threads/hr2ctusL9HiXSPDLN-TfdKD94me
• https://www.flowdock.com/app/rulemotion/resin-tech/threads/BGWoubu72Hv9NdKH3Ypy7JNVfdZ

debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_rsa RSA SHA256:NaezTQw6QChMfd+Kd05cp2tcW719hzYJAJTtj54ZuiE
debug1: send_pubkey_test: no mutual signature algorithm

This is related to the fact that openssl started refusing the use of the ssh-rsa signature algorithm, as described in the following release notes (search for "deprecation" and "Potentially-incompatible changes"):

• https://www.openssh.com/txt/release-8.7
• https://www.openssh.com/txt/release-8.8

When the user updates their host OS, the ssh tool may be updated as well, bringing to light the openssl changes above. Behind the scenes, the balena ssh command uses the ssh tool available on the host OS.

Known workarounds

A known workaround is to replace RSA keys with ECDSA or Ed25519 keys. When you generate new keys, they need to be updated both in your balenaCloud account (which stores the public component of keys) and your workstation's homedir, typically changing from ~/.ssh/id_rsa[.pub] to ~/.ssh/id_ecdsa[.pub].

Root cause

It has also been reported that changes to the ssh server in the balenaCloud backend, at ssh.balena-devices.com, could solve this problem by allowing more secure signature algorithms like rsa-sha2-256 and rsa-sha2-512, in addition to the old ssh-rsa algorithm:

when I try to connect to github: debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
for ssh.balena-devices.com: debug2: host key algorithms: ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp256,ssh-dss

A balena-proxy PR under test is linked below (restricted access).

[Ereski]

@pdcastro just a pedantic note. I would change "more secure ECDSA keys" into something like "ECDSA keys, which are deemed more secure". They are more resistant to traditional analysis and methods, but we can't say that elliptic-curve cryptography is for a fact more secure than RSA in an absolute sense.

[Ereski]

The conclusion after a lot of digging is that a server-side fix is not straightforward at the moment. We are blocked by go/x/crypto/ssh lack of support for SHA2 signatures with RSA keys. We would be unblocked if this PR was merged: golang/crypto#187.