SSH key format deprecated in Ubuntu 22.04

[cpmills1975]

Expected Behaviour

K3S is installed?

Current Behaviour

Brand new install of Ubuntu 22.04 LTS on a Raspberry Pi 4.
I have an SSH key on my MacBook
I've ssh-copy-id'd that over to ubuntu@pi
I've confirmed I can ssh on to the pi using ssh ubuntu@pi and it logs me straight in.
Attempting to run k3sup install --ip 192.168.10.1 --user ubuntu with or without --ssh-key ~/.ssh/id_rsa always results in

2022/04/29 01:03:02 192.168.10.1
Public IP: 192.168.10.1
Error: unable to connect to 192.168.10.1:22 over ssh: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

On the ubuntu side, I'm seeing the following logged in /var/log/auth.log

Apr 29 00:12:18 pi1 sshd[2515]: Connection closed by authenticating user ubuntu <redacted ip> port 61933 [preauth]
Apr 29 00:12:18 pi1 sshd[2517]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Apr 29 00:12:18 pi1 sshd[2517]: Connection closed by authenticating user ubuntu <redacted ip> port 61934 [preauth]

Are you a GitHub Sponsor (Yes/No?)

• Yes
• No

Possible Solution

Steps to Reproduce

1. Install brand new Ubuntu 22.04 LTS
2. Download k3sup on MacBook using curl -sLS https://get.k3sup.dev | sh
3. Attempt to install K3S on Pi using

export SERVER_IP=192.168.1.43
export USER=ubuntu

k3sup install --ip $SERVER_IP --user $USER

Context

I'm simply trying to install K3S

Your Environment

• What Kubernetes distribution are you using?
  Can't get that far.

• What OS or type or VM are you using for your cluster? Where is it hosted? (for k3sup install/join):
  Ubuntu 22.04 LTS on a Raspberry Pi

• Operating System and version (e.g. Linux, Windows, MacOS):
  MacOS

"Be part of the solution"

Subject to approval, are you willing to work on a Pull Request for this issue or feature request?

• Yes
• No

Depends on language though - I have some development experience in a number of languages.

[dwojcik92]

Had the same problem with Ubuntu 22.04LTS. Switched to 20.04 and this solved the problem.

[phcco]

I had the same issue. Seems Ubuntu 22.04 LTS openssh version deprecated RSA SHA-1 hash algorithm.

Create a new key using ssh-keygen or add to /etc/ssh/sshd_config on the server-side:

PubkeyAcceptedKeyTypes=+ssh-rsa

https://www.openssh.com/txt/release-8.2

[alexellis]

@phcco thanks for the workaround. What format of SSH key gets generated if you run ssh-keygen on an 22.04 machine?

[alexellis]

/set title: SSH key format deprecated in Ubuntu 22.04

[alexellis]

What format of SSH key gets generated if you run ssh-keygen on an 22.04 machine?

@rgee0 could you take a look into this? Is there an option for ssh-keygen which would "just work" without changing the server's config?

[rgee0]

This seems to be a recognised wider issue. Dropping a link here for reference:

hashicorp/packer#11733 (comment)

I replicated the original issue report and then used the following steps to work around it without hacking the server config:

Generate a new non-rsa key-pair (Link to some detail on ed25519)

# ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
Generating public/private ed25519 key pair.
...

Transfer this to the remote machine:

# ssh-copy-id -i ./id_ed25519.pub ${USER}@${IP}

Run k3sup with --ssh-key:

k3sup install --ip ${IP} --user ${USER} --ssh-key ~/.ssh/id_ed25519
Running: k3sup install
2022/05/04 19:36:31 <IP_ADDRESS>
Public IP: <IP_ADDRESS>
[INFO]  Finding release for channel stable
[INFO]  Using v1.23.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.23.6+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.23.6+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Result: [INFO]  Finding release for channel stable
[INFO]  Using v1.23.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.23.6+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.23.6+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
[INFO]  systemd: Starting k3s
 Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

Saving file to: /Users/rgee0/go/src/github.com/alexellis/arkade/kubeconfig

# Test your cluster with:
export KUBECONFIG=/Users/rgee0/go/src/github.com/alexellis/arkade/kubeconfig
kubectl config set-context default
kubectl get node -o wide

I'll do a sweep of the doco and update where/if necessary

[cpmills1975]

I'm struggling to understand how this problem comes to light when running k3sup, but not simple ssh from the command line. How is k3sup initiating the ssh connection differently to how the Mac ssh command line initiates it?

[rgee0]

This is suggested as fixing the issue, and updates only the imported packages, which suggests there is something in the k3sup binary that is a factor. You'd have to dig into the crypto & sys packages to understand the actual issue.

[phcco]

@alexellis Ubuntu 22.04 generates RSA by default, users must use explicitly ssh-keygen -t ed25519. I think no distro ships with a different default key type.

Reading the release again seems they deprecated RSA SHA-1, but RSA with SHA-256/512 signature should still work. I presume the Go crypto library had/has a bug on this signature negotiation/verification, this would explain why the ssh still works but not the binary. I couldn't test the suggested PR with the updated packages.

[smerlos]

I had the same issue. Seems Ubuntu 22.04 LTS openssh version deprecated RSA SHA-1 hash algorithm.

Create a new key using ssh-keygen or add to /etc/ssh/sshd_config on the server-side:

PubkeyAcceptedKeyTypes=+ssh-rsa

https://www.openssh.com/txt/release-8.2
this solution works for me. adding some tips remember to restart ssh service

[alexellis]

I ran into this yesterday whilst testing on Linode, the suggestion by @phcco was the solution that I went with:

#377 (comment)

In the meantime, let's monitor the ssh library being used to see if this gets fixed or has a workaround that doesn't require logging into each server to change it, or any changes in the userdata at VM creation time?

[alexellis]

Closed via: c59d67b

@untcha - I don't know how you're using k3sup, but please see this comment.

[alexellis]

Thanks to everyone who participated in this thread. I'm now going to lock it.

If you use K3sup, then we can speed up how quickly I can respond to issues here by becoming GitHub Sponsors.

I maintain this in my "free time" on a voluntary basis, for free, for your benefit both at work and at home.

https://github.com/sponsors/alexellis