Workaround SSH error "no matching host key type found. Their offer: ssh-rsa" #905

invidian · 2022-05-30T16:06:42Z

What steps did you take and what happened:

Right now, if one builds the CAPI images with OpenSSH version 8.8+ on the host system, the environment variable below is required to make provisioning work. This is because OpenSSH starting from version 8.8 has disabled ssh-rsa algorithm from default configuration and this is what Ansible plugin for Packer still generates.

ANSIBLE_SSH_ARGS="-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa"

What did you expect to happen:

Builds to work on modern systems out of the box.

Anything else you would like to add:

I suggest we commit those arguments as default ones until upstream issue is not resolved.

As an alternative, we could try disabling Ansible provisioner proxy, bypassing local keys generation completely. I don't know what side effects that creates though.

Environment:

Project (Image Builder for Cluster API, kube-deploy/imagebuilder, konfigadm): Image Builder for Cluster API

Additional info for Image Builder for Cluster API related issues:

OS (e.g. from /etc/os-release, or cmd /c ver): Arch Linux
Packer Version: 1.8.1
Packer Provider:
Ansible Version: 2.11.5

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

CC @kopiczko

The text was updated successfully, but these errors were encountered:

k8s-triage-robot · 2022-08-28T16:23:19Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

invidian · 2022-08-28T19:07:23Z

/remove-lifecycle stale

Workaround problems with Packer / Ansible SSH interaction, see kubernetes-sigs#905 for background and suggested fix

AverageMarcus · 2022-10-10T15:09:12Z

Looks like this problem also prevents being able to use Ubuntu 22.04 as a base VM. Doesn't look like the env var can be used as a workaround in that situation (at least it didn't work for me).

Although the error is somewhat different so I may be mistaken that it's the same problem:

ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Meecr0b · 2022-10-14T04:19:19Z

we successfully implemented a workaround in our environment by directly setting

"existing_ansible_ssh_args": "-o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa"

author Nick Jones <nick@dischord.org> 1657794093 +0100 committer Drew <vilesy@gmail.com> 1667210722 +0000 squash and rebase Fixes for SSH provisioning errors Workaround problems with Packer / Ansible SSH interaction, see kubernetes-sigs#905 for background and suggested fix patch: changed deprecated floating_ip_pool to floating_ip_network patch: adding non efi ubuntu fix: updating use_floating var and setting the option to use a floating IP to a var. fix: put quotes around the bool var, didn't realise that was a thing. chore: rmeoved some variable values that are not very global patch: synced master and added config drive option to allow internal/external IP configuration (direct access) on openstack patch: updating variables for openstack to allow for specific Kubernetes/crctl-tools versions to be defined patch: adding ubuntu 22.04 to make options patch: changed deprecated floating_ip_pool to floating_ip_network patch: adding non efi ubuntu patch: updating variables for openstack to allow for specific Kubernetes/crctl-tools versions to be defined

Workaround problems with Packer / Ansible SSH interaction, see kubernetes-sigs#905 for background and suggested fix

This is done to remove repetition of '-o IdentitiesOnly=yes' to make sure it is consistent across all platforms and to reduce amount of churn when adding new default arguments like we plan as part of mitigating issue with ssh-rsa keys (kubernetes-sigs#905).

Since OpenSSH version 8.8+ ssh-rsa key algorithm is disabled by default, which right now causes builds to fail for builders which use OpenSSH version 8.8+. The problematic keys are generated by Ansible plugin for Packer and the problem is currently being discussed in issue hashicorp/packer-plugin-ansible#69. An alternative would be to consider using `use_proxy=false` option in plugin, however we are not sure what could be the implications of this. Given that building machine should be a rather short process, the workaround seem acceptable and actually allows being able to succesfully build images out of the box on more distributions. In implementation, 'PubkeyAcceptedKeyTypes' is used instead of 'PubkeyAcceptedAlgorithms', as it provides better backward compatibility, since 'PubkeyAcceptedAlgorithms' is only available since OpenSSH version 8.4. See issue kubernetes-sigs#905 for more details. Co-authored-by: Jeremi Piotrowski <jeremi.piotrowski@gmail.com>

This is done to remove repetition of '-o IdentitiesOnly=yes' to make sure it is consistent across all platforms and to reduce amount of churn when adding new default arguments like we plan as part of mitigating issue with ssh-rsa keys (kubernetes-sigs#905).

Since OpenSSH version 8.8+ ssh-rsa key algorithm is disabled by default, which right now causes builds to fail for builders which use OpenSSH version 8.8+. The problematic keys are generated by Ansible plugin for Packer and the problem is currently being discussed in issue hashicorp/packer-plugin-ansible#69. An alternative would be to consider using `use_proxy=false` option in plugin, however we are not sure what could be the implications of this. Given that building machine should be a rather short process, the workaround seem acceptable and actually allows being able to succesfully build images out of the box on more distributions. In implementation, 'PubkeyAcceptedKeyTypes' is used instead of 'PubkeyAcceptedAlgorithms', as it provides better backward compatibility, since 'PubkeyAcceptedAlgorithms' is only available since OpenSSH version 8.4. See issue kubernetes-sigs#905 for more details. Co-authored-by: Jeremi Piotrowski <jeremi.piotrowski@gmail.com>

Below commit messages from squashed commits: images/capi/packer: extract ansible common SSH args to a single place This is done to remove repetition of '-o IdentitiesOnly=yes' to make sure it is consistent across all platforms and to reduce amount of churn when adding new default arguments like we plan as part of mitigating issue with ssh-rsa keys (kubernetes-sigs#905). images/capi/packer: allow specifying extra scp arguments for Ansible This allows a workaround for issue kubernetes-sigs#859 when building host uses OpenSSH version 9.0+, which uses SFTP protocol for SCP instead of a legacy SCP protocol, which right now causes builds to fail with error message as below when Ansible is trying to copy files over to remote host. bash: line 1: /usr/lib/sftp-server: No such file or directory\nscp: Connection closed\r\n" This commit allows users with new OpenSSH version to specify ANSIBLE_SCP_EXTRA_ARGS="-O" to fix their builds. I plan to automate this in another commit, as it should be relatively simple and harmless. Refs kubernetes-sigs#859. images/capi/packer: allow using ssh-rsa keys with OpenSSH 8.8+ Since OpenSSH version 8.8+ ssh-rsa key algorithm is disabled by default, which right now causes builds to fail for builders which use OpenSSH version 8.8+. The problematic keys are generated by Ansible plugin for Packer and the problem is currently being discussed in issue hashicorp/packer-plugin-ansible#69. An alternative would be to consider using `use_proxy=false` option in plugin, however we are not sure what could be the implications of this. Given that building machine should be a rather short process, the workaround seem acceptable and actually allows being able to succesfully build images out of the box on more distributions. In implementation, 'PubkeyAcceptedKeyTypes' is used instead of 'PubkeyAcceptedAlgorithms', as it provides better backward compatibility, since 'PubkeyAcceptedAlgorithms' is only available since OpenSSH version 8.4. See issue kubernetes-sigs#905 for more details. Co-authored-by: Jeremi Piotrowski <jeremi.piotrowski@gmail.com> images/capi/Makefile: set ANSIBLE_SCP_EXTRA_ARGS="-O" when needed Since OpenSSH 9.0+ 'scp' uses SFTP protocol instead of legacy SCP protocol, which causes building errors like: bash: line 1: /usr/lib/sftp-server: No such file or directory\nscp: Connection closed\r\n"" However, -O option is not available in older OpenSSH version, so we cannot always set it as an option to use. To provide better out-of-the-box experience for users with newer versions of OpenSSH, we conditionally ensure -O is used when used OpenSSH version requires it. See kubernetes-sigs#859 and hashicorp/packer-plugin-ansible#100 for more details. Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com> Co-authored-by: Jeremi Piotrowski <jeremi.piotrowski@gmail.com>

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label May 30, 2022

kopiczko mentioned this issue Jun 1, 2022

Fix SSH/scp/SFTP issues for OpenSSH 9.0+ and Flatcar stable #907

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 28, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 28, 2022

yankcrime added a commit to eschercloudai/image-builder that referenced this issue Oct 5, 2022

Fixes for SSH provisioning errors

d685177

Workaround problems with Packer / Ansible SSH interaction, see kubernetes-sigs#905 for background and suggested fix

drew-viles pushed a commit to eschercloudai/image-builder that referenced this issue Oct 31, 2022

Fixes for SSH provisioning errors

43a4e99

Workaround problems with Packer / Ansible SSH interaction, see kubernetes-sigs#905 for background and suggested fix

Meecr0b mentioned this issue Dec 2, 2022

Fix ansible SSH not connecting #1014

Closed

invidian mentioned this issue Dec 14, 2022

images/capi: improve out-of-the-box compatibility with OpenSSH versions 8.8+ and 9.0+ #1035

Merged

k8s-ci-robot closed this as completed in #1035 Dec 22, 2022

invidian mentioned this issue Jul 12, 2023

images/capi/packer/config/ansible-args.json: remove ssh-rsa workaround #1224

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workaround SSH error "no matching host key type found. Their offer: ssh-rsa" #905

Workaround SSH error "no matching host key type found. Their offer: ssh-rsa" #905

invidian commented May 30, 2022 •

edited

Loading

k8s-triage-robot commented Aug 28, 2022

invidian commented Aug 28, 2022

AverageMarcus commented Oct 10, 2022 •

edited

Loading

Meecr0b commented Oct 14, 2022

Workaround SSH error "no matching host key type found. Their offer: ssh-rsa" #905

Workaround SSH error "no matching host key type found. Their offer: ssh-rsa" #905

Comments

invidian commented May 30, 2022 • edited Loading

k8s-triage-robot commented Aug 28, 2022

invidian commented Aug 28, 2022

AverageMarcus commented Oct 10, 2022 • edited Loading

Meecr0b commented Oct 14, 2022

invidian commented May 30, 2022 •

edited

Loading

AverageMarcus commented Oct 10, 2022 •

edited

Loading