Add support for host key algorithms other than ssh-rsa on local ssh proxy

[toadjaune]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Overview of the Issue

When using packer with the ansible provisionner, packer creates a local ssh server for ansible to connect to, and injects necessary configurations for ansible to connect to it. This design allows packer to use ansible regardless of the type of target and transport (for example, it makes possible to provision a local docker container with ansible, even though there isn't a ssh server in the container).

The only host key signature algorithm currently supported by this local server is ssh-rsa (as evidenced by the detailed logs below)

Openssh 8.8 has ssh-rsa signature algorithm disabled by default (release notes).

This makes a successful ssh handshake between a local ssh client and the packer local ssh server impossible, which in turn makes ansible fail.

Reproduction Steps

• Have an operating system with openssh client >=8.8
  • Or manually disable ssh-rsa with HostKeyAlgorithms=-ssh-rsa (cf workarounds below on where to put this config)
• Install packer
• Install ansible
• Create the following files in a directory :

debug.pkr.hcl

packer {
  required_plugins {
    docker = {
      version = ">= 0.0.7"
      source  = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "python" {
  image  = "python:3"
  commit = true
}

build {
  name = "learn-packer"
  sources = [
    "source.docker.python"
  ]

  provisioner "ansible" {
    playbook_file = "./playbook.yml"
    extra_arguments = ["-vvvv"] # Make ansible and ssh client very verbose
    ansible_env_vars = [
      "ANSIBLE_CONFIG=./ansible.cfg", # Just to make the ansible debug output more readable
      "ANSIBLE_PIPELINING=true",
    ]
  }
}

ansible.cfg

[defaults]
stdout_callback=debug
stderr_callback=debug

playbook.yml

- name: Dummy test playbook
  hosts: all
  tasks:
    - debug: "msg=plop"

• run packer build debug.pkr.hcl
• Get a Unable to negotiate with 127.0.0.1 port XXXXX: no matching host key type found. Their offer: ssh-rsa error

Here is a detailed (although still abridged) log, the interesting bits are the host key algorithm lines.

Detailed log

learn-packer.docker.python: output will be in this color.

==> learn-packer.docker.python: Creating a temporary directory for sharing data...
==> learn-packer.docker.python: Pulling Docker image: python:3
[...]
==> learn-packer.docker.python: Starting docker container...
[...]
==> learn-packer.docker.python: Using docker communicator to connect:
==> learn-packer.docker.python: Provisioning with Ansible...
learn-packer.docker.python: Setting up proxy adapter for Ansible....
==> learn-packer.docker.python: Executing Ansible: ansible-playbook -e packer_build_name="python" -e packer_builder_type=docker -vvvv -e ansible_ssh_private_key_file=/tmp/ansible-key1138914215 -i /tmp/packer-provisioner-ansible3443892612 /path/packer/playbook.yml
[...]
learn-packer.docker.python: PLAYBOOK: playbook.yml *********************************************************
[...]
learn-packer.docker.python: PLAY [Dummy test playbook] *****************************************************
learn-packer.docker.python:
learn-packer.docker.python: TASK [Gathering Facts] *********************************************************
learn-packer.docker.python: task path: /path/packer/playbook.yml:1
==> learn-packer.docker.python: failed to handshake
[...]
learn-packer.docker.python: debug3: ssh_connect_direct: entering
learn-packer.docker.python: debug1: Connecting to 127.0.0.1 [127.0.0.1] port 42711.
learn-packer.docker.python: debug3: set_sock_tos: set socket 3 IP_TOS 0x48
learn-packer.docker.python: debug2: fd 3 setting O_NONBLOCK
learn-packer.docker.python: debug1: fd 3 clearing O_NONBLOCK
learn-packer.docker.python: debug1: Connection established.
learn-packer.docker.python: debug3: timeout: 10000 ms remain after connect
learn-packer.docker.python: debug1: identity file /tmp/ansible-key1138914215 type -1
learn-packer.docker.python: debug1: identity file /tmp/ansible-key1138914215-cert type -1
learn-packer.docker.python: debug1: Local version string SSH-2.0-OpenSSH_8.8
learn-packer.docker.python: debug1: Remote protocol version 2.0, remote software version Go
learn-packer.docker.python: debug1: compat_banner: no match: Go
learn-packer.docker.python: debug2: fd 3 setting O_NONBLOCK
learn-packer.docker.python: debug1: Authenticating to 127.0.0.1:42711 as 'toadjaune'
learn-packer.docker.python: debug3: send packet: type 20
learn-packer.docker.python: debug1: SSH2_MSG_KEXINIT sent
learn-packer.docker.python: debug3: receive packet: type 20
learn-packer.docker.python: debug1: SSH2_MSG_KEXINIT received
learn-packer.docker.python: debug2: local client KEXINIT proposal
learn-packer.docker.python: debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,ext-info-c
learn-packer.docker.python: debug2: host key algorithms: sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519
learn-packer.docker.python: debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
learn-packer.docker.python: debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
learn-packer.docker.python: debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
learn-packer.docker.python: debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
learn-packer.docker.python: debug2: compression ctos: zlib@openssh.com,zlib,none
learn-packer.docker.python: debug2: compression stoc: zlib@openssh.com,zlib,none
learn-packer.docker.python: debug2: languages ctos:
learn-packer.docker.python: debug2: languages stoc:
learn-packer.docker.python: debug2: first_kex_follows 0
learn-packer.docker.python: debug2: reserved 0
learn-packer.docker.python: debug2: peer server KEXINIT proposal
learn-packer.docker.python: debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
learn-packer.docker.python: debug2: host key algorithms: ssh-rsa
learn-packer.docker.python: debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
learn-packer.docker.python: debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
learn-packer.docker.python: debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
learn-packer.docker.python: debug2: compression ctos: none
learn-packer.docker.python: debug2: compression stoc: none
learn-packer.docker.python: debug2: languages ctos:
learn-packer.docker.python: debug2: languages stoc:
learn-packer.docker.python: debug2: first_kex_follows 0
learn-packer.docker.python: debug2: reserved 0
learn-packer.docker.python: debug1: kex: algorithm: curve25519-sha256@libssh.org
learn-packer.docker.python: debug1: kex: host key algorithm: (no match)
learn-packer.docker.python: Unable to negotiate with 127.0.0.1 port 42711: no matching host key type found. Their offer: ssh-rsa
learn-packer.docker.python:
learn-packer.docker.python:
learn-packer.docker.python: PLAY RECAP *********************************************************************
learn-packer.docker.python: default : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
learn-packer.docker.python:
==> learn-packer.docker.python: Provisioning step had errors: Running the cleanup provisioner, if present...
==> learn-packer.docker.python: Killing the container: a79558c88072c48122a6e63257f52bdb9d28ce9815d198e627ed73a283d4a88d
Build 'learn-packer.docker.python' errored after 3 seconds 367 milliseconds: Error executing Ansible: Non-zero exit status: exit status 4

==> Wait completed after 3 seconds 367 milliseconds

==> Some builds didn't complete successfully and had errors:
--> learn-packer.docker.python: Error executing Ansible: Non-zero exit status: exit status 4

==> Builds finished but no artifacts were created.

### Packer / system versions

• Packer : Packer v1.8.5
• Ansible : 2.14.1
• OS : Fedora 37 Workstation, with Linux 6.0.15-300.fc37.x86_64
• SSH client : OpenSSH_8.8p1, OpenSSL 3.0.5

Workarounds

Re-enabling ssh-rsa with the HostKeyAlgorithms client configuration is usually enough to work around this problem.

On Fedora (and presumably RedHat), though, there are system-wide policies in /etc/crypto-policies/back-ends, enforcing this deprecations, and making the ssh client discard ssh-rsa when manually added by the user. We'll need the PubkeyAcceptedAlgorithms for that part.

Resuting configurations :

• In packer configuration
  • extra_arguments = ["-vvvv", "--ssh-extra-args", "-o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa"] in the example config file above
  • ansible_ssh_extra_args should do it too, although in my testing, it did not seem to work.
• In ssh client configuration
  • Add the following at the top of ~/.ssh/config :

  Host 127.0.0.1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedAlgorithms +ssh-rsa
  

Fixes suggestions

I see a few different ways to fix this on the packer side :

• Adding HostKeyAlgorithms and PubkeyAcceptedAlgorithms to the default ssh options sent by packer :
  While this option is fast to implement and deploy, if users have defined custom ssh options, this would not apply to them. It is therefore quite brittle
• Implementing support for newer rsa-based signature algorithms (rsa-sha2-512,rsa-sha2-256) :
  This would allow ssh clients to connect with their default configuration, and I assume does not require much development ?
• Implementing support for ed25519 host keys :
  This would also work with default client configurations, but I assume represents more work to implement, as we're talking about an entirely distinct key/signature type.

[github-actions]

This issue has been synced to JIRA for planning.

JIRA ID: HPR-981

[nywilken]

@toadjaune thanks for calling this out. This sounds like a needed enhancement here. I've added this issue to our internal board for prioritization with the team.

[toadjaune]

Great news, thanks for the update ! :)

[SylvainMartel]

Any update?

[lbajolet-hashicorp]

Hi @toadjaune,

Forgive me if I misunderstand the issue here, but I believe this is essentially similar to #69, isn't it?

If so, this has been fixed in the latest release of the plugin, so now when you are starting the adapter, the Ansible plugin will use ECDSA keys for the SSH communication, and not ssh-rsa by default.

Can you confirm if that's the case for you?

[toadjaune]

Hi @lbajolet-hashicorp,

I can confim that the issue is no longer present in version 1.1.0 of the plugin. and that the connection successfully establishes with a ecdsa-sha2-nistp384 handshake.

Thanks a lot for this !