SSH with commands during provisioning don't return

[jacklund]

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

I'm creating a ubuntu 21.10 raspberry pi image using packer-builder-arm and ansible 2.9.6. My ansible playbook is very simple (so far), but it hangs when it tries to run echo ~root && sleep 0, which, AFAICT, ansible runs to test the connection. When I try to connect via ssh to the packer ssh communicator and run a command, the command output returns but the connection never does, which is what I think ansible is waiting on.

Reproduction Steps

% sudo PACKER_DEBUG=1 packer build packer/ubuntu_server_21.10_arm64.json.pkr.hcl

Plugin and Packer version

v1.7.8

Simplified Packer Buildfile

source "arm" "raspberry_pi_k8s" {
  file_checksum_type    = "sha256"
  file_checksum_url     = "http://cdimage.ubuntu.com/releases/21.10/release/SHA256SUMS"
  file_target_extension = "xz"
  file_unarchive_cmd    = ["xz", "--decompress", "$ARCHIVE_PATH"]
  file_urls             = ["http://cdimage.ubuntu.com/releases/21.10/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz"]
  image_build_method    = "reuse"
  image_chroot_env      = ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"]
  image_partitions {
    filesystem   = "fat"
    mountpoint   = "/boot/firmware"
    name         = "boot"
    size         = "256M"
    start_sector = "2048"
    type         = "c"
  }
  image_partitions {
    filesystem   = "ext4"
    mountpoint   = "/"
    name         = "root"
    size         = "2.8G"
    start_sector = "526336"
    type         = "83"
  }
  image_path                   = "ubuntu-21.10.img"
  image_size                   = "3.1G"
  image_type                   = "dos"
  qemu_binary_destination_path = "/usr/bin/qemu-aarch64-static"
  qemu_binary_source_path      = "/usr/bin/qemu-aarch64-static"
}

build {
  sources = ["source.arm.raspberry_pi_k8s"]

  provisioner "ansible" {
    playbook_file = "ansible/playbook.yml"
    extra_arguments = [ "-vvvv" ]
  }
}

Ansible Playbook

---
- name: 'Provision image'
  hosts: default
  become: true
  gather_facts: no
  vars_files:
    - vars.yml

  tasks:
    - name: Debug
      debug:
        msg: "Testing 1 2 3"

    - name: Create user group
      group:
        name: "{{ username }}"
        state: present

Operating system and Environment details

Ubuntu 20.04 on x86_64

Log Fragments and crash.log files

Packer log

Output of manually running ssh command

% sudo ssh -v 127.0.0.1 -o Port=34669 -o 'IdentityFile="/tmp/ansible-key544648103"' 'pwd' 
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 34669.
debug1: Connection established.
debug1: identity file /tmp/ansible-key544648103 type -1
debug1: identity file /tmp/ansible-key544648103-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug1: Authenticating to 127.0.0.1:34669 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:9ZU07UhwYA8vXs1g6QILOxwnzOYM4peD+AX1ZoB1tks
debug1: Host '[127.0.0.1]:34669' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:16
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /tmp/ansible-key544648103  explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /tmp/ansible-key544648103
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:34669).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: pwd
/
<Command hangs here>
^Cdebug1: channel 0: free: client-session, nchannels 1
Killed by signal 2.

[jacklund]

This turns out to be an issue with packer-builder-arm: mkaczanowski/packer-builder-arm#121.