Proxy fails with error "Failed to connect to the host via scp: /bin/sh: 1: /usr/lib/sftp-server: not found\nscp: Connection closed\r\n" OpenSSH 9.0+

[invidian]

Overview of the Issue

If one has OpenSSH version 9.0+ installed on their system, provisioning using this provisioner will fail with the following error:

learn-packer.docker.ubuntu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via scp: /bin/sh: 1: /usr/lib/sftp-server: not found\nscp: Connection closed\r\n", "unreachable": true}

The error occurs even though I have /usr/lib/sftp-server file on my machine:

$ ls -la /usr/lib/sftp-server
lrwxrwxrwx 1 root root 24 Apr 13 19:29 /usr/lib/sftp-server -> /usr/lib/ssh/sftp-server

This is because OpenSSH 9.0 has switched the default mode for scp from legacy mode to sftp mode and it seems SSH proxy set up by the plugin does not handle this correctly.

Reproduction Steps

Steps to reproduce this issue:

1. Prerequisites:

• OpenSSH version 9.0+
• Docker
• Packer

2. Run the commands below:

cat <<EOF > docker-ubuntu.pkr.hcl
packer {
  required_plugins {
    docker = {
      version = ">= 0.0.7"
      source  = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "ubuntu" {
  image  = "ubuntu:xenial"
  commit = true
}

build {
  name = "learn-packer"
  sources = [
    "source.docker.ubuntu"
  ]

  provisioner "ansible" {
    playbook_file = "./playbook.yml"
  }
}
EOF
cat <<EOF > playbook.yml
---
# playbook.yml
- name: 'Provision Image'
  hosts: default
  become: true

  tasks:
    - name: install Apache
      package:
        name: 'httpd'
        state: present
EOF
packer init .
ANSIBLE_SSH_ARGS="-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa" packer build docker-ubuntu.pkr.hcl

Plugin and Packer version

From packer version: Packer v1.8.1

Operating system and Environment details

OS: Arch Linux
OpenSSH version:

$ pacman -Qi openssh
Name            : openssh
Version         : 9.0p1-1

Log Fragments and crash.log files

https://gist.github.com/invidian/a9dbab69d1541b83f728f6009e27f593

CC @kopiczko

[dometto]

Still an issue on packer v1.9.4. Any fixes or workarounds?

[dometto]

Found a workaround in a related thread: hashicorp/packer#11783 (comment)

#110 may be a duplicate of this.

[bo-iliev]

Found a workaround in a related thread: hashicorp/packer#11783 (comment)

#110 may be a duplicate of this.

I also faced this issue and this workaround did the job for me. Thanks for sharing it and it would be lovely if this could get fixed.