scp: fail on newer OpenSSH builds - protocol error: filename does not match request

[jborean93]

SUMMARY

A newer release of OpenSSH has included the commit openssh/openssh-portable@3d896c1 which adds validation around the file name returned by the server. This causes issues when scp is used to fetch a file as Ansible will use single quotes around the path causing a validation failure.

We've currently started to see this issue in #47732 and have added a workaround by setting ansible_scp_extra_args: -T to get the test working again but this should be removed once this has been fixed.

This does not necessarily have to be handled in Ansible, this issue is just to keep track of the issue until further details can be gotten from OpenSSH. If left unfixed we may start seeing these issues the next time we build a new test container, currently the default container is affected.

Potentially related to https://bugzilla.mindrot.org/show_bug.cgi?id=2966.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ssh

ANSIBLE VERSION

devel

CONFIGURATION

N/A

OS / ENVIRONMENT

Linux with the latest OpenSSH build. This is something that affects the scp client and not the scp version on the target.

STEPS TO REPRODUCE

You can easily test this out by running a container that installs the latest nightly of OpenSSH.

# Create file with a space in it
touch "/tmp/file space.txt"

# Start an interactive container
docker run -it ubuntu:16.04 /bin/bash

# Install Ansible and compile the latest OpenSSH version
apt-get update -y
apt-get install software-properties-common
apt-add-repository --yes --update ppa:ansible/ansible
apt-get install -y ansible build-essential zlib1g-dev libssl-dev wget sshpass
mkdir /var/lib/sshd
chmod -R 700 /var/lib/sshd/
chown -R root:sys /var/lib/sshd/
useradd -r -U -d /var/lib/sshd/ -c "sshd privsep" -s /bin/false sshd
wget -c http://www.mindrot.org/openssh_snap/openssh-SNAP-20190220.tar.gz
tar -xzf openssh-SNAP-20190220.tar.gz
cd openssh
./configure --with-privsep-path=/var/lib/sshd/ --sysconfdir=/etc/ssh 
make
make install

echo "host  ansible_host=host.docker.internal ansible_user=<username>" > inventory.ini
ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_SCP_IF_SSH=True ansible -i inventory.ini host -k -m fetch -a "src='/tmp/file space.txt' dest=/tmp/file.txt flat=yes"

EXPECTED RESULTS

The file is fetched without any issues

ACTUAL RESULTS

root@d8d1f97503dc:/openssh# ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_SCP_IF_SSH=True ansible -i inventory.ini host -k -m fetch -a "src='/tmp/file space.txt' dest=/tmp/file.txt flat=yes"
SSH password: 
host | FAILED! => {
    "msg": "failed to transfer file to /tmp/file space.txt /tmp/file.txt:\n\n/etc/ssh/ssh_config line 55: Unsupported option \"gssapiauthentication\"\r\n/etc/ssh/ssh_config line 56: Unsupported option \"gssapidelegatecredentials\"\r\nprotocol error: filename does not match request\n"
}

You can manually run the scp command outside of Ansible to verify that this is what scp is returning. More specifically it's the fact that the path of the remote file is enclosed in single quotes like '[host.docker.internal]:'"'"'/tmp/file space.txt'"'"''

[ansibot]

Files identified in the description:

• lib/ansible/plugins/connection/ssh.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[abadger]

An updated openssh had been shipped on ubuntu 16 & 18 now. Marking this as p2. It's a bit tricky as this isn't a regression so i don't think it blocks the 2.8 release but it is very important that we fix this

[abadger]

@jborean93 @mattclay I think I found a way that we can quote compatibly:

scp '127.0.0.2:/var/tmp/file\ with\ space.txt

In other words, instead of passing interior quotes to scp, pass backslash escaped spaces to scp.

[abadger]

@jborean93 @mattclay, yeah, the fix is to change connection/ssh.py to quote spaces in the remote path with backslashes instead of calling shell.quote() on them.

[jborean93]

Using \ to escape spaces is not enough to continue working the way it does today. It would solve paths with spaces in them but we would now need to escape other chars like $ as using ' before meant those weren't evaluated.

[jborean93]

Added a further comment on https://bugzilla.mindrot.org/show_bug.cgi?id=2966, will wait until we get a reply.

While waiting for a reply I thought it best to recap this issue. Currently this should only affect you if;

• You have a patched version of OpenSSH on the Ansible controller. We know the following distributions are affected;
  • Ubuntu 16.04
  • Ubuntu 18.04
  • OpenSUSE Leap 15.0
  • Probably others, especially as time goes on
• Are explicitly using scp as the file transfer mechanism
  • By default smart is used which tries sftp then uses scp and piped if that failed.
  • Can be controlled with the scp_if_ssh option being set to scp
  • smart or sftp should work in the majority of situations but if sftp isn't available and this fails, it might be best to set the mechanism to piped, e.g. ansible_scp_if_ssh: piped
• Only affects fetching a file, copying a file should be unaffected
• The remote file has a space or non-ascii char in the path

If you are affected by this issue you have a few options to workaround this;

• Do not use scp as the transfer mechanism. Either set ansible_scp_if_ssh to sftp or piped
• If scp is required, then add the -T argument to tell scp to not validate the remote filename.
  • This can be done by setting ansible_scp_extra_args: -T, or
  • Setting the env value ANSIBLE_SCP_EXTRA_ARGS=-T when invoking Ansible, or
  • Adding the config entry

[ssh_connection]
scp_extra_args = -T

If you do come across any other scenario that we missed above, please let us know.

[abadger]

@jborean93 Just making sure that you are going to have a PR to "fix" this? It looks like upstream isn't going to fix this interface that they broke and we've decided that we're going to just warn people when this problem occurs?

rc1 is next week so all P2s should be fixed by next week.