Setting ssh_args disables pipelining mode incorrectly

[axelson]

Issue Type:

Bug Report

Ansible Version:

ansible 1.5.5

Environment:

Running Ansible on Mac OS X Mavericks and managing SmartOS (Solaris).

Summary:

SSH Pipelining [1] doesn't work if sudo_user is set. If I comment out the sudo_user and sudo then the pipelining works.

[1] http://docs.ansible.com/intro_configuration.html#pipelining

Steps To Reproduce:

Here's my simple.yml file:

---
- hosts: all
  sudo: yes
  sudo_user: jason
  tasks:
    - name: git
      git: repo=https://github.com/ansible/ansible.git dest=~/ansible update=yes force=no
      environment: env_var

ansible-playbook -e "pipelining=True" simple.yml

With sudo_user and sudo and running with -vvv I can see that Ansible is running separate mkdir and chmod commands over ssh.

env_var is needed to set PATH for git

Expected Results:

I expect ansible to not send separate SSH commands for mkdir and chmod.

Actual Results:

SSH Pipelining is working so the ssh command for mkdir and chmod are combined.

[mpdehaan]

Can you please show the output from -vvv ?

Thanks!

[axelson]

Here you go (I've only changed the hostname):

PLAY [middle_tier] ************************************************************ 

GATHERING FACTS *************************************************************** 
<example.com> ESTABLISH CONNECTION FOR USER: jason
<example.com> REMOTE_MODULE setup
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'mkdir -p /tmp/ansible-tmp-1398987879.46-122273948422799 && chmod a+rx /tmp/ansible-tmp-1398987879.46-122273948422799 && echo /tmp/ansible-tmp-1398987879.46-122273948422799'"]
<example.com> PUT /var/folders/cg/6yn4m8mn65x8ts7swwp6jbf80000gn/T/tmpTWheW1 TO /tmp/ansible-tmp-1398987879.46-122273948422799/setup
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'chmod a+r /tmp/ansible-tmp-1398987879.46-122273948422799/setup'"]
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c '/opt/local/bin/python /tmp/ansible-tmp-1398987879.46-122273948422799/setup; rm -rf /tmp/ansible-tmp-1398987879.46-122273948422799/ >/dev/null 2>&1'"]
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'rm -rf /tmp/ansible-tmp-1398987879.46-122273948422799/ >/dev/null 2>&1'"]
ok: [example.com]

TASK: [git] ******************************************************************* 
<example.com> ESTABLISH CONNECTION FOR USER: jason
<example.com> REMOTE_MODULE git repo=https://github.com/ansible/ansible.git dest=~/ansible update=yes force=no
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'mkdir -p /tmp/ansible-tmp-1398987882.43-255311493710624 && chmod a+rx /tmp/ansible-tmp-1398987882.43-255311493710624 && echo /tmp/ansible-tmp-1398987882.43-255311493710624'"]
<example.com> PUT /var/folders/cg/6yn4m8mn65x8ts7swwp6jbf80000gn/T/tmp257YFW TO /tmp/ansible-tmp-1398987882.43-255311493710624/git
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'chmod a+r /tmp/ansible-tmp-1398987882.43-255311493710624/git'"]
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'PATH=/home/jason/bin:/usr/local/bin:/usr/local/jdk/bin/amd64:/opt/local/bin:/opt/local/sbin:/usr/ucb:/usr/bin:/bin:/usr/sbin:/usr/xpg4/bin:/usr/X11R6/bin:.:/sbin:/usr/sfw/bin:/usr/openwin/bin:/opt/SUNWspro/bin:/usr/ccs/bin /opt/local/bin/python /tmp/ansible-tmp-1398987882.43-255311493710624/git'"]
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'rm -rf /tmp/ansible-tmp-1398987882.43-255311493710624/ >/dev/null 2>&1'"]
ok: [example.com] => {"after": "c6f3a0a4ccc7dceda0946d3b4c38797ba18ce59d", "before": "c6f3a0a4ccc7dceda0946d3b4c38797ba18ce59d", "changed": false}

PLAY RECAP ******************************************************************** 
example.com         : ok=2    changed=0    unreachable=0    failed=0   

Note: this is obviously with sudo_user set to jason, which is also the username I'm ssh'ing as, setting the username to a different existing username on the remote system looks similar.

[jimi-c]

@axelson the above output looks like pipelining is disabled, correct? There should never be a PUT operation when pipelining is enabled, or are you saying that's the problem?

[axelson]

@jimi-c Yes, that is exactly the problem I'm running ansible-playbook -i staging -e "pipelining=True" simple.yml -vvv yet pipelining is not enabled. Commenting out sudo_user will re-enable pipelining but I need sudo_user for my playbook.

For completeness the contents of my ansible.cfg are:

[defaults]
sudo_exe=/opt/local/bin/sudo
pipelining=True

And the contents of my inventory file:

[host]
example.com

[all:vars]
ansible_python_interpreter=/opt/local/bin/python
path_var=/home/jason/bin:/usr/local/bin:/usr/local/jdk/bin/amd64:/opt/local/bin:/opt/local/sbin:/usr/ucb:/usr/bin:/bin:/usr/sbin:/usr/xpg4/bin:/usr/X11R6/bin:.:/sbin:/usr/sfw/bin:/usr/openwin/bin:/opt/SUNWspro/bin:/usr/ccs/bin

and group_vars/all:

---
env_var:
  PATH: /home/jason/bin:/usr/local/bin:/usr/local/jdk/bin/amd64:/opt/local/bin:/opt/local/sbin:/usr/ucb:/usr/bin:/bin:/usr/sbin:/usr/xpg4/bin:/usr/X11R6/bin:.:/sbin:/usr/sfw/bin:/usr/openwin/bin:/opt/SUNWspro/bin:/usr/ccs/bin

[jimi-c]

What is your transport set to? If you're using EL6 (Centos, RHEL, etc.), you're probably using "smart" by default, which will use paramiko on those systems. The paramiko transport does not do pipelining, only the SSH one does.

[axelson]

@jimi-c The transport is set to smart. Also judging from the output of -vvv above it is using ssh with ControlPersist: <example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'mkdir -p /tmp/ansible-tmp-1398987879.46-122273948422799 && chmod a+rx /tmp/ansible-tmp-1398987879.46-122273948422799 && echo /tmp/ansible-tmp-1398987879.46-122273948422799'"]

Here's an example run with sudo_user commented out:

PLAY [middle_tier] ************************************************************

GATHERING FACTS ***************************************************************
<example.com> ESTABLISH CONNECTION FOR USER: jason
<example.com> REMOTE_MODULE setup
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1398992209.27-196441784872190 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1398992209.27-196441784872190 && echo $HOME/.ansible/tmp/ansible-tmp-1398992209.27-196441784872190'"]
<example.com> PUT /var/folders/cg/6yn4m8mn65x8ts7swwp6jbf80000gn/T/tmpxRrnsj TO /home/jason/.ansible/tmp/ansible-tmp-1398992209.27-196441784872190/setup
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', '/bin/sh -c \'/opt/local/bin/sudo -k && /opt/local/bin/sudo -H -S -p "[sudo via ansible, key=yjohkedidkssenhrbqutzjlknazrwwop] password: " -u root /bin/sh -c \'"\'"\'echo SUDO-SUCCESS-yjohkedidkssenhrbqutzjlknazrwwop; /opt/local/bin/python /home/jason/.ansible/tmp/ansible-tmp-1398992209.27-196441784872190/setup; rm -rf /home/jason/.ansible/tmp/ansible-tmp-1398992209.27-196441784872190/ >/dev/null 2>&1\'"\'"\'\'']
ok: [example.com]

TASK: [git] *******************************************************************
<example.com> ESTABLISH CONNECTION FOR USER: jason
<example.com> REMOTE_MODULE git repo=https://github.com/ansible/ansible.git dest=~/ansible update=yes force=no
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1398992213.33-162919704073112 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1398992213.33-162919704073112 && echo $HOME/.ansible/tmp/ansible-tmp-1398992213.33-162919704073112'"]
<example.com> PUT /var/folders/cg/6yn4m8mn65x8ts7swwp6jbf80000gn/T/tmpIunLjT TO /home/jason/.ansible/tmp/ansible-tmp-1398992213.33-162919704073112/git
<example.com> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/jason/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', 'example.com', '/bin/sh -c \'/opt/local/bin/sudo -k && /opt/local/bin/sudo -H -S -p "[sudo via ansible, key=iylkwcjwxnuarllcjgvibrdzcceubrts] password: " -u root /bin/sh -c \'"\'"\'echo SUDO-SUCCESS-iylkwcjwxnuarllcjgvibrdzcceubrts; PATH=/home/jason/bin:/usr/local/bin:/usr/local/jdk/bin/amd64:/opt/local/bin:/opt/local/sbin:/usr/ucb:/usr/bin:/bin:/usr/sbin:/usr/xpg4/bin:/usr/X11R6/bin:.:/sbin:/usr/sfw/bin:/usr/openwin/bin:/opt/SUNWspro/bin:/usr/ccs/bin /opt/local/bin/python /home/jason/.ansible/tmp/ansible-tmp-1398992213.33-162919704073112/git; rm -rf /home/jason/.ansible/tmp/ansible-tmp-1398992213.33-162919704073112/ >/dev/null 2>&1\'"\'"\'\'']
ok: [example.com] => {"after": "c6f3a0a4ccc7dceda0946d3b4c38797ba18ce59d", "before": "c6f3a0a4ccc7dceda0946d3b4c38797ba18ce59d", "changed": false}

PLAY RECAP ********************************************************************
example.com         : ok=2    changed=0    unreachable=0    failed=0

You can see in that run that SSH pipelining is working (from my understanding since there are no separate mkdir and chmod command steps).

[jimi-c]

I've tried this with both 1.5.5 and the devel branch and can't replicate it to an Ubuntu system. Unfortunately I don't have a Solaris-based system to test against, but it would seem to be something specific to your environment. Using "-e pipelining=True" doesn't do anything, but you have it in your ansible.cfg so that's not the issue.

Also, when you set sudo_user to the same user connecting to the system, sudo is actually silently disabled (since there's no need for it), yet in the example output you show above it is still using sudo - so something odd is going on here.

[axelson]

If I were to get you access to a solaris box to test on, would that help? I might be able to do that.

[jimi-c]

Yes, that might help.

[mpdehaan]

Probably wouldn't, OS X is the control host...

[jefferai]

I'm not sure if this is the same problem, but I have found (via -vvv) that having any value set for ssh_args under [ssh_connection] will cause pipelining to go away. This is not documented behavior.

(I have also, oddly, found that putting "-l myuser" as ssh_args will cause Ansible to be unable to log in, even though it sets -o User=myuser later on in the command string...running the same command manually is successful.)

[dpkirchner]

This bug is still present in 1.7.2. Removing the ssh_args directive allows pipelining to function. Also, specifying both ssh_args and pipelining does not result in an exception or warning. The above PR ensures the user knows about the problem.

[jimi-c]

@dpkirchner thanks, I've updated the title to more accurately reflect the issue here, and promoted it to a P2.

[jimi-c]

@dpkirchner / @jefferai - I'm unable to replicate the behavior with ssh_args. Could one of you please share what value you're setting that to, so that I can see if it causes the same breakage on my system?

Also, you should not specify the -l option, as it could break the logic of the ssh connection code Ansible uses.

[dpkirchner]

For me, ssh_args was just -o ForwardAgent=yes. The code sees that an argument is set and disables pipelining:

https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/connection_plugins/ssh.py#L62

Here's my playbook: https://gist.github.com/dpkirchner/c5092daa208df081fc25
and the vvvv output: https://gist.github.com/dpkirchner/383bffa182175acae247

I created these yesterday and forgot to upload the ansible.cfg, but it was:

[defaults]
transport = ssh
sudo_flags=-HE

[ssh_connection]
ssh_args=-o ForwardAgent=yes
pipelining=True

As for -l, I can't get around that. The repro above is trivial but the real use case requires that we log in as a specific user instead of. But the whole thing works fine with -l and without ssh_args (by moving the ForwardAgent to .ssh/config). It just makes the ansible process less self-contained.

IMO, the extra_args is not None test is unnecessary. Ideally, if someone adds bogus values they'll break their build and fix it.

[jimi-c]

Yeah odd, I can't reproduce the ssh_args issue still:

# ansible -m ping ubuntu1404 -vvv
<192.168.122.113> ESTABLISH CONNECTION FOR USER: root
<192.168.122.113> REMOTE_MODULE ping
<192.168.122.113> EXEC ssh -C -q -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 192.168.122.113 /bin/sh -c 'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'
192.168.122.113 | success >> {
    "changed": false, 
    "ping": "pong"
}

You can see that option being inserted, and pipelining still occurs.

[sivel]

I would like to attempt clarify something. ControlMaster/ControlPersist/ControlPath do not indicate "pipelining", while that is beneficial, I just want to point out that there is a difference. Pipelining is where there is a single 'EXEC ssh' line, instead of a EXEC ssh, PUT and a final EXEC ssh.

With that being said, can we confirm that we are talking about the same things at this point?

The original issue as reported in the first comment is that setting 'sudo_user' caused pipelining to fail, which could not be replicated either. It wasn't until #7251 (comment) (14 days after the issue was opened) that ssh_args was mentioned by someone else.

The basic logic that determines if pipelining can be used is:

if conn.has_pipelining and C.ANSIBLE_SSH_PIPELINING and not C.DEFAULT_KEEP_REMOTE_FILES and not self.su

[dpkirchner]

Ahh, so this does not enable a single pipeline for many separate modules
sent to a particular host?

I must have messed up my test w/o the ssh_args flag set.

[sivel]

Yes, so to explain a little further, ControlMaster and pipelining while related are not the same thing. And although not required together, will provide better performance if both used at the same time.

ControlMaster as described in the man page for ssh_config:

Enables the sharing of multiple sessions over a single network connection...These sessions will try to reuse the master instance's network connection rather than initiating new ones

When you set ssh_args, and do not explicitly declare ControlMaster/ControlPersist/ControlPath then they are omitted from the args passed to ssh. If you want the ControlMaster settings and override ssh_args, you will have to explicitly (re)declare them.

Pipelining as defined at http://docs.ansible.com/intro_configuration.html#pipelining is:

Enabling pipelining reduces the number of SSH operations required to execute a module on the remote server, by executing many ansible modules without actual file transfer (SFTP).

[jimi-c]

Ok, my mistake for mixing up the two issues. At this point, since the original issue is 5 months old, reported on a version 2 releases old, and was not reproducible, I'm going to go ahead and close this.

If you continue seeing any problems related to this issue, or if you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

• https://groups.google.com/forum/#!forum/ansible-project - for user questions, tips, and tricks
• https://groups.google.com/forum/#!forum/ansible-devel - for strategy, future planning, and questions about writing code

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular issue is resolved.

Thank you!

[jmcvetta]

Quick answer, if you want to use SSH agent forwarding without a big performance penalty, put this in your ansible.cfg file:

[ssh_connection]
pipelining = true
ssh_args = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=~/.ssh/ansible-%r@%h:%p

More reading: https://groups.google.com/forum/#!topic/ansible-project/Q44ykJkl3c0