Copy is stuck, and leaves lots of running processes when Ctrl+C'd

[Gui13]

ISSUE TYPE

• Bug Report

COMPONENT NAME

core

ANSIBLE VERSION

ansible 2.1.1.0
  config file = /home/guillaume/.ansible.cfg
  configured module search path = Default w/o overrides

CONFIGURATION

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=30m%  

(but the problem exists without this line)

OS / ENVIRONMENT

Ubuntu 16.04 on my machine, the managed machine is a Debian 8

SUMMARY

When ansible tries to copy files from my machine to the host, it hangs and never returns.
I'm trying to put 2 files (domain.crt and domain.key) onto my managed machine.

My inventory:

[kanboard]
kanboard ansible_user=ansible  ansible_become=yes ansible_become_method=sudo

STEPS TO REPRODUCE

### THESE 2 STEPS SUCCEED
- name: Crée le dossier SSL
  file: name=/etc/nginx/ssl state=directory

- name: Copie du fichier de config kanboard
  copy:
    src: config.php
    dest: /var/www/html/kanboard/data/config.php
    group: www-data
    owner: www-data
    mode: 0600

### THIS ONE IS STUCK
- name: Copie des certificats SSL
  copy: src="{{ item }}" dest=/etc/nginx/ssl/
  with_items:
    - domain.key
    - domain.crt

Note that {{ certdir }} is a prompted variable, whereas the previous copy uses the files/config.php in my role. Not a problem with prompted variable, see my comments below.

EXPECTED RESULTS

Either the copy succeeds or it fails

ACTUAL RESULTS

The copy is stuck. I have to CTRL C to exit.

Also, as a side-effect, a session is left running on my machine:

➜  Ansible git:(feature/GB/kanboard) ✗ ps auxwf |grep ansible                               
guillau+  2639  0.0  0.0  21312   944 pts/2    S+   11:58   0:00          |   |   \_ grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn ansible
guillau+  2272  0.0  0.0  49444  3520 ?        Ss   11:51   0:00          \_ ssh: /home/guillaume/.ansible/cp/ansible-ssh-kanboard.xxxxx-22-ansible [mux]

I have to kill this "[mux]" process for any subsequent ansible command to succeed, otherwise it stays stuck at the SSH connection...

This is the step that is stuck:

TASK [kanboard : Copie des certificats SSL dans le home] ***********************
task path: /home/guillaume/Dev/qa-infra/Ansible/roles/kanboard/tasks/main.yml:74
<kanboard> ESTABLISH SSH CONNECTION FOR USER: ansible
<kanboard> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=30m)
<kanboard> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<kanboard> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=ansible)
<kanboard> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<kanboard> SSH: PlayContext set ssh_common_args: ()
<kanboard> SSH: PlayContext set ssh_extra_args: ()
<kanboard> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r)
<kanboard> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r kanboard '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1470218510.98-70222443469466 `" && echo ansible-tmp-1470218510.98-70222443469466="` echo $HOME/.ansible/tmp/ansible-tmp-1470218510.98-70222443469466 `" ) && sleep 0'"'"''
<kanboard> PUT /tmp/tmpbOZZIy TO /userHome/ansible/.ansible/tmp/ansible-tmp-1470218510.98-70222443469466/stat
<kanboard> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=30m)
<kanboard> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<kanboard> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=ansible)
<kanboard> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<kanboard> SSH: PlayContext set ssh_common_args: ()
<kanboard> SSH: PlayContext set sftp_extra_args: ()
<kanboard> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r)
<kanboard> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r '[kanboard]'
<kanboard> ESTABLISH SSH CONNECTION FOR USER: ansible
<kanboard> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=30m)
<kanboard> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<kanboard> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=ansible)
<kanboard> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<kanboard> SSH: PlayContext set ssh_common_args: ()
<kanboard> SSH: PlayContext set ssh_extra_args: ()
<kanboard> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r)
<kanboard> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r -tt kanboard '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-msrlxieuahyxrfaaecafhauuaksxizil; LANG=fr_FR.UTF-8 LC_ALL=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 /usr/bin/python /userHome/ansible/.ansible/tmp/ansible-tmp-1470218510.98-70222443469466/stat'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<kanboard> PUT /home/guillaume/Dev/qa-infra/certs/domain.key TO /userHome/ansible/.ansible/tmp/ansible-tmp-1470218510.98-70222443469466/source
<kanboard> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=30m)
<kanboard> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<kanboard> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=ansible)
<kanboard> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<kanboard> SSH: PlayContext set ssh_common_args: ()
<kanboard> SSH: PlayContext set sftp_extra_args: ()
<kanboard> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r)
<kanboard> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/guillaume/.ansible/cp/ansible-ssh-%h-%p-%r '[kanboard]'

After this, my machine is stuck and I have to exit with CTRL+C.

I have no idea how to fix this...

[Gui13]

I just tried to copy the files into my <role>/files/ directory and changed my rule to this:

- name: Copie des certificats SSL
  copy: src="{{ item }}" dest=/etc/nginx/ssl/
  with_items:
    - domain.key
    - domain.crt

And it stays stuck. So it's not the fault of the prompted variable. I suspect a "file rights" issue, but I can't be sure. Also, since the copy of the config.php file succeeds just a moment before, I don't understand what is going on.

As a note: my ansible user only has sudo rights to execute the .ansible/xxx python scripts. That is all. Could this be the reason?

[Gui13]

Some more details: on the managed host, I see openssh sftp-server instances with the user 'ansible' that are left dangling:

root     15857  0.0  0.4 128444  9356 ?        Ss   11:49   0:00  \_ sshd: ansible [priv]
ansible  15862  0.0  0.2 128544  5224 ?        S    11:49   0:00  |   \_ sshd: ansible@notty 
ansible  16489  0.0  0.0   4504   820 ?        Ss   11:49   0:00  |       \_ /usr/lib/openssh/sftp-server
root     16503  0.0  0.4 128444  9408 ?        Ss   11:59   0:00  \_ sshd: ansible [priv]
ansible  16505  0.0  0.2 128444  5432 ?        S    11:59   0:00  |   \_ sshd: ansible@notty 
ansible  17133  0.0  0.0   4504   812 ?        Ss   11:59   0:00  |       \_ /usr/lib/openssh/sftp-server
root     17134  0.0  0.4 128444  9412 ?        Ss   12:03   0:00  \_ sshd: ansible [priv]
ansible  17136  0.0  0.2 128548  5444 ?        S    12:03   0:00  |   \_ sshd: ansible@notty 
ansible  17763  0.0  0.0   4504   852 ?        Ss   12:03   0:00  |       \_ /usr/lib/openssh/sftp-server
root     17796  0.0  0.4 128444  9328 ?        Ss   12:13   0:00  \_ sshd: ansible [priv]
ansible  17798  0.0  0.2 128552  5340 ?        S    12:13   0:00      \_ sshd: ansible@notty 
ansible  18433  0.0  0.0   4504   764 ?        Ss   12:14   0:00          \_ /usr/lib/openssh/sftp-server

There's one dangling for each try that I did..

[Gui13]

I fixed my issue by forcing the use of scp instead of sftp in my ansible.cfg:

[ssh_connection]
scp_if_ssh = true

I still don't know the reason why sftp-server would fail in that case (whereas is seems to behave correctly for all the previous files...).

[jctanner]

@Gui13 does -every- transfer in sftp mode hang, or just some?

[Gui13]

Hi @jctanner, it only hangs on the certificates, not the previous transfers (some other configuration files).

Sorry for the delay!

[sivel]

We do agree that the copy module should not result in a hang. However, the complaint that the ssh connection [mux] processes are left running, is not a problem of ansible.

Those processes are managed by SSH, and their timeout is dictated by the ControlPersist setting. We can also not guarantee that the control persist is only used by ansible, so we cannot just erap them ourselves.

If you have further questions please stop by IRC or the mailing list:

• IRC: #ansible on irc.freenode.net
• mailing list: https://groups.google.com/forum/#!forum/ansible-project