Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delegate_to hangs in v2.0 #12816

Closed
kettlewell opened this issue Oct 18, 2015 · 8 comments
Closed

Delegate_to hangs in v2.0 #12816

kettlewell opened this issue Oct 18, 2015 · 8 comments
Assignees
@abadger
Labels
bug This issue/PR relates to a bug. P2 Priority 2 - Issue Blocks Release
Milestone
2.0

Comments

@kettlewell
Copy link

Issue Type:
Bug Report

Ansible Version:
v2.0.0-0.3.beta1-70-g71a1a3c

Ansible Configuration:
mostly default

Environment:
Ubuntu

Summary:
Delegate_to appears to be causing the playbook to hang indefinitely.

Possibly related to #12527 , which was fixed by 3bd71d0

Here's the inputs:

- name: Install to Staging Directory
  synchronize:
    src: "{{ tmp_dir }}/"
    dest: "{{ publish_directory }}/{{ instance_name }}_{{ build_version }}"
    delete: yes
    recursive: yes
    mode: push
  delegate_to: "{{ inventory_hostname }}"

and the output:

task path: /Depots/mkettlewell_ansible_linux/roles/maxpoint_minion_deploy/tasks/install.yml:10
 [WARNING]: no remote address found for delegated host "hostname redacted", using its name by default

<"hostname redacted"> ESTABLISH SSH CONNECTION FOR USER: someuser
<"hostname redacted"> SSH: EXEC sshpass -d30 ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=someuser-o ConnectTimeout=10 -o ControlPath=/home/matt/.ansible/cp/ansible-ssh-%h-%p-%r -tt "hostname redacted" (umask 22 && mkdir -p "$(echo $HOME/.ansible/tmp/ansible-tmp-1445193136.45-83011757570616)" && echo "$(echo $HOME/.ansible/tmp/ansible-tmp-1445193136.45-83011757570616)")
<"hostname redacted"> PUT /tmp/tmp8_8fXp TO /home/someuser/.ansible/tmp/ansible-tmp-1445193136.45-83011757570616/synchronize
<"hostname redacted"> SSH: EXEC sshpass -d30 scp -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=someuser-o ConnectTimeout=10 -o ControlPath=/home/matt/.ansible/cp/ansible-ssh-%h-%p-%r /tmp/tmp8_8fXp ["hostname redacted"]:/home/someuser/.ansible/tmp/ansible-tmp-1445193136.45-83011757570616/synchronize
<"hostname redacted"> ESTABLISH SSH CONNECTION FOR USER: someuser
<"hostname redacted"> SSH: EXEC sshpass -d30 ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=someuser-o ConnectTimeout=10 -o ControlPath=/home/matt/.ansible/cp/ansible-ssh-%h-%p-%r -tt "hostname redacted" /bin/sh -c 'sudo -H -p "[sudo via ansible, key=gdmdattgwkuwumqyrupgbzonxclpnnen] password: " -S -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-gdmdattgwkuwumqyrupgbzonxclpnnen; LANG=C LC_MESSAGES=C LC_CTYPE=C /usr/bin/python /home/someuser/.ansible/tmp/ansible-tmp-1445193136.45-83011757570616/synchronize; rm -rf "/home/someuser/.ansible/tmp/ansible-tmp-1445193136.45-83011757570616/" > /dev/null 2>&1'"'"''


^CProcess WorkerProcess-7:
Traceback (most recent call last):

There's additionally a long traceback upon hitting Cntl-C that may be useful.

^CProcess WorkerProcess-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/ansible-2.0.0-py2.7.egg/ansible/executor/process/worker.py", line 100, in run
    (host, task, basedir, job_vars, play_context, shared_loader_obj) = self._main_q.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    res = self._recv()
KeyboardInterrupt
Process WorkerProcess-15:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
Process WorkerProcess-4:
Process WorkerProcess-9:
  File "/usr/local/lib/python2.7/dist-packages/ansible-2.0.0-py2.7.egg/ansible/executor/process/worker.py", line 100, in run
Process WorkerProcess-5:
Traceback (most recent call last):
Process WorkerProcess-18:
Process WorkerProcess-23:
    (host, task, basedir, job_vars, play_context, shared_loader_obj) = self._main_q.get()
Process WorkerProcess-8:
Traceback (most recent call last):
Process WorkerProcess-13:
Traceback (most recent call last):
Process WorkerProcess-16:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process WorkerProcess-22:
Traceback (most recent call last):
Process WorkerProcess-20:
Traceback (most recent call last):
Process WorkerProcess-12:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/ansible-2.0.0-py2.7.egg/ansible/executor/process/worker.py", line 100, in run
    (host, task, basedir, job_vars, play_context, shared_loader_obj) = self._main_q.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    res = self._recv()
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/ansible-2.0.0-py2.7.egg/ansible/executor/process/worker.py", line 100, in run
    (host, task, basedir, job_vars, play_context, shared_loader_obj) = self._main_q.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    res = self._recv()
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/ansible-2.0.0-py2.7.egg/ansible/executor/process/worker.py", line 100, in run
    self.run()
    (host, task, basedir, job_vars, play_context, shared_loader_obj) = self._main_q.get()
  File "/usr/local/lib/python2.7/dist-packages/ansible-2.0.0-py2.7.egg/ansible/executor/process/worker.py", line 100, in run
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    (host, task, basedir, job_vars, play_context, shared_loader_obj) = self._main_q.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    res = self._recv()
KeyboardInterrupt
Process WorkerProcess-24:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
Process WorkerProcess-14:
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process WorkerProcess-21:
Process WorkerProcess-17:
Process WorkerProcess-19:
Process WorkerProcess-25:
    self.run()
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/ansible-2.0.0-py2.7.egg/ansible/executor/process/worker.py", line 100, in run
    (host, task, basedir, job_vars, play_context, shared_loader_obj) = self._main_q.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 117, in get
    res = self._recv()
KeyboardInterrupt
 ____________
< PLAY RECAP >
 ------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

"hostname redacted" : ok=26   changed=4    unreachable=0    failed=0
@jimi-c jimi-c added bug_report P2 Priority 2 - Issue Blocks Release labels Oct 19, 2015
@jimi-c jimi-c added this to the v2 milestone Oct 19, 2015
@abadger
Copy link
Contributor

abadger commented Oct 22, 2015

@kettlewell, couple questions, did this work in 1.9? And what are you using to authenticate your ssh connection?

Oh, and what are you expecting to happen? I think you want to copy a directory tree on the remote host to a new directory tree on the remote host. Is that correct?

@kettlewell
Copy link
Author

We rolled back to 1.9.4, and it works flawlessly.

Our authentication is SSH with passwords protected with Ansible vault.

Your assumption is correct - we're copying files from tmp to another folder on the remote host.

I believe the output above is with -vvvv, but if there's any other logs or info you need from this end, let me know.

@abadger
Copy link
Contributor

abadger commented Oct 22, 2015

So probably in 1.9.x we're having rsync do a local copy on the remote box but in 2.0 we're telling rsync to do a copy via ssh on the remote box. I'll look at what code we might have left out that made that case work before.

abadger added a commit that referenced this issue Oct 23, 2015
@abadger
Potential fix for #12816
360aba4
@abadger
Copy link
Contributor

abadger commented Oct 23, 2015

That's a possible fix for this issue^ but it doesn't look like what 1.9.x is doing to prevent this from happening so I'm going to look into it more. There could be other behaviour that is buggy with this change so I need to do some more testing of this code to see what else is affected before merging it to the devel branch.

abadger added a commit that referenced this issue Oct 23, 2015
@abadger
Potential fix for #12816
ca32d5b
abadger added a commit that referenced this issue Oct 23, 2015
@abadger
Merge pull request #12892 from ansible/synchronize-remote-to-itself
1644eeb 
Potential fix for #12816
@abadger
Copy link
Contributor

abadger commented Oct 23, 2015

I couldn't find any problems with the fix in testing so I've gone ahead and merged it.

#12892

@abadger
Copy link
Contributor

abadger commented Oct 23, 2015

Closing This Ticket

Hi!

We believe recent commits (likely detailed above) should resolve this question or problem for you.

This will also be included in the next major release.

If you continue seeing any problems related to this issue, or if you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular
issue is resolved.

Thank you!

@abadger abadger closed this as completed Oct 23, 2015
photoninger pushed a commit to photoninger/ansible that referenced this issue Oct 26, 2015
@abadger @photoninger
Potential fix for ansible#12816
9082247
@cnighojkar
Copy link

Hello ,
we have your fix available , but still ansible-playbook hang at the task having delgate_to: localhost

Kindly let us know , if you need additional info on this.

@dannyman
Copy link

dannyman commented Oct 11, 2016 •
edited

Q: Is the Ansible version on the managed hosts a factor or just on the driving host?
A: No (Edit)

Therefor, the problem I am experiencing is either unrelated to this issue, or a manifestation of the same issue, but in 1.9.6

We may be seeing this issue as well. One playbook with synchronize and delegate_to among Ansible-1.9 hosts works fine. I'm trying to do the same with Ubuntu 16.04 hosts, which have Ansible 2.0.0.2 (... also tried 2.1.2.0 via main PPA ... requested Ansible to add xenial to the ansible-1.9 PPA for testing ...)

We have:
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r -o ForwardAgent=yes

Here is our failing task:

- name: Rsync /etc/kubernetes/pki to master hosts
  synchronize: src=/etc/kubernetes/pki/ dest=/etc/kubernetes/pki/
  delegate_to: "{{kubeadm_master_host}}"
  remote_user: root
  when: ansible_fqdn in etcd_master_hosts and inventory_hostname != "{{kubeadm_master_host}}"

Here is output:

TASK: [kub | Rsync /etc/kubernetes/pki to master hosts] *********************** 
skipping: [kub-test0.mtv.qxxxxxxxd.com]
skipping: [kub-test4.mtv.qxxxxxxxd.com]
skipping: [kub-test2.mtv.qxxxxxxxd.com]
<kub-test0.mtv.qxxxxxxxd.com> ESTABLISH CONNECTION FOR USER: root
<kub-test0.mtv.qxxxxxxxd.com> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r -o ForwardAgent=yes -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 kub-test0.mtv.qxxxxxxxd.com /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1476145162.01-188872116897460 && echo $HOME/.ansible/tmp/ansible-tmp-1476145162.01-188872116897460'
<kub-test0.mtv.qxxxxxxxd.com> ESTABLISH CONNECTION FOR USER: root
<kub-test0.mtv.qxxxxxxxd.com> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r -o ForwardAgent=yes -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 kub-test0.mtv.qxxxxxxxd.com /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1476145162.02-35924124767365 && echo $HOME/.ansible/tmp/ansible-tmp-1476145162.02-35924124767365'
<kub-test0.mtv.qxxxxxxxd.com> PUT /tmp/tmp7D_MLC TO /root/.ansible/tmp/ansible-tmp-1476145162.02-35924124767365/synchronize
<kub-test0.mtv.qxxxxxxxd.com> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r -o ForwardAgent=yes -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 kub-test0.mtv.qxxxxxxxd.com /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=vdooohebmkaroanvshbgmswsjmhxbazs] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-vdooohebmkaroanvshbgmswsjmhxbazs; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1476145162.02-35924124767365/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-1476145162.02-35924124767365/ >/dev/null 2>&1'"'"''
<kub-test0.mtv.qxxxxxxxd.com> PUT /tmp/tmpxs0QdY TO /root/.ansible/tmp/ansible-tmp-1476145162.01-188872116897460/synchronize
<kub-test1.mtv.qxxxxxxxd.com> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r -o ForwardAgent=yes -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 kub-test0.mtv.qxxxxxxxd.com /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=iilazdxcviftzbybdcafqigwtssowkgn] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-iilazdxcviftzbybdcafqigwtssowkgn; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1476145162.01-188872116897460/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-1476145162.01-188872116897460/ >/dev/null 2>&1'"'"''

If I ssh root@kubeadm_master_host I can then ssh root@inventory host manually without any trouble.

@ansibot ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 7, 2018
@ansible ansible locked and limited conversation to collaborators Apr 25, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@abadger abadger

Labels
bug This issue/PR relates to a bug. P2 Priority 2 - Issue Blocks Release
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
6 participants
@jimi-c @abadger @dannyman @cnighojkar @kettlewell @ansibot