Solaris hosts can't be pinged #13410

fboender · 2015-12-02T21:54:53Z

I'm running into an issue with the latest (commit 9bf1aaf; Wed Dec 2 12:11:16 2015) ansible code from github. When trying to ping (or really any ansible action) solaris 10 hosts, the following error is shown:

$ ./ansible --version | head -n1
ansible 2.1.0 (devel 9bf1aaf7f5) last updated 2015/12/02 21:28:57 (GMT +200)
$ ./ansible -m ping -i ~/Projects/ansible-solaris/hosts sol.local
sol.local | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}

However, an older version of Ansibe (v1.9.1) works fine:

$ ansible --version 
ansible 1.9.1
$ ansible -m ping -i ~/Projects/ansible-solaris/hosts sol.local
sol.local | success >> {
    "changed": false,
    "ping": "pong"
}

The sol.local machine is: SunOS unknown 5.10 Generic_147148-26 i86pc i386 i86pc (python v 2.6.4)

Output of -vvvvv:

./ansible -vvvvv -m ping -i ~/Projects/ansible-solaris/hosts sol.local
Using /etc/ansible/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: fboender
<127.0.0.1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<127.0.0.1> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=8022)
<127.0.0.1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<127.0.0.1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=fboender)
<127.0.0.1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<127.0.0.1> SSH: PlayContext set ssh_common_args: ()
<127.0.0.1> SSH: PlayContext set ssh_extra_args: ()
<127.0.0.1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r)
<127.0.0.1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python
sol.local | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}

Output of only the 'ssh' part without running any remote commands:

ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1
--debug output--
Last login: Wed Dec  2 22:46:23 2015 from 10.0.2.2
Oracle Corporation      SunOS 5.10      Generic Patch   January 2005
-bash-3.2$

So it doesn't seem to be a connection problem.

This part may help (some output removed for brevity; I can provide a full dump):

ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o 
KbdInteractiveAuthentication=no -o 
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o 
PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o 
ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 LANG=C LC_ALL=C 
LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, 
"__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'
channel 2: will not send data after close
bash: -c: line 0: syntax error near unexpected token `s'
bash: -c: line 0: `LANG=C LC_ALL=C LC_MESSAGES=C python -uc import sys; [sys.stdout.write(s) 
for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'

Details:

Ansible version:

ansible 2.1.0 (devel 9bf1aaf7f5) last updated 2015/12/02 21:28:57 (GMT +200)
  lib/ansible/modules/core: (detached HEAD cd9a7667aa) last updated 2015/12/02 21:40:36 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD 3c4f954f0f) last updated 2015/12/02 21:40:43 (GMT +200)
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

Host / Master OS: Linux Mint 17 Qiana (Ubuntu 14.04) with Python 2.7.6
Guest OS: SunOS unknown 5.10 Generic_147148-26 i86pc i386 i86pc with python v2.6.4

The text was updated successfully, but these errors were encountered:

bcoca · 2015-12-02T21:56:06Z

cc @amenonsen @abadger @mgedmin

abadger · 2015-12-02T21:59:24Z

Changing to next as the code we think is causing this is only in devel, not in stable-2.0

fboender · 2015-12-02T22:02:17Z

Confirmed that it works in stable-2.0:

$ git checkout stable-2.0
$ ./ansible -m ping -i ~/Projects/ansible-solaris/hosts sol.local
sol.local | SUCCESS => {

fboender · 2015-12-02T22:10:04Z

Turning off pipelining gives a different error:

cat /etc/ansible/ansible.cfg | grep pipe
# Enabling pipelining reduces the number of SSH operations required to 
pipelining = False

sol.local | FAILED! => {
    "failed": true, 
    "msg": "ERROR! failed to resolve remote temporary directory from ansible-tmp-1449094103.56-4432319489801: `(umask 22 && mkdir -p \"`echo $HOME/.ansible/tmp/ansible-tmp-1449094103.56-4432319489801`\" && echo \"`echo $HOME/.ansible/tmp/ansible-tmp-1449094103.56-4432319489801`\")` returned empty string"
}

amenonsen · 2015-12-02T23:56:53Z

@fboender could you please post the full output of ANSIBLE_DEBUG=1 ansible … (with pipelining enabled), please? (And I don't understand the non-pipelining error at all.)

fboender · 2015-12-03T06:34:02Z

With pipelining enabled again:

ANSIBLE_DEBUG=1 ./ansible -m ping -i ~/Projects/ansible-solaris/hosts sol.local
 27895 1449124196.63535: in VariableManager get_vars()
 27895 1449124196.63553: done with get_vars()
 27895 1449124196.64462: getting the remaining hosts for this loop
 27895 1449124196.64475: done getting the remaining hosts for this loop
 27895 1449124196.64492: building list of next tasks for hosts
 27895 1449124196.64498: getting the next task for host sol.local
 27895 1449124196.64503: done getting next task for host sol.local
 27895 1449124196.64510:  ^ task is: TASK: meta 
 27895 1449124196.64513:  ^ state is: HOST STATE: block=0, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 27895 1449124196.64516: done building task lists
 27895 1449124196.64520: counting tasks in each state of execution
 27895 1449124196.64523: done counting tasks in each state of execution
 27895 1449124196.64524: advancing hosts in ITERATING_TASKS
 27895 1449124196.64526: starting to advance hosts
 27895 1449124196.64528: getting the next task for host sol.local
 27895 1449124196.64531: done getting next task for host sol.local
 27895 1449124196.64536:  ^ task is: TASK: meta 
 27895 1449124196.64539:  ^ state is: HOST STATE: block=0, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 27895 1449124196.64542: done advancing hosts to next task
 27895 1449124196.64658: done queuing things up, now waiting for results queue to drain
 27895 1449124196.64664: waiting for pending results...
 27895 1449124196.64666: no more pending results, returning what we have
 27895 1449124196.64668: results queue empty
 27895 1449124196.64670: getting the remaining hosts for this loop
 27895 1449124196.64672: done getting the remaining hosts for this loop
 27895 1449124196.64684: building list of next tasks for hosts
 27895 1449124196.64687: getting the next task for host sol.local
 27895 1449124196.64691: done getting next task for host sol.local
 27895 1449124196.64693:  ^ task is: TASK: ping 
 27895 1449124196.64696:  ^ state is: HOST STATE: block=1, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 27895 1449124196.64698: done building task lists
 27895 1449124196.64699: counting tasks in each state of execution
 27895 1449124196.64701: done counting tasks in each state of execution
 27895 1449124196.64702: advancing hosts in ITERATING_TASKS
 27895 1449124196.64704: starting to advance hosts
 27895 1449124196.64705: getting the next task for host sol.local
 27895 1449124196.64710: done getting next task for host sol.local
 27895 1449124196.64713:  ^ task is: TASK: ping 
 27895 1449124196.64716:  ^ state is: HOST STATE: block=1, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
 27895 1449124196.64717: done advancing hosts to next task
 27895 1449124196.64722: getting variables
 27895 1449124196.64725: in VariableManager get_vars()
 27895 1449124196.64750: done with get_vars()
 27895 1449124196.64765: done getting variables
 27895 1449124196.64768: sending task start callback, copying the task so we can template it temporarily
 27895 1449124196.64770: done copying, going to template now
 27895 1449124196.64773: done templating
 27895 1449124196.64775: here goes the callback...
 27895 1449124196.64777: sending task start callback
 27895 1449124196.64781: entering _queue_task() for sol.local/TASK: ping 
 27895 1449124196.64784: sol.local - putting task (TASK: ping ) in queue
 27895 1449124196.64786: worker is 1 (out of 1 available)
 27895 1449124196.64811: exiting _queue_task() for sol.local/TASK: ping 
 27895 1449124196.64981: done queuing things up, now waiting for results queue to drain
 27895 1449124196.64983: waiting for pending results...
 27903 1449124196.65449: there's work to be done! got a task/handler to work on: TASK: ping 
 27903 1449124196.65456: running TaskExecutor() for sol.local/TASK: ping 
 27903 1449124196.65461: in run()
 27903 1449124196.65493: calling self._execute()
 27903 1449124196.66256: starting attempt loop
 27903 1449124196.66266: running the handler
 27903 1449124196.66342: calling _low_level_execute_command() for command LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python
 27903 1449124196.66349: in _low_level_execute_command() (LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python)
 27903 1449124196.66353: executing the command LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python through the connection
 27903 1449124196.66666: Sending initial data
 27903 1449124196.71122: Sent initial data (73089 bytes)
 27903 1449124196.86390: stdout chunk (state=3):
>>><<<

 27903 1449124196.86399: stderr chunk (state=3):
>>><<<

 27903 1449124196.86416: _execute() done
 27903 1449124196.86420: dumping result to json
 27903 1449124196.86423: done dumping result, returning
 27903 1449124196.86433: done running TaskExecutor() for sol.local/TASK: ping 
 27903 1449124196.86435: sending task result
 27903 1449124196.86468: done sending task result
 27904 1449124196.86525: worker 0 has data to read
 27904 1449124196.86589: got a result from worker 0: <ansible.executor.task_result.TaskResult object at 0x7f122c5d79d0>
 27904 1449124196.86597: sending result: [u'host_unreachable', u'<ansible.executor.task_result.TaskResult object at 0x7f122c5d79d0>']
 27904 1449124196.86622: done sending result
 27895 1449124196.86717: got result from result worker: [u'host_unreachable', u'<ansible.executor.task_result.TaskResult object at 0x7f122c5e0cd0>']
sol.local | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}
 27895 1449124196.86753: no more pending results, returning what we have
 27895 1449124196.86757: results queue empty
 27895 1449124196.86759: getting the remaining hosts for this loop
 27895 1449124196.86762: done getting the remaining hosts for this loop
 27895 1449124196.86774: building list of next tasks for hosts
 27895 1449124196.86777: done building task lists
 27895 1449124196.86778: counting tasks in each state of execution
 27895 1449124196.86780: done counting tasks in each state of execution
 27895 1449124196.86782: all hosts are done, so returning None's for all hosts
 27895 1449124196.86784: done queuing things up, now waiting for results queue to drain
 27895 1449124196.86786: waiting for pending results...
 27895 1449124196.86787: no more pending results, returning what we have
 27895 1449124196.86790: results queue empty
 27895 1449124196.86792: running handlers
 27895 1449124196.86961: RUNNING CLEANUP

abadger · 2015-12-03T15:56:06Z

Reverting the commit in devel for now as @amenonsen will be busy for a few days and others are busy with the 2.0 release. I'll leave this open for now as we'll want to remerge once the problems with the code are fixed.

…ange" This reverts commit f873cc0. Reverting pipelining change for now due to hard to pin down bugs: #13410 #13411

…th sudo+requiretty" This reverts commit 1d8e178. Reverting for now due to hard to pin down bugs: #13410 #13411

This reverts commit f488de8. Reverting for now due to hard to pin down bugs: #13410 #13411

fboender · 2015-12-10T06:30:57Z

The latest develop branch is now working again with Solaris 10.1:

./ansible -i ../../ansible-solaris/hosts -m ping sol.local
sol.local | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
./ansible -i ../../ansible-solaris/hosts -m setup sol.local
sol.local | SUCCESS => {
    "ansible_facts": {

fboender · 2015-12-10T06:54:34Z

Also confirmed to work with both pipelining enabled and disabled.

amenonsen · 2015-12-10T07:01:36Z

No no, devel isn't what you should be testing. I'm afraid you will need to apply the change from PR #13487 on top of devel and test that. devel works only because the pipelining+requiretty change was reverted. The new change has not yet been merged into devel.

fboender · 2015-12-10T08:51:03Z

Whoops, sorry, didn't read it properly.

Just to make sure I'm doing this right:

I applied the PR

$ git fetch origin pull/13487/head:13487
$ git checkout 13487
$ git log --pretty=oneline | head -n1
c09212a4e9a239c0ce16d7e1d2e1da37a38a7fbd Make sudo+requiretty and ANSIBLE_PIPELINING work together

Ran ansible:

$ grep "pipelining =" /etc/ansible/ansible.cfg 
pipelining = True
$ cd bin
$ export PYTHONPATH=../lib
$ ./ansible -i ~/Projects/prs/ansible-solaris/hosts -m ping sol.local
sol.local | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}

Checked against stable-2.0 to see if it's a problem with my setup:

$ git checkout stable-2.0
$ ./ansible -i ~/Projects/prs/ansible-solaris/hosts -m ping sol.local
sol.local | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Output of verbose:

$ git checkout 13487
$ ./ansible -vvvvv -i ~/Projects/prs/ansible-solaris/hosts -m ping sol.local
Using /etc/ansible/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: fboender
<127.0.0.1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<127.0.0.1> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=8022)
<127.0.0.1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<127.0.0.1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=fboender)
<127.0.0.1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<127.0.0.1> SSH: PlayContext set ssh_common_args: ()
<127.0.0.1> SSH: PlayContext set ssh_extra_args: ()
<127.0.0.1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r)
<127.0.0.1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python
sol.local | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}

If I manually run the listed SSH command:

ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/fboender/.ssh/config
debug1: /home/fboender/.ssh/config line 1: Applying options for *
debug1: /home/fboender/.ssh/config line 20: Applying options for 127.0.0.1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 6578
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
  File "<stdin>debug2: Received exit status from master 2
", line Shared connection to 127.0.0.1 closed.
1
    bash: -c: line 0: syntax error near unexpected token `s'
        ^
SyntaxError: invalid syntax

If I remove the remote argument (everything from python -uc... to the end of the line) it works:

ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 LANG=C LC_ALL=C LC_MESSAGES=C ls -lad .
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/fboender/.ssh/config
debug1: /home/fboender/.ssh/config line 1: Applying options for *
debug1: /home/fboender/.ssh/config line 20: Applying options for 127.0.0.1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 6578
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
drwxr-xr-x   4 fboender root           6 Dec 10 09:43 .
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0
Shared connection to 127.0.0.1 closed.

Again with ANSIBLE_DEBUG=1:

$ ANSIBLE_DEBUG=1 ./ansible -vvvvv -i ~/Projects/prs/ansible-solaris/hosts -m ping sol.local
Using /etc/ansible/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
  6917 1449737379.03360: in VariableManager get_vars()
  6917 1449737379.03379: done with get_vars()
  6917 1449737379.04418: getting the remaining hosts for this loop
  6917 1449737379.04430: done getting the remaining hosts for this loop
  6917 1449737379.04446: building list of next tasks for hosts
  6917 1449737379.04450: getting the next task for host sol.local
  6917 1449737379.04458: done getting next task for host sol.local
  6917 1449737379.04465:  ^ task is: TASK: meta (flush_handlers)
  6917 1449737379.04470:  ^ state is: HOST STATE: block=0, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
  6917 1449737379.04473: done building task lists
  6917 1449737379.04477: counting tasks in each state of execution
  6917 1449737379.04480: done counting tasks in each state of execution
  6917 1449737379.04484: advancing hosts in ITERATING_TASKS
  6917 1449737379.04487: starting to advance hosts
  6917 1449737379.04491: getting the next task for host sol.local
  6917 1449737379.04497: done getting next task for host sol.local
  6917 1449737379.04502:  ^ task is: TASK: meta (flush_handlers)
  6917 1449737379.04505:  ^ state is: HOST STATE: block=0, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
  6917 1449737379.04509: done advancing hosts to next task
  6917 1449737379.04634: done queuing things up, now waiting for results queue to drain
  6917 1449737379.04639: waiting for pending results...
  6917 1449737379.04642: no more pending results, returning what we have
  6917 1449737379.04646: results queue empty
  6917 1449737379.04650: getting the remaining hosts for this loop
  6917 1449737379.04654: done getting the remaining hosts for this loop
  6917 1449737379.04673: building list of next tasks for hosts
  6917 1449737379.04677: getting the next task for host sol.local
  6917 1449737379.04682: done getting next task for host sol.local
  6917 1449737379.04686:  ^ task is: TASK: ping 
  6917 1449737379.04689:  ^ state is: HOST STATE: block=1, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
  6917 1449737379.04691: done building task lists
  6917 1449737379.04692: counting tasks in each state of execution
  6917 1449737379.04694: done counting tasks in each state of execution
  6917 1449737379.04696: advancing hosts in ITERATING_TASKS
  6917 1449737379.04697: starting to advance hosts
  6917 1449737379.04699: getting the next task for host sol.local
  6917 1449737379.04703: done getting next task for host sol.local
  6917 1449737379.04706:  ^ task is: TASK: ping 
  6917 1449737379.04708:  ^ state is: HOST STATE: block=1, task=1, rescue=0, always=0, role=None, run_state=1, fail_state=0, pending_setup=False, tasks child state? None, rescue child state? None, always child state? None
  6917 1449737379.04710: done advancing hosts to next task
  6917 1449737379.04715: getting variables
  6917 1449737379.04718: in VariableManager get_vars()
  6917 1449737379.04744: done with get_vars()
  6917 1449737379.04760: done getting variables
  6917 1449737379.04763: sending task start callback, copying the task so we can template it temporarily
  6917 1449737379.04765: done copying, going to template now
  6917 1449737379.04768: done templating
  6917 1449737379.04770: here goes the callback...
  6917 1449737379.04772: sending task start callback
  6917 1449737379.04776: entering _queue_task() for sol.local/TASK: ping 
  6917 1449737379.04779: sol.local - putting task (TASK: ping ) in queue
  6917 1449737379.04781: worker is 1 (out of 1 available)
  6917 1449737379.04806: exiting _queue_task() for sol.local/TASK: ping 
  6917 1449737379.05015: done queuing things up, now waiting for results queue to drain
  6917 1449737379.05020: waiting for pending results...
  6925 1449737379.05521: there's work to be done! got a task/handler to work on: TASK: ping 
  6925 1449737379.05528: running TaskExecutor() for sol.local/TASK: ping 
  6925 1449737379.05533: in run()
  6925 1449737379.05566: calling self._execute()
  6925 1449737379.06296: starting attempt loop
  6925 1449737379.06303: running the handler
  6925 1449737379.06355: calling _low_level_execute_command() for command LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python
  6925 1449737379.06359: in _low_level_execute_command() (LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python)
  6925 1449737379.06362: executing the command LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python through the connection
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: fboender
<127.0.0.1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<127.0.0.1> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=8022)
<127.0.0.1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<127.0.0.1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=fboender)
<127.0.0.1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<127.0.0.1> SSH: PlayContext set ssh_common_args: ()
<127.0.0.1> SSH: PlayContext set ssh_extra_args: ()
<127.0.0.1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r)
<127.0.0.1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python
  6925 1449737379.06590: Sending initial data
  6925 1449737379.11086: Sent initial data (73811 bytes)
  6925 1449737379.11099: stderr chunk (state=3):
>>>OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/fboender/.ssh/config
debug1: /home/fboender/.ssh/config line 1: Applying options for *
debug1: /home/fboender/.ssh/config line 20: Applying options for 127.0.0.1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 6911
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
<<<

  6925 1449737379.26473: stderr chunk (state=3):
>>>debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Control master terminated unexpectedly
Shared connection to 127.0.0.1 closed.
<<<

  6925 1449737379.26507: stdout chunk (state=3):
>>><<<

  6925 1449737379.26511: stderr chunk (state=3):
>>><<<

  6925 1449737379.26527: _execute() done
  6925 1449737379.26530: dumping result to json
  6925 1449737379.26533: done dumping result, returning
  6925 1449737379.26543: done running TaskExecutor() for sol.local/TASK: ping 
  6925 1449737379.26545: sending task result
  6925 1449737379.26572: done sending task result
  6926 1449737379.26630: worker 0 has data to read
  6926 1449737379.26702: got a result from worker 0: <ansible.executor.task_result.TaskResult object at 0x7f4cc28e2dd0>
  6926 1449737379.26709: sending result: [u'host_unreachable', u'<ansible.executor.task_result.TaskResult object at 0x7f4cc28e2dd0>']
  6926 1449737379.26733: done sending result
  6917 1449737379.26822: got result from result worker: [u'host_unreachable', u'<ansible.executor.task_result.TaskResult object at 0x7f4cc28efb10>']
sol.local | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}
  6917 1449737379.26861: no more pending results, returning what we have
  6917 1449737379.26866: results queue empty
  6917 1449737379.26868: getting the remaining hosts for this loop
  6917 1449737379.26871: done getting the remaining hosts for this loop
  6917 1449737379.26886: building list of next tasks for hosts
  6917 1449737379.26890: done building task lists
  6917 1449737379.26892: counting tasks in each state of execution
  6917 1449737379.26894: done counting tasks in each state of execution
  6917 1449737379.26895: all hosts are done, so returning None's for all hosts
  6917 1449737379.26899: done queuing things up, now waiting for results queue to drain
  6917 1449737379.26902: waiting for pending results...
  6917 1449737379.26904: no more pending results, returning what we have
  6917 1449737379.26907: results queue empty
  6917 1449737379.26909: running handlers
  6917 1449737379.27077: RUNNING CLEANUP

amenonsen · 2015-12-10T15:30:23Z

At 2015-12-10 00:51:04 -0800, notifications@github.com wrote:

If I manually run the listed SSH command: …

(This is normal. The entire command is passed as a single string to
execve, but it's separate strings when you execute it in the shell.)

Looking at the command that is being executed:

<127.0.0.1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o Port=8022 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=fboender -o ConnectTimeout=10 -o ControlPath=/home/fboender/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 LANG=C LC_ALL=C LC_MESSAGES=C python -uc 'import sys; [sys.stdout.write(s) for s in iter(sys.stdin.readline, "__EOF__942d747a0772c3284ffb5920e234bd57__\n")]'| LANG=C LC_ALL=C LC_MESSAGES=C /usr/bin/python

I notice it says 'python' in one place and '/usr/bin/python' in another.
What happens if you run "ssh … 127.0.0.1 python" and "ssh … 127.0.0.1
/usr/bin/python"? Does either one fail? Seems unlikely, but…

-- Abhijit

fboender · 2015-12-11T09:38:23Z

Both work:

$ ssh sol.local python --version
Python 2.6.4
$ ssh sol.local /usr/bin/python --version
Python 2.6.4

cristianoliveira · 2016-03-05T01:53:15Z

There is some update in this issue?
Ubuntu 15.04 here and getting same problem >.<

Also, there is some alternative/workaround at this moment?

bcoca · 2016-03-05T04:56:07Z

@cristianoliveira this issue should not be happening to you unless you are running the PR linked above.

cristianoliveira · 2016-03-05T13:31:54Z

@bcoca Yeah you are right. Actually, my error is beeing discussed in that issue #13401

Thanks!

zch-bit · 2016-04-21T22:27:32Z

Anyone know how to avoid/fix this issue ? I also has this one when running on CentOS 6.5 and Ansible 2.0.1.0, Python 2.6.6. This issue happens some times, when the network is good, it looks like it has lower possibilities come out.

lukasojd · 2016-10-29T13:02:46Z

👍 hotfix is install older version :(

ansibot · 2017-04-11T19:11:51Z

@fboender Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information.

Here are the items we could not find in your description:

component name

Please set the description of this issue with this template:
https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE.md

click here for bot help

ansibotdev · 2017-11-22T00:37:49Z

@fboender You have not responded to information requests in this issue so we will assume it no longer affects you. If you are still interested in this, please create a new issue with the requested information.

click here for bot help

bcoca added this to the v2 milestone Dec 2, 2015

bcoca added the bug_report label Dec 2, 2015

abadger modified the milestones: next, v2 Dec 2, 2015

abadger added a commit that referenced this issue Dec 3, 2015

Revert "Update docs and example config for requiretty + pipelining ch…

9caa2b0

…ange" This reverts commit f873cc0. Reverting pipelining change for now due to hard to pin down bugs: #13410 #13411

abadger added a commit that referenced this issue Dec 3, 2015

Revert "Note crab and mgedmin's work to make pipelining compatible wi…

fbb63d6

…th sudo+requiretty" This reverts commit 1d8e178. Reverting for now due to hard to pin down bugs: #13410 #13411

abadger added a commit that referenced this issue Dec 3, 2015

Revert "Make sudo+requiretty and ANSIBLE_PIPELINING work together"

e201a25

This reverts commit f488de8. Reverting for now due to hard to pin down bugs: #13410 #13411

fboender mentioned this issue Dec 4, 2015

KeyError: 'size_total' fboender/ansible-cmdb#24

Closed

amenonsen mentioned this issue Dec 9, 2015

Make sudo+requiretty and ANSIBLE_PIPELINING work together #13487

Closed

fboender closed this as completed Dec 10, 2015

fboender reopened this Dec 10, 2015

ansibot added the affects_2.1 This issue/PR affects Ansible v2.1 label Sep 8, 2016

ansibot added needs_info This issue requires further information. Please answer any outstanding questions. needs_template This issue/PR has an incomplete description. Please fill in the proposed template correctly. labels Apr 11, 2017

gundalow modified the milestone: stable-2.1 Apr 13, 2017

ansibot added the support:core This issue/PR relates to code supported by the Ansible Engineering Team. label Jun 29, 2017

ansibotdev closed this as completed Nov 22, 2017

ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 7, 2018

dagwieers added the solaris Solaris community label Jan 9, 2019

ansible locked and limited conversation to collaborators Apr 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Solaris hosts can't be pinged #13410

Solaris hosts can't be pinged #13410

fboender commented Dec 2, 2015

bcoca commented Dec 2, 2015

abadger commented Dec 2, 2015

fboender commented Dec 2, 2015

fboender commented Dec 2, 2015

amenonsen commented Dec 2, 2015

fboender commented Dec 3, 2015

abadger commented Dec 3, 2015

fboender commented Dec 10, 2015

fboender commented Dec 10, 2015

amenonsen commented Dec 10, 2015

fboender commented Dec 10, 2015

amenonsen commented Dec 10, 2015

fboender commented Dec 11, 2015

cristianoliveira commented Mar 5, 2016

bcoca commented Mar 5, 2016

cristianoliveira commented Mar 5, 2016

zch-bit commented Apr 21, 2016 •

edited

lukasojd commented Oct 29, 2016 •

edited

ansibot commented Apr 11, 2017

ansibotdev commented Nov 22, 2017

Solaris hosts can't be pinged #13410

Solaris hosts can't be pinged #13410

Comments

fboender commented Dec 2, 2015

bcoca commented Dec 2, 2015

abadger commented Dec 2, 2015

fboender commented Dec 2, 2015

fboender commented Dec 2, 2015

amenonsen commented Dec 2, 2015

fboender commented Dec 3, 2015

abadger commented Dec 3, 2015

fboender commented Dec 10, 2015

fboender commented Dec 10, 2015

amenonsen commented Dec 10, 2015

fboender commented Dec 10, 2015

amenonsen commented Dec 10, 2015

fboender commented Dec 11, 2015

cristianoliveira commented Mar 5, 2016

bcoca commented Mar 5, 2016

cristianoliveira commented Mar 5, 2016

zch-bit commented Apr 21, 2016 • edited

lukasojd commented Oct 29, 2016 • edited

ansibot commented Apr 11, 2017

ansibotdev commented Nov 22, 2017

zch-bit commented Apr 21, 2016 •

edited

lukasojd commented Oct 29, 2016 •

edited