Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh #13401

Closed
tobinlandricombe opened this issue Dec 2, 2015 · 79 comments
Labels
bug This issue/PR relates to a bug.
Milestone

Comments

@tobinlandricombe
Copy link

tobinlandricombe commented Dec 2, 2015

Issue Type:

Bug Report - related to #12594

Ansible Version & Configuration:

$ ansible --version
ansible 2.0.0
  config file = 
  configured module search path = Default w/o overrides

Environment: OSX 10.9.5

Summary:
Upgraded to 2.0.0 and started getting error above

Steps To Reproduce:

$ cat .ssh/config
# remote.site requires a cert; test requires a password
Host test
        ProxyCommand ssh -q -A -W %h:22 me@remote.site -p 28194
$ ansible -m ping -c ssh -u root -k -i test test

Expected Results:

No config file found; using defaults
SSH password: 
test | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Actual Results:

No config file found; using defaults
SSH password: 
test | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}
@tobinlandricombe
Copy link
Author

$ ansible -vvvvv -m ping -c ssh -u root -k -i test test
No config file found; using defaults
SSH password: 
Loaded callback minimal of type stdout, v2.0
<test> ESTABLISH SSH CONNECTION FOR USER: root
<test> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<test> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<test> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<test> SSH: PlayContext set ssh_common_args: ()
<test> SSH: PlayContext set ssh_extra_args: ()
<test> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r)
<test> SSH: EXEC sshpass -d24 ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r -tt test mkdir -p "`echo $HOME/.ansible/tmp/ansible-tmp-1449073900.68-163071728385222`" && echo "`echo $HOME/.ansible/tmp/ansible-tmp-1449073900.68-163071728385222`"
<test> PUT /var/folders/lz/_p94hp8d2wg6jt_3njwddkk00000gn/T/tmp7pzEEQ TO /root/.ansible/tmp/ansible-tmp-1449073900.68-163071728385222/ping
<test> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<test> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<test> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<test> SSH: PlayContext set ssh_common_args: ()
<test> SSH: PlayContext set sftp_extra_args: ()
<test> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r)
<test> SSH: EXEC sshpass -d24 sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r [test]
test | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}

@tobinlandricombe
Copy link
Author

When testing against the box that is acting as the ssh proxy the response is as expected. Implication is that it's something to do with the ProxyCommand or, as in #12594 , the password.

$ cat .ssh/config
# remote.site requires a cert
Host remote.site
Port 67355
$ ansible -m ping -c ssh -i test remote.site
remote.site | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

@tobinlandricombe
Copy link
Author

OpenSSH_7.1p1, OpenSSL 1.0.2d 9 Jul 2015

Note that ssh did not upgrade when ansible did. With ssh as above and ansible 1.9.4, ansible worked. With ssh as above and ansible 2.0.0, ansible started exhibiting the issue.

@amenonsen
Copy link
Contributor

What happens if you run:

sftp -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r [test]

Could you show us the output, please?

@tobinlandricombe
Copy link
Author

$ sftp -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r test
OpenSSH_7.1p1, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /Users/tobin/.ssh/config
debug1: /Users/tobin/.ssh/config line 2: Applying options for *
debug1: /Users/tobin/.ssh/config line 61: Applying options for test
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/tobin/.ansible/cp/ansible-ssh-test-22-root" does not exist
debug1: Executing proxy command: exec ssh -q -A -W test:22 tobin@remote.site -p 28194
debug3: timeout: 10000 ms remain after connect
debug1: permanently_drop_suid: 501
debug1: identity file /Users/tobin/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tobin/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tobin/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tobin/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tobin/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tobin/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tobin/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tobin/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to test:22 as 'root'
debug3: hostkeys_foreach: reading file "/Users/tobin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/tobin/.ssh/known_hosts:167
debug3: load_hostkeys: loaded 1 keys from test
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> zlib@openssh.com
debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> zlib@openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:A4zFNpkYSMSXXXXXXXXXXXYEwsXaFu6X5s
debug3: hostkeys_foreach: reading file "/Users/tobin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/tobin/.ssh/known_hosts:167
debug3: load_hostkeys: loaded 1 keys from test
debug1: Host 'test' is known and matches the ECDSA host key.
debug1: Found key in /Users/tobin/.ssh/known_hosts:167
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /Users/tobin/.ssh/id_rsa (0x7fc100d008a0),
debug2: key: /Users/tobin/.ssh/id_dsa (0x0),
debug2: key: /Users/tobin/.ssh/id_ecdsa (0x0),
debug2: key: /Users/tobin/.ssh/id_ed25519 (0x0),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/tobin/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /Users/tobin/.ssh/id_dsa
debug3: no such identity: /Users/tobin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /Users/tobin/.ssh/id_ecdsa
debug3: no such identity: /Users/tobin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /Users/tobin/.ssh/id_ed25519
debug3: no such identity: /Users/tobin/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@test's password: 
debug2: we sent a password packet, wait for reply
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to test (via proxy).
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path /Users/tobin/.ansible/cp/ansible-ssh-test-22-root.p96LHRy2WRdl0EdA
debug2: fd 6 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [/Users/tobin/.ansible/cp/ansible-ssh-test-22-root]
debug3: muxserver_listen: mux listener channel 0 fd 6
debug1: control_persist_detach: backgrounding master process
debug2: control_persist_detach: background process is 1993
debug2: fd 6 setting O_NONBLOCK
debug1: forking to background
debug1: Entering interactive session.
debug2: set_control_persist_exit_time: schedule exit in 60 seconds
debug1: multiplexing control connection
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [mux-control]
debug3: channel_post_mux_listener: new mux channel 1 fd 7
debug3: mux_master_read_cb: channel 1: hello sent
debug2: set_control_persist_exit_time: cancel scheduled exit
debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4
debug2: process_mux_master_hello: channel 1 slave version 4
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4
debug2: process_mux_alive_check: channel 1: alive check
debug3: mux_client_request_alive: done pid = 1995
debug3: mux_client_request_session: session request sent
debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 54
debug2: process_mux_new_session: channel 1: request tty 0, X 0, agent 0, subsys 1, term "xterm-256color", cmd "sftp", env 0
debug3: process_mux_new_session: got fds stdin 8, stdout 9, stderr 10
debug2: fd 8 setting O_NONBLOCK
debug3: fd 9 is O_NONBLOCK
debug1: channel 2: new [client-session]
debug2: process_mux_new_session: channel_new: 2 linked to control channel 1
debug2: channel 2: send open
debug2: callback start
debug2: client_session2_setup: id 2
debug1: Sending subsystem: sftp
debug2: channel 2: request subsystem confirm 1
debug3: mux_session_confirm: sending success reply
debug2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug1: mux_client_request_session: master session id: 2
debug2: channel 2: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 2
debug2: subsystem request accepted on channel 2
debug2: Remote version: 3
debug2: Server supports extension "posix-rename@openssh.com" revision 1
debug2: Server supports extension "statvfs@openssh.com" revision 2
debug2: Server supports extension "fstatvfs@openssh.com" revision 2
debug2: Server supports extension "hardlink@openssh.com" revision 1
debug2: Server supports extension "fsync@openssh.com" revision 1
Connected to test.
debug3: Sent message fd 3 T:16 I:1
debug3: SSH_FXP_REALPATH . -> /root size 0
sftp> 

@amenonsen
Copy link
Contributor

Could you now try the same command with sshpass -p <your password> prefixed? (Doesn't have to be -p, you can use -e and set SSHPASS=<the password> in the environment if you prefer.) What changes? If that works, the next thing to try is to run your ansible command with ANSIBLE_SCP_IF_SSH=y in the environment. The "data could not be sent to the remote host" means the ssh command is exiting with status 255, we just have to figure out why.

@tobinlandricombe
Copy link
Author

Using sshpass with the sftp command works as expected.

And ANSIBLE_SCP_IF_SSH=y ansible -m ping -c ssh -u root -k -i test test works without issue.

Thanks for your help @amenonsen, that resolves the issue for me.

@amenonsen
Copy link
Contributor

OK, so sshpass and sftp just won't work from within Ansible. sshpass opens a pseudo-tty to talk to sftp and feed it the password, but doesn't expect it to behave interactively thereafter (which ssh normally doesn't, but sftp does; this explains why the first EXEC in your output doesn't fail). If you run it by hand it works because it opens /dev/tty, but not when Ansible executes it. I think (but have not verified) that if you set ANSIBLE_SSH_PIPELINING=y, then sshpass with also break ssh. If you feel like trying that and confirming the results, it would be a useful data point.

This should probably be documented somewhere.

@tobinlandricombe
Copy link
Author

Like this?

$ ANSIBLE_SSH_PIPELINING=y ansible -vvvvv -m ping -c ssh -u root -k -i test test
No config file found; using defaults
SSH password: 
Loaded callback minimal of type stdout, v2.0
<test> ESTABLISH SSH CONNECTION FOR USER: root
<test> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<test> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<test> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<test> SSH: PlayContext set ssh_common_args: ()
<test> SSH: PlayContext set ssh_extra_args: ()
<test> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r)
<test> SSH: EXEC sshpass -d24 ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/tobin/.ansible/cp/ansible-ssh-%h-%p-%r test LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python
test | SUCCESS => {
    "changed": false, 
    "invocation": {
        "module_args": {}, 
        "module_name": "ping"
    }, 
    "ping": "pong"
}

@jooadam
Copy link

jooadam commented Jan 16, 2016

Sorry, but it’s not clear: is this a bug and can we expect this to be fixed, or should we treat scp_if_ssh as the permanent solution?

@moises-silva
Copy link

I have the same issue as soon as I upgraded to Ansible 2.0 on Arch Linux.

OpenSSH_7.1p2, OpenSSL 1.0.2e 3 Dec 2015
ansible 2.0.0.2

Exporting ANSIBLE_SCP_IF_SSH=y works as expected

@soar
Copy link

soar commented Jan 24, 2016

I can confirm this issue. Using add_host module with ansible_ssh_pass - gives me SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh. But with ANSIBLE_SCP_IF_SSH=y - all works fine. Ansible 2.0.

@evannook
Copy link

Same here. ANSIBLE_SCP_IF_SSH=y solves this problem.

@nbari
Copy link

nbari commented Feb 9, 2016

Same here:

$ ansible --version                                                                                
ansible 2.0.0.2
  config file = /Users/nbari/ansible/ansible.cfg
  configured module search path = Default w/o overrides

Error I get:
fatal: [10.0.8.4]: UNREACHABLE! => {"changed": false, "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", "unreachable": true}

Used to work with ansible < 2

@dennis-benzinger-hybris

Strange. If I use a hostname in my inventory I have this issue. But with an IP address it's ok.

@m3nu
Copy link

m3nu commented Feb 19, 2016

Same issue. ANSIBLE_SCP_IF_SSH=ymakes it work.

@lintmint
Copy link

Same issue on CentOS 6.5 with Ansible 2.0.0.2
Deployment works will work by first doing:
export ANSIBLE_SCP_IF_SSH=y

@odinsy
Copy link

odinsy commented Feb 20, 2016

CentOS 6
ansible 2.1.0

Same issue. Resolved with:
sed -i '/# scp_if_ssh = True/s/^# //g' /etc/ansible/ansible.cfg

@greenscar
Copy link

OSX
Same issue. export ANSIBLE_SCP_IF_SSH=y provides a workaround.

@newhomie
Copy link

Same issue.

Ubuntu 14.04
ansible 2.0.0.2

It does work after added a line scp_if_ssh = True in /etc/ansible/ansible.cfg

@bencromwell
Copy link

Is anyone else having the same issue even after setting scp_if_ssh to true?

Ubuntu 14.04.4
Ansible 2.0.0.2

@chekolyn
Copy link

@bencromwell

Yes, but exporting the variable in the env works.
export ANSIBLE_SCP_IF_SSH=y

@jacobwoffenden
Copy link

OSX 10.11.3
ansible 2.0.1.0
Uncommenting scp_if_ssh = True resolved this

@maedox
Copy link
Contributor

maedox commented Mar 4, 2016

I'm running Ansible on Linux Mint 17.3 and none of the proposed settings work for me. ansible-playbook works fine, but trying to run ansible -i hosts $somehost -m setup fails with this error.

ansible 2.0.1.0

@cristianoliveira
Copy link

Even setting ANSIBLE_SCP_IF_SSH=y. I am getting this error below

My current env:
Ubuntu: 15.04
Ansible: 2.0.0
vagrant: 1.8

fatal: [vagrant@127.0.0.1]: FAILED! => {"failed": true, "msg": "ERROR! failed to transfer file to /home/vagrant/.ansible/tmp/ansible-tmp-1457146157.15-52289027718893/setup:\n\nssh: Could not resolve hostname 127.0.0.1]: Name or service not known\r\nlost connection\n"}

@jacobwoffenden
Copy link

@cristianoliveira have you tried adding scp_if_ssh = True to your ansible.cfg ?

@cristianoliveira
Copy link

Hey @Jacoblw I hoped both configs were the same. Anyway, I did and got the same error.

zmc added a commit to ceph/ceph-cm-ansible that referenced this issue Aug 1, 2016
This may help with some of the problems we've been seeing.

Related to http://tracker.ceph.com/issues/16826
Inspired by
ansible/ansible#13401 (comment)

Signed-off-by: Zack Cerza <zack@redhat.com>
@babibza
Copy link

babibza commented Aug 11, 2016

is there anything in ansible that if ssh or ping failed then wait and retry.

@saeeda
Copy link

saeeda commented Aug 16, 2016

Receiving "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh",
"unreachable": true
Current versions: ansible 2.1.1.0; DISTRIB_DESCRIPTION="Ubuntu 15.10"
And the following resolved it.
Export ANSIBLE_SCP_IF_SSH=y
I wish it would have been resolved in the code itself, rather than users setting env variable for it to work.

@tima
Copy link
Contributor

tima commented Aug 16, 2016

@saeeda: You cannot discern a downed SFTP service from one that isn't there by default.

@saeeda
Copy link

saeeda commented Aug 17, 2016

@tima when activating my virtualenv, I just export that env variable, and the hosts that were giving error, start working.
I was just wondering whether it is possible for the Ansible to fall back to this option, when SSH connection to that host fails. Instead of user having to set the environment variable for it to work.

@ssbarnea
Copy link
Member

Why is this task marked as resolved when we can all reproduce it with current version of Ansible? -- 2.1.1.0

It is clear to me the without setting ANSIBLE_SCP_IF_SSH=y before running ansible, the template module will fail with a meaningless error message: complaining about not being able to ssh. That's clearly a bug in the template module.

@ssbarnea
Copy link
Member

At least I am glad that I was able to workaround this bug by adding this to ansible.cfg:

[ssh_connection]
scp_if_ssh = True

@joaosa
Copy link

joaosa commented Oct 1, 2016

In my instance of this issue, adding --ssh-extra-args="-o ControlPath=none" --sftp-extra-args="-o ControlPath=none" (as mentioned on #13401 (comment)) worked around this.

@ahrib
Copy link

ahrib commented Oct 5, 2016

thanks all, workaround corrects issue for me too

Ansible 2.1.2.0

@boonchu
Copy link

boonchu commented Oct 15, 2016

This works for me if you want to automate it.
$ sed -i ‘/[ssh_connection/ s/$/ \nscp_if_ssh = True/' ansible.cfg

@dave-martin
Copy link

dave-martin commented Oct 20, 2016

I'm getting the same error with ansible 2.1.2.0 trying to connect to a fresh Ubuntu 16.04 server installation.

My ansible.cfg file:

[defaults]
hostfile    = hosts
remote_user = root
log_path    = /var/log/ansible.log

[ssh_connection]
pipelining   = True
scp_if_ssh   = True
control_path = /tmp/ansible-ssh-%%h-%%p-%%r

Here's the output from ansible -vvvv -m setup 192.168.5.160:

Using /home/ansible/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<192.168.5.160> ESTABLISH SSH CONNECTION FOR USER: root
<192.168.5.160> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/tmp/ansible-ssh-%h-%p-%r 192.168.5.160 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python && sleep 0'"'"''
192.168.5.160 | UNREACHABLE! => {
    "changed": false, 
    "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}

I can SSH to the remote system as root using the public key.

Adding the --ssh-extra-args made no difference.

Any hints on what to try in order to fix the problem?

Thanks!

@shoan
Copy link

shoan commented Oct 20, 2016

Remember that scp_if_ssh = True has to be under the [ssh_connection] section in ansible.cfg. I spent some time breaking my head about this.

@dave-martin
Copy link

Thanks. I do have scp_if_ssh in the [ssh_connection] section (see above). Putting it there allowed the ssh connection process to proceed further, but there is still no data transfer.

@shoan
Copy link

shoan commented Oct 21, 2016

<192.168.5.160> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/tmp/ansible-ssh-%h-%p-%r 192.168.5.160 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python && sleep 0'"'"''

Are you able to connect using the ssh command from the ansible output?

@dave-martin
Copy link

I believe my problem was that I needed to have the public key for the account I was running the playbook from installed in the root account on the remote system. I thought for sure I had checked that, but...

It's working OK now. Sorry for the false alarm.

@jhkrischel
Copy link

Just thought I'd leave a note here - I ran into a very similar symptom, but it was due to pbrun being enabled on the target servers.

I fixed it by setting ansible_become_method=pbrun

@O2Graphics
Copy link

Hello,
This almost one year old critical bug still isn't solved.
On Debian, with ansible 2.1.1, I tried changing scp_if_ssh, control_path, pipelining, sftp_batch_mode, retries (and different combinations of them) according to the comments above and I'm still getting this "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh" error.

In ansible 2.1.1, compared to 2.0.x, it's a bit better : now the error appears only when using async/polling commands.
But it still makes ansible 2.x completely unusable, I had to go back to 1.7.2.

Please fix this bug, rollbacking to the code which was working before 2.x if needed.

@nirik
Copy link

nirik commented Oct 31, 2016

Those of you still seeing this should bring it up on the mailing list and/or IRC. As noted when this was closed: "Because this project is very active, we're unlikely to see comments on closed tickets, though the mailing list is a great way to get involved or discuss this one further."

So, it's unlikely any ansible folks are seeing your comments...

@O2Graphics
Copy link

O2Graphics commented Oct 31, 2016

Thanks @nirik for your answer.
Well, finally, by running ansible with "ANSIBLE_DEBUG=1 ansible-playbook..." I found my issue isn't related to this one anymore. The problem is in fact similar to #15766 or #16938

@untoldone
Copy link

@O2Graphics do you have specifics on how you fixed it? I am having the same type of issue with Debian

@O2Graphics
Copy link

O2Graphics commented Nov 16, 2016

@untoldone sadly we still have weird issues with ansible 2.1.1 (bugs differents to #13401, #15766 and #16938).

Here are some of the stuff we needed to change from 1.7.2 to 2.1.1:

  • For FreeBSD hosts, using a group var of "ansible_python_interpreter: /usr/local/bin/python2.7" instead of "ansible_python_interpreter: /usr/bin/env python2.7"
  • Sometime ansible-playbook doesn't find some variables, and need to be run with "--flush-cache"
  • Fix some templates: some braces and double quotes were missing (it was probably our fault and 1.x was more tolerant than 2.x)

Lastly, the best option we had was to go back to ansible 1.7.2 until we find more time to test the strange bugs of v2.1.x, search the bugtracker and/or report them.
At least 1.7.2 works great with no issue at all for us and seems much faster than 2.1.1.

@untoldone
Copy link

@O2Graphics Ugh ... that's really a bummer. I feel like these types of bugs + breaking changes + not obvious amounts of attention from the project owners makes me uncomfortable with Ansible as a whole.

@abadger
Copy link
Contributor

abadger commented Nov 16, 2016

Thanks @nirik for pointing me at this bug.

@O2Graphics Your first bullet point should be fixed via #15638

The second one is probably an issue with var caching. That's not an area I'm familiar with so we'll need a bug report with a reproducer for someone familiar with the cache plugins to look into.

The last one I believe you're right. There was a lot of tightening of the code from 1.x to 2.x. Some of that made previously silent bugs now throw errors.

For everyone in general: this bug is closed because, to the best of our knowledge and ability to test, the initially reported problem in this bug has been fixed via #15829 This can sometimes be tricky for us to determine on a bug this size when the original reporter isn't able to test our fix: we may have fixed a different issue with similar symptoms. In a similar vein, the reports of errors with similar symptoms are not the same issue (because they have a different cause than what was fixed) even if they result in similar symptoms. The best thing to do if you are still experiencing problems with code containing the fix mentioned above is to open a new bug report with enough detail that we can reproduce the issue. That will help us figure out in what other circumstances the code does the wrong thing.

I'm going to lock this issue to further comments because the core team does not review closed bugs as part of their regular workflow and therefore comments here would likely never be seen leading to frustration that they never get addressed. Thanks for your understanding.

@ansible ansible locked and limited conversation to collaborators Nov 16, 2016
@ansibot ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 7, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue/PR relates to a bug.
Projects
None yet
Development

No branches or pull requests