reject_unknown_hosts triggers password prompt

[bitprophet]

ML user Roy Smith reported that he was getting a password prompt when setting reject_unknown_hosts = True, instead of an abort.

Confirmed myself, quickly checked the code and noticed that the host rejection is, sadly, a generic SSHException instance (but with a usefulish message attr) and is caught by the general "did auth fail?" exception check.

Curious whether it ever truly worked, I git bisected; the culprit is fac670a, a commit about fixing password auth that was put in about 10 days after reject_unknown_hosts was implemented. This implies the two features are incompatible as-implemented.

Will check that above commit-diff in detail, but suspect we can route around this by special casing SSHExceptions with the 'unknown server' message.

(Eventually, this should be fixed in the ssh lib itself, so it raises a distinct exception class.)

[roysmith]

Thanks for the quick fix! I tested this out. It does indeed cause a failure on a missing host key, but in the form of an uncaught exception. It would be cleaner if it caught the NetworkError and just printed a failure message.

Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/tasks.py", line 169, in inner
    submit(task.run(*args, **kwargs))
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/tasks.py", line 112, in run
    return self.wrapped(*args, **kwargs)
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/decorators.py", line 178, in inner
    return func(*args, **kwargs)
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/decorators.py", line 47, in inner_decorator
    return func(*args, **kwargs)
  File "/home/roy/songza/deploy/fabfile.py", line 536, in preflight
    run("hostname")
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/network.py", line 462, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/operations.py", line 947, in run
    stdout=stdout, stderr=stderr)
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/operations.py", line 832, in _run_command
    result_stdout, result_stderr, status = _execute(default_channel(), wrapped_command,
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/state.py", line 342, in default_channel
    chan = connections[env.host_string].get_transport().open_session()
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/network.py", line 82, in __getitem__
    self.connect(key)
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/network.py", line 74, in connect
    self[key] = connect(user, host, port)
  File "/home/roy/play/songza/lib/python2.6/site-packages/fabric/network.py", line 324, in connect
    raise NetworkError(msg, e)
NetworkError: Unknown server ap

[bitprophet]

It should be doing so, and was for me when I tested with fab. It looks like you're using Fabric as a library? The custom exception stuff is only handled at higher levels in Fabric, such as when you're using execute (which is almost always a good idea anyways, many newish things are implemented within it.)

[roysmith]

The problem seems to be related to parallel operation. If I run the following script with "fab --fabfile=try.py preflight", I get the stack dump. If I comment out the @parallel, it works (i.e. fails with the message "Fatal error: Unknown server app1.songza.com" and no stack).

---

from fabric.api import run, task, env
from fabric.decorators import parallel, roles

env.reject_unknown_hosts = True

env.roledefs = {
'app_servers': ['app1.songza.com',
'app2.songza.com',
],
}

@task
@parallel
@roles('app_servers')
def preflight():

run("hostname")

[muellert]

I am on an RFC1918 network, and within my network, I can ssh without password, using keys. I also can ssh by hand to remote servers using keys, but when I set 'env.reject_unknown_hosts = True', I get asked for a password for the same remote server, but not the local machines.

I tried this with both 1.7.0 (system package) and a 1.8.1 from a virtualenv, both using Python 2.7.3 on Debian Wheezy/amd64.