When connecting through a gateway, fabric will treat a connection failure as an auth failure and give a password prompt (even when using keyed auth).
You can easily reproduce this issue with the following program:
from fabric.api import *
@task
def reboot_host():
sudo('whoami')
reboot(wait=3600)
sudo('whoami')
Connecting to a host directly and running this task gives
a45e60c70e75:test stwillar$ fab reboot_host
[****.us-west-2.compute.amazonaws.com] Executing task 'reboot_host'
[****.us-west-2.compute.amazonaws.com] sudo: whoami
[****.us-west-2.compute.amazonaws.com] out: sudo password:
[****.us-west-2.compute.amazonaws.com] out: root
[****.us-west-2.compute.amazonaws.com] out:
[****.us-west-2.compute.amazonaws.com] out: sudo password:
[****.us-west-2.compute.amazonaws.com] out:
[****.us-west-2.compute.amazonaws.com] sudo: whoami
[****.us-west-2.compute.amazonaws.com] out: sudo password:
[****.us-west-2.compute.amazonaws.com] out: root
[****.us-west-2.compute.amazonaws.com] out:
This same task, run through a gateway gives the following output:
a45e60c70e75:test stwillar$ fab -g ****.us-west-2.compute.amazonaws.com reboot_host
[****.us-west-2.compute.amazonaws.com] Executing task 'reboot_host'
[****.us-west-2.compute.amazonaws.com] sudo: whoami
[****.us-west-2.compute.amazonaws.com] out: sudo password:
[****.us-west-2.compute.amazonaws.com] out: root
[****.us-west-2.compute.amazonaws.com] out:
[****.us-west-2.compute.amazonaws.com] out: sudo password:
[****.us-west-2.compute.amazonaws.com] out:
No handlers could be found for logger "paramiko.transport"
[****.us-west-2.compute.amazonaws.com] Login password for 'ec2-user':
[****.us-west-2.compute.amazonaws.com] Login password for 'ec2-user':
[****.us-west-2.compute.amazonaws.com] Login password for 'ec2-user':
[****.us-west-2.compute.amazonaws.com] Login password for 'ec2-user':
It will continue to loop asking for the password until the host comes back. I did some digging and found that the issue is because paramiko gives a ChannelException when the gateway fails to connect to the remote host. Fabric treats this as a generic SSHException which will in turn cause a reauthentication attempt. I wrote a unit test that reproduces the issue, and a patch that fixes it.
A pull request will follow shortly.
When connecting through a gateway, fabric will treat a connection failure as an auth failure and give a password prompt (even when using keyed auth).
You can easily reproduce this issue with the following program:
Connecting to a host directly and running this task gives
This same task, run through a gateway gives the following output:
It will continue to loop asking for the password until the host comes back. I did some digging and found that the issue is because paramiko gives a ChannelException when the gateway fails to connect to the remote host. Fabric treats this as a generic SSHException which will in turn cause a reauthentication attempt. I wrote a unit test that reproduces the issue, and a patch that fixes it.
A pull request will follow shortly.