Gateway fails when target prompts for password #957
Comments
I've got the exact same problem using fabric 1.8 and using roles decorators and env.gateway. |
That's an issue for me too, env.gateway and target asks for password. It seems to hang/stall. |
@adrianbn gave more details in this mailing list thread (check both of his messages for whole picture): http://lists.nongnu.org/archive/html/fab-user/2013-10/msg00004.html |
Some more debugging: gateway$ strace -p 12345 gateway$ lsof -p 12345 |
Going to see if I can replicate this; most of my personal testing of the gateway feature was using SSH keys, not passwords, so it's possible password prompt interaction is straight up broken with the gateway feature enabled. Hopefully I will figure this out this afternoon, but if I don't, it would be super helpful to get this info from @darthmdh @adrianbn @Bengrunt and @dgonza:
My guess is that the main brokenness is when the gateway requires a password, since the gateway's own IO streams are suppressed in this setup. Will poke. |
Interestingly when I attempt to perform password auth to my localhost, via my localhost, I get a traceback, not a hang (this is under the 1.5 branch, same result under 1.8 and master):
Will poke more with the above combinations of key/pass on gateway/target. |
Using my localhost as a gateway, with SSH keys enabled, and targeting a virtual machine with only password auth enabled, I get the hang behavior and these tracebacks (on 1.5):
Which is the same as @adrianbn's email description. So this implies that folks getting the hang behavior are using keys on their gateway but not their target. (Though as above, using password auth on both remote points is also problematic.) Confirmed that on the 1.5 branch, using SSH keys on both gateway + target still works fine. Will work off of this version of the code & port any fixes forwards all the way to 1.8. |
OK, it's actually hanging on the attempted 2nd connect to the target (via the gateway) after dealing w/ password prompt entry. So it's something in the Paramiko level. Will scan open Paramiko tickets real quick, then dig. |
To answer your question: I was using SSH pubkey auth on both the gateway and final host. |
It looks like the gateway socket is being closed during the exception/prompt/try again cycle (going by a repr() print of the socket during debugging). Sounds like a reasonable thing to cause this issue. Hopefully it's our doing & we can not do it. @Bengrunt If the above is right, your situation may be similar, since SSH auth (at least in paramiko/fab) is kinda-sorta a chain of exceptions + trying subsequent keys. (Especially given that paramiko's multi-key handling is kinda wonky even without gateway stuff involved.) If I can find a solution to this that is basically "don't close the gateway socket prematurely / recreate it as needed", I'd definitely recommend you retry once it's published :) watch this space... |
Yup, derp. We have an explicit Problem here is the flow looks like this, for gateways:
So we need a way to either:
|
Yea, I think I'm being dumb about exceptions - if we move the socket closure outside the while loop, in a simple "try/finally", it should do what we want (maximally lazy socket closure within |
With that "fixed" I now get our old friend
|
Main issue now is that the top level Basically |
While I'm talking to myself:
|
Trying this command
$ fab -H usr-a@target-host -g usr-b@intermediate-host -- whoami
will fail if the key for the local user is not present in the target host.
It will prompt for the password but it won't respond after it is entered.
If the local key is present in the target host it will log in normally.
The text was updated successfully, but these errors were encountered: