Fabric hangs when using SSH proxy and paramiko 1.12 #1020

Closed
mkobos opened this Issue Nov 10, 2013 · 27 comments

Projects

None yet
@mkobos
mkobos commented Nov 10, 2013

[Maintainer edit: #1042 seems to be a dupe with its own big set of affected users.]

I'm using the most current Fabric version available in pypi: Fabric 1.8.0 with Paramiko 1.12 on Ubuntu 13.04. When I execute fab deploy using the following sample fabfile

#!/usr/bin/env python

from fabric.api import *

# Respect the proxy SSH host setting.
env.use_ssh_config = True 
env.hosts = ["my_host_name"]

def deploy():
    run("ls .")
    return

Fabric executes the ls commnad, prints Done. and hangs indefinitely. When I interrupt it with ^C, it prints:

File "/usr/local/bin/fab", line 9, in <module>
  load_entry_point('Fabric==1.8.0', 'console_scripts', 'fab')()
File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 760, in main
  disconnect_all()
File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 597, in disconnect_all
  connections[key].close()
File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 350, in close
  self._transport.close()
File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 618, in close
  self.stop_thread()
File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1395, in stop_thread
  self.join(10)
File "/usr/lib/python2.7/threading.py", line 960, in join
  self.__block.wait(delay)
File "/usr/lib/python2.7/threading.py", line 358, in wait
  _sleep(delay)

Note that it happens only when I use SSH proxy, i.e. this is the important part of the ~/.ssh/config file:

Host my_host_name
  User my_login
  ForwardAgent yes
  ProxyCommand ssh -W %h:%p my_proxy_host

A solution to this problem turned out to be downgrading paramiko library installed in my system to version 1.11.1 (downgrading to a little more current version 1.11.2 doesn't fix the problem).

@lazzurs
lazzurs commented Nov 29, 2013

Hello,

I can +1 this bug when using the proxycommand setting in my ssh config the latest version of paramiko hangs and does not close the connection but 1.11.1 does work fine.

However I would suggest this should be a bug for paramiko.

@cholick
cholick commented Dec 17, 2013

Another +1 for this issue - it's creating a problem for me as well.

@xanbuck
xanbuck commented Dec 17, 2013

+1

@dragon3
dragon3 commented Dec 18, 2013

+1 ๐Ÿ˜„

@DavidWittman

๐Ÿ‘

@samyscoub

๐Ÿ‘

@giuseppe7

+1

@xsb
xsb commented Feb 5, 2014

๐Ÿ‘

@mgedmin
mgedmin commented Feb 11, 2014

Here's a paramiko bug report that claims to fix this: paramiko/paramiko#252

@bitprophet
Member

While I am poking at this in Paramiko, can anybody else who +1'd this verify that Paramiko 1.11.1 works OK and 1.11.2 is subject to the problem? Thanks!

@bitprophet bitprophet referenced this issue in paramiko/paramiko Feb 12, 2014
Closed

fix hang if using proxycommand #252

@bitprophet
Member

Definitely seems to be a patch submitted as part of 1.11.2, however that patch itself is fixing a different deadlock problem so we can't simply revert it. I'd like more info from y'all if I can get it, please!

  • Is this happening every time you run Fabric or is it intermittent?
  • Does it still happen when you switch from using ProxyCommand+ssh to using the Fabric gateway option (http://docs.fabfile.org/en/1.8/usage/env.html#gateway)?
    • NOTE: Using this instead of ProxyCommand is recommended regardless, as it's cleaner & leverages a native sshd forwarding feature. ProxyCommand is more for when you need to proxy over other protocols/programs.
@mgedmin
mgedmin commented Feb 12, 2014

This happens every time.

This doesn't happen if I use fab --gateway.

(Unfortunately env.gateway is not a suitable workaround for me: fab -g fridge -H fridge task fails with a "paramiko.SSHException: SSH session not active". I'll file a bug.)

@bitprophet
Member

Thanks, and yea, was just asking to help pin down exactly what is affected by the threading problem.

@bitprophet
Member

Also now I am wondering how y'all are getting around #1046 o_O I hate new features, so much.

@mgedmin
mgedmin commented Feb 12, 2014

Also now I am wondering how y'all are getting around #1046

Easy: pip install Fabric==1.8.0 ;)

@bitprophet
Member

Well that's the thing, I went back to check and the typo present in #1046 appeared to be consistent back to ~Fabric 1.5 :) Moot now, anyways.

I can confirm this issue on my end pretty consistently, so that is good, will let me test potential fixes.

@bitprophet
Member

Dear all,

Please try using this branch (it's based off of the 1.11 line, FYI) and see how it addresses the problem for you. It may cause things overall to be slower than normal, though that may be my paranoia at work.

Makes things terminate cleanly for me in a base case setup (ProxyCommand of ssh -W to machine A, run 'ls /' on machine B).

See Paramiko #252 discussion for details on what I dug up, if you care, but that branch is the result.

@bitprophet
Member

Pushed more related fixes to same branch just now. cc @mkobos @lazzurs @mcholick @abuckles @dragon3 @DavidWittman @samyscoub @giuseppe7 @zroadhouse-wsm and anybody else I missed. GITHUB NOTIFICATION PARANOIA ENABLED

@dragon3
dragon3 commented Feb 14, 2014

@bitprophet Thank you for notification!
I tried "fix-proxycommand-infinite-loop-252" branch,
and I confirmed all connection terminated cleanly, my problem was fixed! ๐Ÿ˜ƒ

@beaugunderson

@bitprophet Verified that 1.11.1 works and that 1.12.0 does not.

@DavidWittman

@bitprophet fix-proxycommand-infinite-loop-252 addressed my issues as well.

Thanks!

@bitprophet
Member

Awesome, thanks guys. Merged the branch for Paramiko 1.11.x and 1.12.x, will release those lines today if I can, then you can just grab latest Paramiko in your desired line and it will include this change.

New Fabric will be out around the same time but besides the typo fix mentioned above, nothing had to change to support the Paramiko-level fixes.

@bitprophet bitprophet closed this Feb 14, 2014
@dragon3
dragon3 commented Feb 15, 2014

I tried Fabric 1.8.2 + Paramiko 1.12.2, they work fine!!
@bitprophet Thank you!

@mkobos
mkobos commented Feb 15, 2014

I checked my script with the most current versions of Fabric (1.8.2) and Paramiko (1.12.2) and it works fine now. Thanks!

@lepture
lepture commented Sep 29, 2014

It still hangs!

Fabric==1.10.0
paramiko==1.15.1
@bitprophet
Member

@lepture Can you open a new ticket with some details please? E.g. are you using the example snippets up top here; if not, what's your config & fabfile look like; etc. Also, please try to use multiple Paramiko versions (something below 1.15 for sure, as that made moderate changes) and see how that affects things.

(Again - in a new ticket, please & thanks!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment