New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fabric hangs when using SSH proxy and paramiko 1.12 #1020

Closed
mkobos opened this Issue Nov 10, 2013 · 27 comments

Comments

Projects
None yet
@mkobos

mkobos commented Nov 10, 2013

[Maintainer edit: #1042 seems to be a dupe with its own big set of affected users.]

I'm using the most current Fabric version available in pypi: Fabric 1.8.0 with Paramiko 1.12 on Ubuntu 13.04. When I execute fab deploy using the following sample fabfile

#!/usr/bin/env python

from fabric.api import *

# Respect the proxy SSH host setting.
env.use_ssh_config = True 
env.hosts = ["my_host_name"]

def deploy():
    run("ls .")
    return

Fabric executes the ls commnad, prints Done. and hangs indefinitely. When I interrupt it with ^C, it prints:

File "/usr/local/bin/fab", line 9, in <module>
  load_entry_point('Fabric==1.8.0', 'console_scripts', 'fab')()
File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 760, in main
  disconnect_all()
File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 597, in disconnect_all
  connections[key].close()
File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 350, in close
  self._transport.close()
File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 618, in close
  self.stop_thread()
File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1395, in stop_thread
  self.join(10)
File "/usr/lib/python2.7/threading.py", line 960, in join
  self.__block.wait(delay)
File "/usr/lib/python2.7/threading.py", line 358, in wait
  _sleep(delay)

Note that it happens only when I use SSH proxy, i.e. this is the important part of the ~/.ssh/config file:

Host my_host_name
  User my_login
  ForwardAgent yes
  ProxyCommand ssh -W %h:%p my_proxy_host

A solution to this problem turned out to be downgrading paramiko library installed in my system to version 1.11.1 (downgrading to a little more current version 1.11.2 doesn't fix the problem).

@lazzurs

This comment has been minimized.

lazzurs commented Nov 29, 2013

Hello,

I can +1 this bug when using the proxycommand setting in my ssh config the latest version of paramiko hangs and does not close the connection but 1.11.1 does work fine.

However I would suggest this should be a bug for paramiko.

@cholick

This comment has been minimized.

cholick commented Dec 17, 2013

Another +1 for this issue - it's creating a problem for me as well.

@xanbuck

This comment has been minimized.

xanbuck commented Dec 17, 2013

+1

@dragon3

This comment has been minimized.

dragon3 commented Dec 18, 2013

+1 😄

@DavidWittman

This comment has been minimized.

DavidWittman commented Jan 21, 2014

👍

4 similar comments
@samyscoub

This comment has been minimized.

samyscoub commented Jan 21, 2014

👍

@giuseppe7

This comment has been minimized.

giuseppe7 commented Jan 23, 2014

+1

@zroadhouse-rmn

This comment has been minimized.

zroadhouse-rmn commented Jan 24, 2014

+1

@xsb

This comment has been minimized.

xsb commented Feb 5, 2014

👍

@mgedmin

This comment has been minimized.

mgedmin commented Feb 11, 2014

Here's a paramiko bug report that claims to fix this: paramiko/paramiko#252

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 12, 2014

While I am poking at this in Paramiko, can anybody else who +1'd this verify that Paramiko 1.11.1 works OK and 1.11.2 is subject to the problem? Thanks!

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 12, 2014

Definitely seems to be a patch submitted as part of 1.11.2, however that patch itself is fixing a different deadlock problem so we can't simply revert it. I'd like more info from y'all if I can get it, please!

  • Is this happening every time you run Fabric or is it intermittent?
  • Does it still happen when you switch from using ProxyCommand+ssh to using the Fabric gateway option (http://docs.fabfile.org/en/1.8/usage/env.html#gateway)?
    • NOTE: Using this instead of ProxyCommand is recommended regardless, as it's cleaner & leverages a native sshd forwarding feature. ProxyCommand is more for when you need to proxy over other protocols/programs.
@mgedmin

This comment has been minimized.

mgedmin commented Feb 12, 2014

This happens every time.

This doesn't happen if I use fab --gateway.

(Unfortunately env.gateway is not a suitable workaround for me: fab -g fridge -H fridge task fails with a "paramiko.SSHException: SSH session not active". I'll file a bug.)

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 12, 2014

Thanks, and yea, was just asking to help pin down exactly what is affected by the threading problem.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 12, 2014

Also now I am wondering how y'all are getting around #1046 o_O I hate new features, so much.

@mgedmin

This comment has been minimized.

mgedmin commented Feb 12, 2014

Also now I am wondering how y'all are getting around #1046

Easy: pip install Fabric==1.8.0 ;)

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 12, 2014

Well that's the thing, I went back to check and the typo present in #1046 appeared to be consistent back to ~Fabric 1.5 :) Moot now, anyways.

I can confirm this issue on my end pretty consistently, so that is good, will let me test potential fixes.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 13, 2014

Dear all,

Please try using this branch (it's based off of the 1.11 line, FYI) and see how it addresses the problem for you. It may cause things overall to be slower than normal, though that may be my paranoia at work.

Makes things terminate cleanly for me in a base case setup (ProxyCommand of ssh -W to machine A, run 'ls /' on machine B).

See Paramiko #252 discussion for details on what I dug up, if you care, but that branch is the result.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 13, 2014

Pushed more related fixes to same branch just now. cc @mkobos @lazzurs @mcholick @abuckles @dragon3 @DavidWittman @samyscoub @giuseppe7 @zroadhouse-wsm and anybody else I missed. GITHUB NOTIFICATION PARANOIA ENABLED

@dragon3

This comment has been minimized.

dragon3 commented Feb 14, 2014

@bitprophet Thank you for notification!
I tried "fix-proxycommand-infinite-loop-252" branch,
and I confirmed all connection terminated cleanly, my problem was fixed! 😃

@beaugunderson

This comment has been minimized.

beaugunderson commented Feb 14, 2014

@bitprophet Verified that 1.11.1 works and that 1.12.0 does not.

@DavidWittman

This comment has been minimized.

DavidWittman commented Feb 14, 2014

@bitprophet fix-proxycommand-infinite-loop-252 addressed my issues as well.

Thanks!

@bitprophet

This comment has been minimized.

Member

bitprophet commented Feb 14, 2014

Awesome, thanks guys. Merged the branch for Paramiko 1.11.x and 1.12.x, will release those lines today if I can, then you can just grab latest Paramiko in your desired line and it will include this change.

New Fabric will be out around the same time but besides the typo fix mentioned above, nothing had to change to support the Paramiko-level fixes.

@bitprophet bitprophet closed this Feb 14, 2014

@dragon3

This comment has been minimized.

dragon3 commented Feb 15, 2014

I tried Fabric 1.8.2 + Paramiko 1.12.2, they work fine!!
@bitprophet Thank you!

@mkobos

This comment has been minimized.

mkobos commented Feb 15, 2014

I checked my script with the most current versions of Fabric (1.8.2) and Paramiko (1.12.2) and it works fine now. Thanks!

@lepture

This comment has been minimized.

lepture commented Sep 29, 2014

It still hangs!

Fabric==1.10.0
paramiko==1.15.1
@bitprophet

This comment has been minimized.

Member

bitprophet commented Sep 29, 2014

@lepture Can you open a new ticket with some details please? E.g. are you using the example snippets up top here; if not, what's your config & fabfile look like; etc. Also, please try to use multiple Paramiko versions (something below 1.15 for sure, as that made moderate changes) and see how that affects things.

(Again - in a new ticket, please & thanks!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment