Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fabric hangs when using SSH proxy and paramiko 1.12 #1020

Closed
mkobos opened this issue Nov 10, 2013 · 27 comments
Closed

Fabric hangs when using SSH proxy and paramiko 1.12 #1020

mkobos opened this issue Nov 10, 2013 · 27 comments

Comments

@mkobos
Copy link

@mkobos mkobos commented Nov 10, 2013

[Maintainer edit: #1042 seems to be a dupe with its own big set of affected users.]

I'm using the most current Fabric version available in pypi: Fabric 1.8.0 with Paramiko 1.12 on Ubuntu 13.04. When I execute fab deploy using the following sample fabfile

#!/usr/bin/env python

from fabric.api import *

# Respect the proxy SSH host setting.
env.use_ssh_config = True 
env.hosts = ["my_host_name"]

def deploy():
    run("ls .")
    return

Fabric executes the ls commnad, prints Done. and hangs indefinitely. When I interrupt it with ^C, it prints:

File "/usr/local/bin/fab", line 9, in <module>
  load_entry_point('Fabric==1.8.0', 'console_scripts', 'fab')()
File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 760, in main
  disconnect_all()
File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 597, in disconnect_all
  connections[key].close()
File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 350, in close
  self._transport.close()
File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 618, in close
  self.stop_thread()
File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1395, in stop_thread
  self.join(10)
File "/usr/lib/python2.7/threading.py", line 960, in join
  self.__block.wait(delay)
File "/usr/lib/python2.7/threading.py", line 358, in wait
  _sleep(delay)

Note that it happens only when I use SSH proxy, i.e. this is the important part of the ~/.ssh/config file:

Host my_host_name
  User my_login
  ForwardAgent yes
  ProxyCommand ssh -W %h:%p my_proxy_host

A solution to this problem turned out to be downgrading paramiko library installed in my system to version 1.11.1 (downgrading to a little more current version 1.11.2 doesn't fix the problem).

@lazzurs
Copy link

@lazzurs lazzurs commented Nov 29, 2013

Hello,

I can +1 this bug when using the proxycommand setting in my ssh config the latest version of paramiko hangs and does not close the connection but 1.11.1 does work fine.

However I would suggest this should be a bug for paramiko.

@cholick
Copy link

@cholick cholick commented Dec 17, 2013

Another +1 for this issue - it's creating a problem for me as well.

@xanbuck
Copy link

@xanbuck xanbuck commented Dec 17, 2013

+1

@dragon3
Copy link

@dragon3 dragon3 commented Dec 18, 2013

+1 😄

@DavidWittman
Copy link

@DavidWittman DavidWittman commented Jan 21, 2014

👍

4 similar comments
@samyscoub
Copy link

@samyscoub samyscoub commented Jan 21, 2014

👍

@giuseppe7
Copy link

@giuseppe7 giuseppe7 commented Jan 23, 2014

+1

@zroadhouse-rmn
Copy link

@zroadhouse-rmn zroadhouse-rmn commented Jan 24, 2014

+1

@xsb
Copy link

@xsb xsb commented Feb 5, 2014

👍

@mgedmin
Copy link

@mgedmin mgedmin commented Feb 11, 2014

Here's a paramiko bug report that claims to fix this: paramiko/paramiko#252

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 12, 2014

While I am poking at this in Paramiko, can anybody else who +1'd this verify that Paramiko 1.11.1 works OK and 1.11.2 is subject to the problem? Thanks!

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 12, 2014

Definitely seems to be a patch submitted as part of 1.11.2, however that patch itself is fixing a different deadlock problem so we can't simply revert it. I'd like more info from y'all if I can get it, please!

  • Is this happening every time you run Fabric or is it intermittent?
  • Does it still happen when you switch from using ProxyCommand+ssh to using the Fabric gateway option (http://docs.fabfile.org/en/1.8/usage/env.html#gateway)?
    • NOTE: Using this instead of ProxyCommand is recommended regardless, as it's cleaner & leverages a native sshd forwarding feature. ProxyCommand is more for when you need to proxy over other protocols/programs.
@mgedmin
Copy link

@mgedmin mgedmin commented Feb 12, 2014

This happens every time.

This doesn't happen if I use fab --gateway.

(Unfortunately env.gateway is not a suitable workaround for me: fab -g fridge -H fridge task fails with a "paramiko.SSHException: SSH session not active". I'll file a bug.)

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 12, 2014

Thanks, and yea, was just asking to help pin down exactly what is affected by the threading problem.

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 12, 2014

Also now I am wondering how y'all are getting around #1046 o_O I hate new features, so much.

@mgedmin
Copy link

@mgedmin mgedmin commented Feb 12, 2014

Also now I am wondering how y'all are getting around #1046

Easy: pip install Fabric==1.8.0 ;)

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 12, 2014

Well that's the thing, I went back to check and the typo present in #1046 appeared to be consistent back to ~Fabric 1.5 :) Moot now, anyways.

I can confirm this issue on my end pretty consistently, so that is good, will let me test potential fixes.

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 13, 2014

Dear all,

Please try using this branch (it's based off of the 1.11 line, FYI) and see how it addresses the problem for you. It may cause things overall to be slower than normal, though that may be my paranoia at work.

Makes things terminate cleanly for me in a base case setup (ProxyCommand of ssh -W to machine A, run 'ls /' on machine B).

See Paramiko #252 discussion for details on what I dug up, if you care, but that branch is the result.

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 13, 2014

Pushed more related fixes to same branch just now. cc @mkobos @lazzurs @mcholick @abuckles @dragon3 @DavidWittman @samyscoub @giuseppe7 @zroadhouse-wsm and anybody else I missed. GITHUB NOTIFICATION PARANOIA ENABLED

@dragon3
Copy link

@dragon3 dragon3 commented Feb 14, 2014

@bitprophet Thank you for notification!
I tried "fix-proxycommand-infinite-loop-252" branch,
and I confirmed all connection terminated cleanly, my problem was fixed! 😃

@beaugunderson
Copy link

@beaugunderson beaugunderson commented Feb 14, 2014

@bitprophet Verified that 1.11.1 works and that 1.12.0 does not.

@DavidWittman
Copy link

@DavidWittman DavidWittman commented Feb 14, 2014

@bitprophet fix-proxycommand-infinite-loop-252 addressed my issues as well.

Thanks!

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Feb 14, 2014

Awesome, thanks guys. Merged the branch for Paramiko 1.11.x and 1.12.x, will release those lines today if I can, then you can just grab latest Paramiko in your desired line and it will include this change.

New Fabric will be out around the same time but besides the typo fix mentioned above, nothing had to change to support the Paramiko-level fixes.

@bitprophet bitprophet closed this Feb 14, 2014
@dragon3
Copy link

@dragon3 dragon3 commented Feb 15, 2014

I tried Fabric 1.8.2 + Paramiko 1.12.2, they work fine!!
@bitprophet Thank you!

@mkobos
Copy link
Author

@mkobos mkobos commented Feb 15, 2014

I checked my script with the most current versions of Fabric (1.8.2) and Paramiko (1.12.2) and it works fine now. Thanks!

@lepture
Copy link

@lepture lepture commented Sep 29, 2014

It still hangs!

Fabric==1.10.0
paramiko==1.15.1
@bitprophet
Copy link
Member

@bitprophet bitprophet commented Sep 29, 2014

@lepture Can you open a new ticket with some details please? E.g. are you using the example snippets up top here; if not, what's your config & fabfile look like; etc. Also, please try to use multiple Paramiko versions (something below 1.15 for sure, as that made moderate changes) and see how that affects things.

(Again - in a new ticket, please & thanks!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet