Implement tunnelling #38

bitprophet opened this Issue Aug 19, 2011 · 42 comments


None yet


It should be possible to tunnel all the commands through a single entry point in the network.

Originally submitted by Anonymous () on 2009-07-27 at 05:22pm EDT



  • Related to #275: Consider forking Paramiko
  • Duplicated by #344: Tunnelling SSH over HTTP Proxies
  • Related to #78: Add Tunneling Context to Fab
  • Related to #72: SSH key forwarding
@bitprophet bitprophet was assigned Aug 19, 2011

Anonymous () posted:

The incredible monkeypatch in the attached file allows some kind of tunnelling. First of all, there would be no need to change the network.connect() function if authentication happened without user interaction (no password prompt). I think I cannot use the same connect while forwarding because I need to create a direct-tcpip connection and recreate it every time I have an error while trying to connect, I suspect paramiko doesn't support some reset packet from SSH, it sends a warning about not supporting type 3.

The second monkeypatch done is to SSHClient to make it accept an already created socket rather than creating one of its own. The system is actually passing an already setup direct-tcpip Channel as a socket, and it can do that because Channel and socket apparently share the same interface so the Transport doesn't freak out.

Then I monkeypatch the ConnectionCache to make it look for a gateway option in the env dictionary, since fab doesn't support it from command line it has to be added in a config file, the format is the same for hosts username@host:port. When the gateway option is present it would then try to connect to the gateway and then to forward connect from it to the clients.

And the last monkeypatch is related to #36 and replaces the connections dictionary in the files where it matters.

It definitely needs more testing and I'm not even sure how to test all this stuff, specifically the actual forwarding... There are so many layers involved that it's a big project to get any of this in the corresponding project on its own...

Hope it helps, also it might contain bugs (mainly not imported names), it's just a reference right now.

on 2009-07-28 at 09:17pm EDT


Valentino Volonghi (dialtone) posted:

Side note... The patch is from me :)

on 2009-07-28 at 09:21pm EDT


Jeff Forcier (bitprophet) posted:

See -- apparently this is a drop-in patch to paramiko that allows ProxyCommand to work (so I assume it reads ssh_config too, haven't tested it myself yet.)

Kudos to IRC user Favoretti for the tip.

IRC log pasted by goosemo

on 2011-06-06 at 05:10pm EDT


Jeff Forcier (bitprophet) posted:

A user (not sure if it's the same IRC user?) also posted this ML thread with patches, probably the same approach -- have not looked yet.

on 2011-06-08 at 02:05am EDT


Jeff Forcier (bitprophet) posted:

Read up on ProxyCommand real quick since I wasn't entirely sure if it was the same thing as using gateways; it sounds like it is one of the more common ways of gatewaying/proxying through one SSH server to another. See e.g. this blog post about using e.g. netcat on the first hop (again, this technique is used in a number of places.)

on 2011-07-03 at 06:03pm EDT


Erwin Bolwidt (ebolwidt) posted:

I tried the monkeypatch and it still works. (The quick way: add to fabric folder, add "import monkeypatch" to, and add an option to so you can specify the SSH gateway that you want on the command line:

  • make_option('-G', '--gateway',
  •    default=None,
  •    help="SSH gateway"
  • ),


I don't get any errors about not supporting type 3, so I guess paramiko resolved that since this issue was opened.
I do get errors during shutdown; my guess is that the connection to the gateway is closed before the connection through the gateway to the target host is closed, which is the wrong way around and causes the thread that runs the connection to the target host to complain.

If that can be resolved, then the monkeypatch could be integrated with the code easily (it only patches fabric, not paramiko, and is quite clean)

on 2011-07-14 at 02:33am EDT


Erwin Bolwidt (ebolwidt) posted:

To make one point: the monkeypatch approach is a lot like the approach taken by capistrano. It allows the SSH gateway host to be specified in any way you like.
The other approach using paraproxy is less flexible in my opinion, as it requires you to specify the SSH gateway host by adding ProxyCommand directories to your SSH configuration files. The monkeypatch works differently and doesn't require that.

on 2011-07-14 at 02:36am EDT


Erwin Bolwidt (ebolwidt) posted:

I transformed the monkeypatch into clean code modifications. See attached git diff file and the master branch of my github fork (tag "monkeypatch_normalized_sshgateway")

On Unix, it works perfectly for me. Example:

fabric --gateway --host who_am_i

Running run("who am i") has the advantage of printing the hostname that you are logging in from, so you can test whether the gateway server is used correctly. Any two ssh hosts to which you have access suffices; the target host doesn't have to be unreachable from the source host. Of course you only need the gateway if the target host is not directly reachable, such as in a DMZ, but for testing there is no need for this.

On my Windows installation with Python 2.7.1 I sometimes have a problem with hanging processes if I use the gateway feature.
Symptoms: the disconnect from the target host doesn't complete, and if I ctrl-C the process from the command-line, it keeps running instead (using 13% CPU consistently) and keeps the network connection to the gateway open (but not to the target host from the gateway).
By running StraceNT on the process I can see that the process is busy with multi-threaded mutexes but no useful system calls being performed.
I'm inclined to think that this has to do with a broken installation on my end; my Windows setup is a bit hackety.

I would appreciate if someone else could try this feature on Windows to see if it works for them.

on 2011-07-18 at 03:54am EDT

parente commented Oct 23, 2011

I tried Erwin Bolwidt's patch on my OS X 10.6.8 box. It worked but with the same hang he reported on his Windows machine.


ML patch came from a colleague of mine. Favoretti would be me then. :)

If I can add something in favor of paraproxy is that using paraproxy doesn't really make it less flexible.
It will just honor whatever SSH proxy options are present in the config of the user running fabric at that particular moment.

If there are no proxies defined in the config - well, we don't use anything.Why not to combine ability to specify gateway and ability of using netcat-based multi-level proxying?

mdz commented Mar 11, 2012

FWIW, I'd really prefer the paraproxy/ProxyCommand approach, because we use this technique for all ssh sessions, not just fabric. Having to store this config in two places (fabric and ssh_config) is more complex and error-prone.

Until this issue is resolved, is there a way to use paraproxy with fabric without modifying fabric?

mdz commented Mar 11, 2012

Ah, I see paramiko is no longer used, so paraproxy doesn't apply


Used SSH lib is a fork of paramiko. Now IMO hacking paraproxy in would be just perfect. Just pull it in as a whole. :)

teeler commented Mar 22, 2012

+1 to what @mdz suggests, but i'd also like to see a ProxyCommand-type thing as a subclass-able ContextManager, allowing for things like:

def myfun():
    with CorkscrewProxy("proxyhost"):
        run(..) # runs on host via proxyhost

Or as an arg to the proxy class itself:

with SshProxy("proxyhost", "host"):
   run(...) # runs on host

@teeler that's probably covered under #78 FWIW

teeler commented Mar 26, 2012

Ah! Thanks @bitprophet! It wasn't terribly clear to me at first, but now i see it. Thanks!

svenXY commented Apr 16, 2012

+1 for @favoretti

The paraproxy approach would still be of much use for me as I already maintain an .ssh/config with a lot of ProxyCommands.

If ssh is a fork of paramiko, it should hot be so hard to make it paraproxy-aware and it would suddenly agwin be usable for me (having an SSH gateway between office and production networks)


dsc commented May 9, 2012


Just to make it explicit to anyone who might not have read carefully enough to figure out how to use the monkeypatch now, while waiting for this to make its way into a release:

  1. Download the patch. My fabric setup uses a folder, so I ran: curl > fabfile/
  2. Install paramiko: pip install paramiko -- You'll probably want to add it to your project dependencies if you're in Python-land. This project wasn't, so I had my fabfile try-except for the deps up top and give the install command.
  3. In your main fabfile add import monkeypatch_sshproxy and set the gateway key: env.gateway = '' -- If you usually set ProxyCommand in ~/.ssh/config, it's the host in that command. The patch does not add ssh-config support, so you do need to set env.gateway.

And that's all. Remote commands will now proxy correctly. If env.gateway isn't set, everything appears to proceed as usual.

Awesome work!


I really would like this. I wasn't able to get the monkey patch to work - it kept complaining about a missing password - and I need different proxies for different hosts anyway.

However, the following replacement for run() seems to do the trick:

  def proxy_run(cmd):
      host, port = env.host_string, 22
      if ':' in host:
          host, port = env.host_string.split(':')

      local('ssh -p %s -A %s "%s" 2>&1 | sed "s/^/[%s] /"' % (port, host, cmd, env.host_string))

I +1 this with my entire being :)

At work, in order to access the servers from the customers, we have to do a hop through an intermediate network. Of course we do this with tunnels, but having proxycommand support built into fabric would be a HUGE help. We have ways to log in directly to the customer's network but that is not completely reliable.

kkumler commented Jul 26, 2012

I use ProxyCommand in ssh_config quite a bit, so honoring that would be quite good. My bastion host can be set to forward the ssh agent as well (ForwardAgent yes), which may also be useful for the target host (if it has to pull something via ssh).

For reference, some of the existing ssh config is like such, with one host entry for the target system and then one host entry that applies the ProxyCommand (note ssh -W) when needed:

Host bastionhost
ForwardAgent yes

Host bastion-* 10.5.5.* !internal-
ProxyCommand ssh -W %h:%p bastionhost

Host bastion-theserver
ForwardAgent yes

clarete commented Aug 1, 2012

Here's a patch that implements the ProxyCommand in fabric's ssh library: bitprophet/ssh#34

clarete commented Aug 2, 2012

Here's the change in fabric to take use the ssh change: #698


@clarete I just edited your comment to reflect this, but in future please use "#NNN" to refer to other tickets, not a hyperlink. See the 'Github Flavored Markdown' link above text fields for details. Thanks!

clarete commented Aug 7, 2012

Hi @bitprophet! Sure I'll follow your advice in my next comments! Sorry for that :)

MiLk commented Sep 25, 2012

I need use ProxyCommand to deploy apps over a gateway.
The monkeypatch works fine with the run() command.
But with rsync_project or put it doesn't work.
How can i solve the problem ?


@MiLk: rsync_project just runs rsync locally and feeds it some of Fabric's connection options. Depending on how the gateway feature is implemented, it might not be possible for rsync to ride on top of the gateway connection we set up. Will have to see.


Paramiko clearinghouse ticket for the socket-add feature (which is the base required to implement direct-tcpip tunneling in Fabric, and is related to but distinct from ProxyCommand's requirements) is Paramiko #77.

The ProxyCommand ticket over there is Paramiko #97.


@clarete's link a few comments up will be my starting point for ProxyCommand stuff (he also provides the equivalent changes at the Paramiko level, see above).

Starting point for the direct-tcpip approach is this compare view to @ebolwidt's fork.

I'm about 75% done with a blog post outlining & comparing both solutions.


Blog post 1st draft is up: Gateway Solutions in Paramiko & Fabric. Will now actually do code work and either write a followup post or edit that one, if the plan changes.

clarete commented Nov 5, 2012

Hi @bitprophet! Thank you for the very nice article! It explains the problem and the two possible solutions in a very clear and elegant way! :)

Just two things I'd say about it are the following:

  • The good thing about ProxyCommand is that you can still use gateways even when you are not using fabric. I used to maintain some programs that access paramiko directly.
  • You don't actually need to have the ssh client installed to use ProxyCommand. For example, I was using a custom program to forward the connection to test the patch I sent. But definitely it's the way that most people use this feature and it's the way the servers of the company I'm working for use it. So, in the end, the post is more than right!

I have the direct-tcpip approach working at the Fabric level now (after merging the core 'overridden socket' feature in Paramiko). Need docs, then will commit that + move on to ProxyCommand which is a little more involved.

Mostly unrelated: I re-checked #78 and offhand I don't think it is actually that strongly related, in the "tunneling non SSH traffic" sense. I've probably directed some folks there thinking it was about having gateway behavior itself be context-manager'd; for that, with this new feature, they can just use settings like so:

env.host_string = "realtarget"
run("this will connect directly to realtarget")
with settings(gateway='mygateway'):
    run("this will connect to realtarget via mygateway")
run("this goes back to direct connections")

EDIT: Though to make that fully workable I need to boost HostConnectionCache so it tracks both host string + the value of env.gateway at the time of connection. That way a direct connect to host foo will be a different cache entry from a gatewayed-via-bar connection to foo.

EDIT 2: I'm punting on that, it requires actually updating HostConnectionCache's key structure (not just the dict-like methods themselves), which is too fragile to reliably update without introducing new bugs, especially given how infrequent the above use case ought to be. Will definitely work better with real host objects in Fab 2.x.


Totally missed @clarete's reply above, sorry! I did mention in the post that one of ProxyCommand's pluses is the ability to use the ssh_config declarations outside of Fabric. In the "Comparison" section, I wrote:

direct-tcpip can only be configured/used by Paramiko & Fabric, whereas ProxyCommand directives stored in ssh_config files can be used by both Python and the regular ssh client.

clarete commented Nov 5, 2012

Thank you for your answer @bitprophet and sorry for missing that part! :)


Somewhere in this thicket of tickets (heh) I recall saying "when both ProxyCommand and env.gateway are set, ProxyCommand wins". However, after thinking about it I've come to the opposite conclusion:

  • If one has an ssh_config file they intend to use with Fabric, chances are they're using it outside Fabric too, and have configured it for that use case primarily. Using it within Fabric may be largely incidental.
  • Conversely, if env.gateway is set, chances are the user set it on purpose and expects it to be operational.
  • Finally, regardless of the reason a user ended up with both set, it is significantly easier for them to set env.gateway back to None (globally or locally) than for them to modify their config file or defer loading it (and there is no way to only load parts of a config file either -- it's on or off.)

So I'll set things up so env.gateway is checked first, and ProxyCommand only tested for & applied if the gateway did not fire.


Had a minor freakout because @clarete was a bad, bad contributor and put Python 2.6+ specific code in his pull request. Thankfully Popen.terminate is super easily replicated in Python 2.5 (it's basically os.kill(, signal.SIGTERM)) but I was worried I'd have to do something ugly like vendorize all of subprocess :(

(It's ok @clarete, I forgive you, I need better contribution docs that absolve me of any blame when people forget the minimum Python version supported ;))

ProxyCommand support appears to be working now. Gonna bang on it a little more but I think we're shipping this puppy tomorrow, sometime after I vote.

@bitprophet bitprophet added a commit that referenced this issue Nov 6, 2012
@bitprophet bitprophet Changelog re #38 5e9cd46

N.B. What's not included yet and will need adding in 1.6 or later, if possible:

  • Non-ssh_config-driven ProxyCommand like behavior.
    • Shouldn't be hard; we could add another Fab-level setting similar to env.gateway (or override that itself, e.g. "if it's not a host string, treat it like a proxy command") which then overrides (or not, depending on what feels sanest) one or both of the existing settings. Would simply replace the value currently sourced from ssh_config.
  • Some way of tunneling external apps, i.e. what #78 is really about. (E.g. this includes rsync_project and any other "shells out but then interpolates Fab connection settings" hacks)
    • Probably not easy, but doable somehow. If I'm wrong and it is easy, even better.
  • Context-manager-aware gatewaying; right now a given target host is either gatewayed or not, at the time it's initially connected to.
    • As above this requires more complexity in the connection caching which should wait until Fab 2.x

Closing this because it's, like, done now and merged into master. Above items to be spun off into real tickets later on.

@bitprophet bitprophet closed this Nov 6, 2012
clarete commented Nov 6, 2012

Hey @bitprophet! Sorry for the 2.6 specific code! In the first place I'm not actually a big fan of subprocess! (you can see why I don't like it here. I just couldn't use it's communicate() api cause it closes stdin even if you explicitly pass closefd=False)!

I was just too focused on the task that I forgot this very important detail!! Thanks for the heads up!


@clarete -- no worries, I was just poking fun at you :) Thanks again for all the work on this!


@bitprophet: Hi.. I am trying to use fabric module to solve this tunneling:
Node1-- ssh--Node2--ssh--Node3--(use namespace of node3 to ssh) --Node4 --run sudo cmds on Node4
Can I achieve the above(my blocker is the use of namespace to ssh from node3 to node4) by using the utility
Any guidance much appreciated


@puppet-py Probably a better question for the mailing list, not a comment on a defunct ticket :) I personally have not done any multi-hop tunneling like that, so not sure how well it'll work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment