Add Tunneling Context to Fab

[bitprophet]

Description

This would be very helpful for me because it would make it very easy
to for example connect to a remote MySQL server using my command line
MySQL client.

We could use a "with" context statement, something like:

with tunnel(local=3307, remote=3306):
    local('mysql --port=3007 --host=localhost' mydb < db/dbdump.sql')

This would eliminate the need to upload mysql dump file to the server
just to be able to run import.

Another application could be for administering Cherokee web servers.

Cherokee web admin by default is only accessible from the server that
it's running on. So you want to access the admin you have to tunnel
into the server and access the admin interface using a local port.
This could also be simplified with this functionality.

with tunnel(local=9090, remote=9090):
   sudo('cherokee-admin')
   prompt('Stop cherokee admin?')

That last line would keep the tunnel open until it's closed by providing input.

---

Originally submitted by Taras Mankovski (tarasm) on 2009-11-02 at 09:30am EST

Relations

• Related to Implement tunnelling #38: Implement tunnelling

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

(modified description so code blocks were indented :))

---

on 2009-11-02 at 09:35am EST

[munhitsu]

It would be awesome if tunnel would also support the other way around. Meaning listening on remote and forwarding it into localhost/other locally available host:port.

[bitprophet]

@munhitsu I'm not sure how that's a use case for Fabric specifically, though. Can you elaborate?

[munhitsu]

Imagine a DMZ setup which does not have an outbound http/proxy access.
Whole deployment goes through fabric.

In such case we could use fabric to tunnel local proxy through ssh, to be temporarily accessible for the host being just provisioned in DMZ. Right now I'm opening a separate ssh tunnel on second console, so that fabric works within "context" of this tunnel.

example usage:

with rev_tunnel(local=8080, remote=8080):
    sudo("http_proxy='http://localhost:8080' apt-get install -y puppet")

[bitprophet]

OK, so it is a fairly standard reverse tunnel setup after all I guess, and your desire re: Fabric is for it to handle doing the tunnel vs you having to run eg local(ssh -R ...).

I went back and forth on whether this is truly worth going into core, but it really would make a decent amount of sense for it to be supported in Fabric proper; the other solutions are hacky (eg some thread or subprocess running ssh -- how to do this well, make sure it shuts down when Fab does, etc) and I do see the validity of the use case (sharing local resources with the remote end during execution.)

Main roadblock is that I am not sure the SSH lib supports this yet; we'll have to figure that out and somebody would have to implement it if it doesn't. (I think that would be a good addition to said library, however, vs solidifying aforementioned ssh -R workaround.)

EDIT: #38 discusses implementation and patching and such. Might be best to actually close this and simply note there that when implemented, it should be possible to trigger with a context manager if possible.

[munhitsu]

I fully agree that having it implemented so that we enter, leave contexts, or even worse multiple levels of contexts is tricky.

Regarding EDIT: Anything that keeps us closer to having this functionality is a good idea.

[bitprophet]

For now I'll leave this open, just to keep things granular. Assigned it to 1.4 so I don't forget, though. Chances are I will try to knock this out right as I am doing #38. Will update here when it's in master, thanks.

[bitprophet]

This is actually not as related to #38 as I thought, since its changes are solely about gatewaying the SSH traffic itself, not tunneling additional ports through the SSH connection. That will require a different (or at least, additional) solution. Punting for now, sorry :) (Meaning: still open, just not going to arrive in 1.5.)

[haridsv]

I recently needed this functionality to sync to a remote rsync server whose port is not directly reachable and found that the paramiko's forward.py demo code has sample code that I could use, so I came up with a solution that worked well for me and I submitted it as a patch for forward.py here: paramiko/paramiko#504

We could add ForwardServer from that patch and have a local_tunnel() that simply returns an instance of ForwardServer. As per recommendation from @bitprophet on the pull request, I will work on a patch for Fabric.

[haridsv]

I actually didn't realize, but there is already a patch for local_tunnel(), though I am not fully sure about its status. What should I do?

[bitprophet]

@haridsv If you're able, testing it out & mentioning that you successfully used it, on that patch (#939), would help, otherwise just have patience :) thanks!

[naphthalene]

Is there any plan to resolve this issue? There are two outstanding pull requests that together resolve this, but they haven't been reviewed? Any way we could speed this along?
#939, #1218

I've been using the code in #1218 with no problems.

[bitprophet]

#939 is still in a release bucket, I just punted it from the upcoming 1.11 because I needed to cut some stuff, but it'll get priority for the next feature cycle. (And it looks like #1218 supersedes #939 so I'll probably end up merging that & crediting #939 in the changelog.)