New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Tunneling Context to Fab #78

Open
bitprophet opened this Issue Aug 19, 2011 · 13 comments

Comments

Projects
None yet
4 participants
@bitprophet
Member

bitprophet commented Aug 19, 2011

Description

This would be very helpful for me because it would make it very easy
to for example connect to a remote MySQL server using my command line
MySQL client.

We could use a "with" context statement, something like:

with tunnel(local=3307, remote=3306):
    local('mysql --port=3007 --host=localhost' mydb < db/dbdump.sql')

This would eliminate the need to upload mysql dump file to the server
just to be able to run import.

Another application could be for administering Cherokee web servers.

Cherokee web admin by default is only accessible from the server that
it's running on. So you want to access the admin you have to tunnel
into the server and access the admin interface using a local port.
This could also be simplified with this functionality.

with tunnel(local=9090, remote=9090):
   sudo('cherokee-admin')
   prompt('Stop cherokee admin?')

That last line would keep the tunnel open until it's closed by providing input.


Originally submitted by Taras Mankovski (tarasm) on 2009-11-02 at 09:30am EST

Relations

  • Related to #38: Implement tunnelling

@ghost ghost assigned bitprophet Aug 19, 2011

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Jeff Forcier (bitprophet) posted:


(modified description so code blocks were indented :))


on 2009-11-02 at 09:35am EST

Member

bitprophet commented Aug 19, 2011

Jeff Forcier (bitprophet) posted:


(modified description so code blocks were indented :))


on 2009-11-02 at 09:35am EST

@munhitsu

This comment has been minimized.

Show comment
Hide comment
@munhitsu

munhitsu Jan 9, 2012

It would be awesome if tunnel would also support the other way around. Meaning listening on remote and forwarding it into localhost/other locally available host:port.

munhitsu commented Jan 9, 2012

It would be awesome if tunnel would also support the other way around. Meaning listening on remote and forwarding it into localhost/other locally available host:port.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Jan 11, 2012

Member

@munhitsu I'm not sure how that's a use case for Fabric specifically, though. Can you elaborate?

Member

bitprophet commented Jan 11, 2012

@munhitsu I'm not sure how that's a use case for Fabric specifically, though. Can you elaborate?

@munhitsu

This comment has been minimized.

Show comment
Hide comment
@munhitsu

munhitsu Jan 11, 2012

Imagine a DMZ setup which does not have an outbound http/proxy access.
Whole deployment goes through fabric.

In such case we could use fabric to tunnel local proxy through ssh, to be temporarily accessible for the host being just provisioned in DMZ. Right now I'm opening a separate ssh tunnel on second console, so that fabric works within "context" of this tunnel.

example usage:

with rev_tunnel(local=8080, remote=8080):
    sudo("http_proxy='http://localhost:8080' apt-get install -y puppet")

munhitsu commented Jan 11, 2012

Imagine a DMZ setup which does not have an outbound http/proxy access.
Whole deployment goes through fabric.

In such case we could use fabric to tunnel local proxy through ssh, to be temporarily accessible for the host being just provisioned in DMZ. Right now I'm opening a separate ssh tunnel on second console, so that fabric works within "context" of this tunnel.

example usage:

with rev_tunnel(local=8080, remote=8080):
    sudo("http_proxy='http://localhost:8080' apt-get install -y puppet")
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Jan 11, 2012

Member

OK, so it is a fairly standard reverse tunnel setup after all I guess, and your desire re: Fabric is for it to handle doing the tunnel vs you having to run eg local(ssh -R ...).

I went back and forth on whether this is truly worth going into core, but it really would make a decent amount of sense for it to be supported in Fabric proper; the other solutions are hacky (eg some thread or subprocess running ssh -- how to do this well, make sure it shuts down when Fab does, etc) and I do see the validity of the use case (sharing local resources with the remote end during execution.)

Main roadblock is that I am not sure the SSH lib supports this yet; we'll have to figure that out and somebody would have to implement it if it doesn't. (I think that would be a good addition to said library, however, vs solidifying aforementioned ssh -R workaround.)

EDIT: #38 discusses implementation and patching and such. Might be best to actually close this and simply note there that when implemented, it should be possible to trigger with a context manager if possible.

Member

bitprophet commented Jan 11, 2012

OK, so it is a fairly standard reverse tunnel setup after all I guess, and your desire re: Fabric is for it to handle doing the tunnel vs you having to run eg local(ssh -R ...).

I went back and forth on whether this is truly worth going into core, but it really would make a decent amount of sense for it to be supported in Fabric proper; the other solutions are hacky (eg some thread or subprocess running ssh -- how to do this well, make sure it shuts down when Fab does, etc) and I do see the validity of the use case (sharing local resources with the remote end during execution.)

Main roadblock is that I am not sure the SSH lib supports this yet; we'll have to figure that out and somebody would have to implement it if it doesn't. (I think that would be a good addition to said library, however, vs solidifying aforementioned ssh -R workaround.)

EDIT: #38 discusses implementation and patching and such. Might be best to actually close this and simply note there that when implemented, it should be possible to trigger with a context manager if possible.

@munhitsu

This comment has been minimized.

Show comment
Hide comment
@munhitsu

munhitsu Jan 17, 2012

I fully agree that having it implemented so that we enter, leave contexts, or even worse multiple levels of contexts is tricky.

Regarding EDIT: Anything that keeps us closer to having this functionality is a good idea.

munhitsu commented Jan 17, 2012

I fully agree that having it implemented so that we enter, leave contexts, or even worse multiple levels of contexts is tricky.

Regarding EDIT: Anything that keeps us closer to having this functionality is a good idea.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Jan 17, 2012

Member

For now I'll leave this open, just to keep things granular. Assigned it to 1.4 so I don't forget, though. Chances are I will try to knock this out right as I am doing #38. Will update here when it's in master, thanks.

Member

bitprophet commented Jan 17, 2012

For now I'll leave this open, just to keep things granular. Assigned it to 1.4 so I don't forget, though. Chances are I will try to knock this out right as I am doing #38. Will update here when it's in master, thanks.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Nov 5, 2012

Member

This is actually not as related to #38 as I thought, since its changes are solely about gatewaying the SSH traffic itself, not tunneling additional ports through the SSH connection. That will require a different (or at least, additional) solution. Punting for now, sorry :) (Meaning: still open, just not going to arrive in 1.5.)

Member

bitprophet commented Nov 5, 2012

This is actually not as related to #38 as I thought, since its changes are solely about gatewaying the SSH traffic itself, not tunneling additional ports through the SSH connection. That will require a different (or at least, additional) solution. Punting for now, sorry :) (Meaning: still open, just not going to arrive in 1.5.)

@haridsv

This comment has been minimized.

Show comment
Hide comment
@haridsv

haridsv Apr 7, 2015

I recently needed this functionality to sync to a remote rsync server whose port is not directly reachable and found that the paramiko's forward.py demo code has sample code that I could use, so I came up with a solution that worked well for me and I submitted it as a patch for forward.py here: paramiko/paramiko#504

We could add ForwardServer from that patch and have a local_tunnel() that simply returns an instance of ForwardServer. As per recommendation from @bitprophet on the pull request, I will work on a patch for Fabric.

haridsv commented Apr 7, 2015

I recently needed this functionality to sync to a remote rsync server whose port is not directly reachable and found that the paramiko's forward.py demo code has sample code that I could use, so I came up with a solution that worked well for me and I submitted it as a patch for forward.py here: paramiko/paramiko#504

We could add ForwardServer from that patch and have a local_tunnel() that simply returns an instance of ForwardServer. As per recommendation from @bitprophet on the pull request, I will work on a patch for Fabric.

@haridsv

This comment has been minimized.

Show comment
Hide comment
@haridsv

haridsv Apr 7, 2015

I actually didn't realize, but there is already a patch for local_tunnel(), though I am not fully sure about its status. What should I do?

haridsv commented Apr 7, 2015

I actually didn't realize, but there is already a patch for local_tunnel(), though I am not fully sure about its status. What should I do?

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet May 4, 2015

Member

@haridsv If you're able, testing it out & mentioning that you successfully used it, on that patch (#939), would help, otherwise just have patience :) thanks!

Member

bitprophet commented May 4, 2015

@haridsv If you're able, testing it out & mentioning that you successfully used it, on that patch (#939), would help, otherwise just have patience :) thanks!

@naphthalene

This comment has been minimized.

Show comment
Hide comment
@naphthalene

naphthalene Apr 8, 2016

Is there any plan to resolve this issue? There are two outstanding pull requests that together resolve this, but they haven't been reviewed? Any way we could speed this along?
#939, #1218

I've been using the code in #1218 with no problems.

naphthalene commented Apr 8, 2016

Is there any plan to resolve this issue? There are two outstanding pull requests that together resolve this, but they haven't been reviewed? Any way we could speed this along?
#939, #1218

I've been using the code in #1218 with no problems.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Apr 9, 2016

Member

#939 is still in a release bucket, I just punted it from the upcoming 1.11 because I needed to cut some stuff, but it'll get priority for the next feature cycle. (And it looks like #1218 supersedes #939 so I'll probably end up merging that & crediting #939 in the changelog.)

Member

bitprophet commented Apr 9, 2016

#939 is still in a release bucket, I just punted it from the upcoming 1.11 because I needed to cut some stuff, but it'll get priority for the next feature cycle. (And it looks like #1218 supersedes #939 so I'll probably end up merging that & crediting #939 in the changelog.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment