NAT busting when server is behind NAT #48

Open
keithw opened this Issue Mar 8, 2012 · 23 comments

Comments

Projects
None yet
@keithw
Member

keithw commented Mar 8, 2012

We want to be able to connect to a server behind a NAT, even after the client roams.

We could use techniques like http://samy.pl/pwnat/ (per Nelson Elhage), except we'd need to keep doing them to pick up any client IP address changes.

@devurandom

This comment has been minimized.

Show comment
Hide comment
@devurandom

devurandom Apr 8, 2012

See also RFC 5128. libnice (ICE) might be an alternative to pwnat.

See also RFC 5128. libnice (ICE) might be an alternative to pwnat.

@lann

This comment has been minimized.

Show comment
Hide comment
@lann

lann Apr 11, 2012

Assuming this is only intended to apply to the UDP portion of the connection, the various ICE methods are unnecessary, and even the pwnat technique is overkill. The client can bind a UDP socket and send the port number to the server over SSH, which will send a dummy UDP packet to the client before returning its own UDP port number to accomplish the hole-punching.

Edit: pwnat's predecessor chownat [http://samy.pl/chownat/] does exactly this

lann commented Apr 11, 2012

Assuming this is only intended to apply to the UDP portion of the connection, the various ICE methods are unnecessary, and even the pwnat technique is overkill. The client can bind a UDP socket and send the port number to the server over SSH, which will send a dummy UDP packet to the client before returning its own UDP port number to accomplish the hole-punching.

Edit: pwnat's predecessor chownat [http://samy.pl/chownat/] does exactly this

@devurandom

This comment has been minimized.

Show comment
Hide comment
@devurandom

devurandom Apr 11, 2012

That is what I suggested in IRC previously. It was rejected.

The issue is that we have no communication channel available to send the IP/port number through. The SSH connection is just established once and for a short time, to start the mosh-server, then it is closed immediately. When the client roams and thus his IP changes, the server will not know about the new IP unless we notify it somehow. Sending him a packet is not an option, because the firewall will block it. Opening a new SSH connection is not an option, because that would require entering the password again.

That is what I suggested in IRC previously. It was rejected.

The issue is that we have no communication channel available to send the IP/port number through. The SSH connection is just established once and for a short time, to start the mosh-server, then it is closed immediately. When the client roams and thus his IP changes, the server will not know about the new IP unless we notify it somehow. Sending him a packet is not an option, because the firewall will block it. Opening a new SSH connection is not an option, because that would require entering the password again.

@lann

This comment has been minimized.

Show comment
Hide comment
@lann

lann Apr 11, 2012

Ah, good point. Still, it would be a good incremental improvement if a more robust solution isn't going to be implemented soon (no roaming behind a NAT vs no nothin' behind a NAT).

lann commented Apr 11, 2012

Ah, good point. Still, it would be a good incremental improvement if a more robust solution isn't going to be implemented soon (no roaming behind a NAT vs no nothin' behind a NAT).

@joneskoo

This comment has been minimized.

Show comment
Hide comment
@joneskoo

joneskoo May 30, 2012

I think it'd be worse to have a situation where you can start mosh but not reconnect to it. You'd get hanging sessions you cannot connect to any other way than ssh'ing in and pkill mosh-server. (Which is the same as now if your client kernel panics?)

I think it'd be worse to have a situation where you can start mosh but not reconnect to it. You'd get hanging sessions you cannot connect to any other way than ssh'ing in and pkill mosh-server. (Which is the same as now if your client kernel panics?)

@kmcallister

This comment has been minimized.

Show comment
Hide comment
@kmcallister

kmcallister May 31, 2012

Contributor

Which is the same as now if your client kernel panics?

Right, the server will hang forever if the client dies without contacting the server. This could happen because the client gets SIGKILL, or the client's machine reboots unexpectedly, or the packets notifying the server of shutdown are lost on each of several attempts.

Contributor

kmcallister commented May 31, 2012

Which is the same as now if your client kernel panics?

Right, the server will hang forever if the client dies without contacting the server. This could happen because the client gets SIGKILL, or the client's machine reboots unexpectedly, or the packets notifying the server of shutdown are lost on each of several attempts.

@inducer

This comment has been minimized.

Show comment
Hide comment
@inducer

inducer Jun 5, 2012

I acknowledge that client roaming is a thorny issue. That said, given how simple UDP hole punching on initial set-up would be to implement, I don't quite understand mosh's current stance of "Your server is behind NAT? Well, that's tough--you know what, we'll just make sure you can't benefit from mosh at all without painful manual setup, just to remind you that there's a thorny issue left to resolve."

inducer commented Jun 5, 2012

I acknowledge that client roaming is a thorny issue. That said, given how simple UDP hole punching on initial set-up would be to implement, I don't quite understand mosh's current stance of "Your server is behind NAT? Well, that's tough--you know what, we'll just make sure you can't benefit from mosh at all without painful manual setup, just to remind you that there's a thorny issue left to resolve."

@inducer

This comment has been minimized.

Show comment
Hide comment
@inducer

inducer Jun 5, 2012

Another suggestion: I imagine the typical set-up for server-behind NAT would be for somebody to have an ssh ProxyCommand that takes care of the onward connection from the publicly visible end of the NAT. mosh doesn't support this use case well, it seems. In particular, it tries to resolve the name passed to ssh as a hostname, when that hostname is likely an internal one, or even only valid as an ssh config name. Instead, it should optionally use a service like http://checkip.dyndns.com/ to find the NAT's public IP, report that via the initial SSH connection and then use that as the address to connect to. Alternatively, there should at be least a way to tell mosh what that public host name is.

inducer commented Jun 5, 2012

Another suggestion: I imagine the typical set-up for server-behind NAT would be for somebody to have an ssh ProxyCommand that takes care of the onward connection from the publicly visible end of the NAT. mosh doesn't support this use case well, it seems. In particular, it tries to resolve the name passed to ssh as a hostname, when that hostname is likely an internal one, or even only valid as an ssh config name. Instead, it should optionally use a service like http://checkip.dyndns.com/ to find the NAT's public IP, report that via the initial SSH connection and then use that as the address to connect to. Alternatively, there should at be least a way to tell mosh what that public host name is.

@kmcallister

This comment has been minimized.

Show comment
Hide comment
@kmcallister

kmcallister Jun 6, 2012

Contributor

I imagine the typical set-up for server-behind NAT would be for somebody to have an ssh ProxyCommand

No, I think the typical setup is just a forwarded TCP port.

Contributor

kmcallister commented Jun 6, 2012

I imagine the typical set-up for server-behind NAT would be for somebody to have an ssh ProxyCommand

No, I think the typical setup is just a forwarded TCP port.

@keithw

This comment has been minimized.

Show comment
Hide comment
@keithw

keithw Jun 6, 2012

Member

inducer: Thanks for your feedback. I would say that mosh is a young project, and the fact that we haven't yet accommodated a use case doesn't mean we're trying to be difficult!

However, generally speaking the philosophy with Mosh has been that things should work predictably and reliably. I don't like the idea that the user could start a session -- thinking it's roamable -- but then it would just hang when they actually tried to roam. That's like the worst kind of program, one that breaks only when you actually try to use it. :-)

Given that we've advertised Mosh as roamable, implementing a NAT-busting technique that doesn't roam worries me. I don't want to mislead users and have them cursing us after they try to roam.

Member

keithw commented Jun 6, 2012

inducer: Thanks for your feedback. I would say that mosh is a young project, and the fact that we haven't yet accommodated a use case doesn't mean we're trying to be difficult!

However, generally speaking the philosophy with Mosh has been that things should work predictably and reliably. I don't like the idea that the user could start a session -- thinking it's roamable -- but then it would just hang when they actually tried to roam. That's like the worst kind of program, one that breaks only when you actually try to use it. :-)

Given that we've advertised Mosh as roamable, implementing a NAT-busting technique that doesn't roam worries me. I don't want to mislead users and have them cursing us after they try to roam.

@inducer

This comment has been minimized.

Show comment
Hide comment
@inducer

inducer Jun 6, 2012

I do see what you're saying. But I get a feeling that a 'proper' solution will be a longer-term project, and in the meantime, getting 90% of the functionality in place (and perhaps warning about the difficult 10% not being in place) still has value--at least IMO.

I also know code speaks louder than me bitching in a bug discussion, but I thought it might be helpful if I tell you about a use case that I tried and that didn't work out using mosh's current state. Perhaps the ProxyCommand+NAT handling should be split off into a different bug, although it is related.

inducer commented Jun 6, 2012

I do see what you're saying. But I get a feeling that a 'proper' solution will be a longer-term project, and in the meantime, getting 90% of the functionality in place (and perhaps warning about the difficult 10% not being in place) still has value--at least IMO.

I also know code speaks louder than me bitching in a bug discussion, but I thought it might be helpful if I tell you about a use case that I tried and that didn't work out using mosh's current state. Perhaps the ProxyCommand+NAT handling should be split off into a different bug, although it is related.

@kmcallister

This comment has been minimized.

Show comment
Hide comment
@kmcallister

kmcallister Jun 6, 2012

Contributor

Perhaps the ProxyCommand+NAT handling should be split off into a different bug

I think so too. We do want to hear about this and keep track of the issue.

Contributor

kmcallister commented Jun 6, 2012

Perhaps the ProxyCommand+NAT handling should be split off into a different bug

I think so too. We do want to hear about this and keep track of the issue.

@inducer

This comment has been minimized.

Show comment
Hide comment
@inducer

inducer Jun 6, 2012

See issue #285.

inducer commented Jun 6, 2012

See issue #285.

@lann

This comment has been minimized.

Show comment
Hide comment
@lann

lann Jun 6, 2012

Given that we've advertised Mosh as roamable, implementing a NAT-busting technique that doesn't roam worries me. I don't want to mislead users and have them cursing us after they try to roam.

@keithw what if simple NAT-busting was enabled with a flag? Maybe even something verbose like --nat-traversal-danger-roaming-will-break :P

I would be happy to work on implementing this - mosh looks perfect for me (heavy console work over sometimes-laggy connections) other than this missing feature.

lann commented Jun 6, 2012

Given that we've advertised Mosh as roamable, implementing a NAT-busting technique that doesn't roam worries me. I don't want to mislead users and have them cursing us after they try to roam.

@keithw what if simple NAT-busting was enabled with a flag? Maybe even something verbose like --nat-traversal-danger-roaming-will-break :P

I would be happy to work on implementing this - mosh looks perfect for me (heavy console work over sometimes-laggy connections) other than this missing feature.

@deutrino

This comment has been minimized.

Show comment
Hide comment
@deutrino

deutrino Feb 20, 2013

Would it be in any way possible without a major architectural change to have a single "hi I roamed" port where mosh-server listens for (cryptographically identifiable) packets from clients to re-establish their particular connection?

Would it be in any way possible without a major architectural change to have a single "hi I roamed" port where mosh-server listens for (cryptographically identifiable) packets from clients to re-establish their particular connection?

@antonpiatek

This comment has been minimized.

Show comment
Hide comment
@antonpiatek

antonpiatek Aug 29, 2013

I am very much in favour of a nat busting flag. I know roaming has no nice solutions, but even a flag to let me get into a natted system once would be rather nice

I am very much in favour of a nat busting flag. I know roaming has no nice solutions, but even a flag to let me get into a natted system once would be rather nice

@lilydjwg

This comment has been minimized.

Show comment
Hide comment
@lilydjwg

lilydjwg Oct 12, 2013

Hi, I want to access my natted system too. I think mosh-client has to use a specific port when both side being restricted cone nats. Of course, roaming can't be done, but at least I can directly access the system now. I've made a dirty patch for mosh-client. And here's my little script for hole punching.

lilydjwg commented Oct 12, 2013

Hi, I want to access my natted system too. I think mosh-client has to use a specific port when both side being restricted cone nats. Of course, roaming can't be done, but at least I can directly access the system now. I've made a dirty patch for mosh-client. And here's my little script for hole punching.

@cvarta

This comment has been minimized.

Show comment
Hide comment
@cvarta

cvarta Mar 20, 2014

I always wondered why I couldn't get mosh-server to work on my home-workstation. It's behind a NAT (I configured port-forwarding of the mosh-UDP Port range 60000:61000) but it didn't help. The server is reachable via dyndns (ip changes maybe once or twice a year depending how long the cable-modem goes down: less than 15 min --> no IP change). Would be nice if you could state in the FAQ that the mosh-server doesn't work behind a NAT (couldn't find it).

cvarta commented Mar 20, 2014

I always wondered why I couldn't get mosh-server to work on my home-workstation. It's behind a NAT (I configured port-forwarding of the mosh-UDP Port range 60000:61000) but it didn't help. The server is reachable via dyndns (ip changes maybe once or twice a year depending how long the cable-modem goes down: less than 15 min --> no IP change). Would be nice if you could state in the FAQ that the mosh-server doesn't work behind a NAT (couldn't find it).

@eatnumber1

This comment has been minimized.

Show comment
Hide comment
@eatnumber1

eatnumber1 Jun 14, 2014

Two years since this bug has been opened. Let me propose a non-optimal solution: support SOCKSv5. Here's why:

I wanted to expose all the hosts on my internal nat's ssh ports to the internet. I could have done the standard method of forwarding non-standard ports over my router manually for each host I wanted to expose, but that doesn't scale well. Instead, I chose to set up a socks server which accepts connections from the internet and allows proxied connections to the ssh port of my internal machines. I then could write some simple ssh configs on my laptop to transparently do this proxying and it Just Works™.

Now this same principle could be applied for mosh. I could configure my proxy server to allow udp packets to be forwarded to my internal network on ports 60000-610000, and have mosh use this proxy server to forward packets to the machine it wants to talk to. This should even work for the roaming case!

It's got some drawbacks of course in that you have to set up a proxy server on the firewall, and you are exposing UDP on quite a few ports (60000-610000) to the internet, but nobody seems to have any better ideas.

I'd be willing to contribute some time towards making this happen if I got the ok for adding SOCKSv5 support to mosh. 😄

Two years since this bug has been opened. Let me propose a non-optimal solution: support SOCKSv5. Here's why:

I wanted to expose all the hosts on my internal nat's ssh ports to the internet. I could have done the standard method of forwarding non-standard ports over my router manually for each host I wanted to expose, but that doesn't scale well. Instead, I chose to set up a socks server which accepts connections from the internet and allows proxied connections to the ssh port of my internal machines. I then could write some simple ssh configs on my laptop to transparently do this proxying and it Just Works™.

Now this same principle could be applied for mosh. I could configure my proxy server to allow udp packets to be forwarded to my internal network on ports 60000-610000, and have mosh use this proxy server to forward packets to the machine it wants to talk to. This should even work for the roaming case!

It's got some drawbacks of course in that you have to set up a proxy server on the firewall, and you are exposing UDP on quite a few ports (60000-610000) to the internet, but nobody seems to have any better ideas.

I'd be willing to contribute some time towards making this happen if I got the ok for adding SOCKSv5 support to mosh. 😄

@akanouras

This comment has been minimized.

Show comment
Hide comment
@akanouras

akanouras Sep 14, 2014

If mosh-server is behind a Full-cone NAT, it supporting STUN (and making sure to refresh the mapping every once in a while) would be enough to solve this reliably.

For other types of NAT (except for Carrier Grade NAT), there are three main standards-based solutions:

  1. UPnP/NAT-PMP/PCP/etc.
  2. ICE, with the client using a new SSH connection every time it roams to coordinate the negotiation. STUN can also help in this scenario, unreliably however.
  3. SOCKSv5/manual port forwarding (both require administrator intervention however)

For Carrier Grade NAT and heavily firewalled NATs, assuming they allow outbound connections, an external TURN server to relay packets between client and server would be needed, if an SSH connection had already been achieved somehow. TURN support would be useful for ICE as well.

IMHO, implementing just STUN for the mosh-server would not be much effort using a library, would relieve a lot of people, and would be a good first step towards implementing the other methods as well.

If mosh-server is behind a Full-cone NAT, it supporting STUN (and making sure to refresh the mapping every once in a while) would be enough to solve this reliably.

For other types of NAT (except for Carrier Grade NAT), there are three main standards-based solutions:

  1. UPnP/NAT-PMP/PCP/etc.
  2. ICE, with the client using a new SSH connection every time it roams to coordinate the negotiation. STUN can also help in this scenario, unreliably however.
  3. SOCKSv5/manual port forwarding (both require administrator intervention however)

For Carrier Grade NAT and heavily firewalled NATs, assuming they allow outbound connections, an external TURN server to relay packets between client and server would be needed, if an SSH connection had already been achieved somehow. TURN support would be useful for ICE as well.

IMHO, implementing just STUN for the mosh-server would not be much effort using a library, would relieve a lot of people, and would be a good first step towards implementing the other methods as well.

@lachesis

This comment has been minimized.

Show comment
Hide comment
@lachesis

lachesis Sep 27, 2014

I'd love to see a way to make mosh work behind NAT with no setup. That'd let me use it in a lot of places I currently cannot.

I'd love to see a way to make mosh work behind NAT with no setup. That'd let me use it in a lot of places I currently cannot.

@Manouchehri

This comment has been minimized.

Show comment
Hide comment
@Manouchehri

Manouchehri Dec 25, 2014

Just thought I'd add my two cents in here. If you want to use mosh behind a NAT now, I would suggest "manually" punching holes yourself. There's already scripts that can run as non-root to get the job done, for example:

https://gist.github.com/somic/224795

If we do end up adding punching into mosh, it would have to be extremely explicit that you are removing the ability to roam; i.e., have a "--disable-roaming" and a "--udp-punch" flag that both have to be defined.

Just thought I'd add my two cents in here. If you want to use mosh behind a NAT now, I would suggest "manually" punching holes yourself. There's already scripts that can run as non-root to get the job done, for example:

https://gist.github.com/somic/224795

If we do end up adding punching into mosh, it would have to be extremely explicit that you are removing the ability to roam; i.e., have a "--disable-roaming" and a "--udp-punch" flag that both have to be defined.

@Neo-Desktop

This comment has been minimized.

Show comment
Hide comment
@Neo-Desktop

Neo-Desktop May 1, 2018

Is there an accepted way of nat-hole-punching?

Is there an accepted way of nat-hole-punching?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment