reconnect if server ip changed

[adzeitor]

If the server changed ip address may mosh try to reconnect to the new address.

[viric]

If the server address was given by name, it could attempt to re-resolve it. I'm interested in this feature too, as most disconnections to my server-behind-dynamic-ip happen for this reason.

[vadipp]

This would definitely be useful for my dynamic IP server with dynamic DNS set up.

[cetex]

I have a domain (say example.com)
When resolving from the internet i get my routers public IP which does NAT to my server.
At home i run a internal dns-zone to resolve example.com to my internal ip-address instead.

Result: I can always do "mosh example.com" to connect to my server.
Problem: Since mosh doesn't seem to try to rediscover the server ip-address it doesnt reconnect if i go from home-network to 3g/4g or my office net, even though i think it should work.

Logic should be something like:
After x seconds of connection loss, if server is specified by a hostname, do a new dns-query for the hostname and if the ip-address has changed try the new ip instead

This would also solve the ipv4/ipv6 roaming issue.
As long as you use a hostname mosh should try ipv6 record first, and if that fails within X seconds, IPV4 records.
If connection is lost, re-resolve the hostname and restart the procedure. IPV6 first, then IPV4.

Roaming between ipv4 and ipv6 probably isn't needed that much, at least not yet for a couple of years. There aren't that many ipv6-only sites available, so ipv4 will almost certainly always work on the server-end. But it's a nice feature none the less. 👍

[icorbett]

This would be awesome as it should make mosh much more capable of switching from vpn to lan connections, and the inverse.

[DaveQB]

I want to vote this up. I am constantly doing the CTRL+SHIFT+6 disconnect to reconnect when I change from home network to mobile or office. Defeating the purpose of mosh in some ways.

[pcchou]

👍

[mlambie]

I'm very keen for this feature too. Is it on the roadmap, or even possible?

[fungs]

This is the reason why I have to restart mosh every 24 hours, so it would boost the usability for me.

[NavinF]

Has this been implemented?

From the readme:

Mosh allows the client and server to "roam" and change IP addresses, while keeping the connection alive. Unlike SSH, Mosh can be used while switching between Wi-Fi networks or from Wi-Fi to cellular data to wired Ethernet.

If the client changes IP addresses, the server will begin sending to the client on the new IP address within a few seconds.

[Nippey]

Plus one.

I have neither control over the internet connection nor over the respective hardware:
The reconnect is unpredictable somewhere during the day.

If Mosh would try to resolve my domain name, which is updated some seconds after the reconnect, I could continue to work.

[osti]

I would also really appreciate this functionality. Is there any update?

[fungs]

For me, there are two major show stoppers in mosh and this is one of them, if you use dynamic dns on the server side (the other being firewalls blocking mosh ports).

[mlambie]

My workaround was to modify my internal DNS to always return the external IP for servers I connect to. Others could achieve the same thing with a static entry in their /etc/hosts file.

Importantly, this doesn't solve the problem of the server changing address dynamically, but it does solve the problem of a server resolving to a different (but static) IP within a set. In my case it was the IP allocated by my ISP, and a 10.0.0.0/8 address that DNS resolved for me when I was on the local network.

[tangledhelix]

I'll offer another use case for this which doesn't have to do with remote networks even. I do much of my development in a local VM on my laptop, which I connect to using a dynamic DNS name (like foo.local, using avahi-daemon and mdns). VMware seems to like changing its IP frequently, so even though I can mosh into the VM and not have it drop my session just because I sleep the laptop, it will stop working when the VM decides to change the IP. I'm not sure of a way to make VMware stop doing this.

[eskhool]

It looks like this bug will also go the way of the ssh-agent forwarding i.e. even if someone were to fork and implement it will not see the light of day from the core dev since none of them have even commented on this inspite of so many +1s

[fungs]

My summary: SSH survives only small interruptions with constant server and client IP. MOSH survives longer connection breaks and only when the client IP changes. None of them will survive an interruption due to a server IP change. Sad but true, MOSH does not live up to its promise to be an robust mobile shell not implementing this simple feature (more a bug than a feature in my opinion). In the future, even more devices which might act as terminal servers will have a dynamic IP. I probably would use MOSH every day if I would work around this bug, but I simply don't. Sorry, there is so much constructive feedback but no appreciation by the devs.

[lilydjwg]

Hi there, today I discover that a VPN tool, WireGuard, supports both side roaming. So you can mosh through that :-)

[tlhonmey]

If the documentation is accurate, if you can manage to extract the MOSH_KEY environment variable you'd then be able to reconnect to the session by calling mosh-client yourself. I'll see if I can figure out how to do this once my server comes back up.

[eminence]

This won't work, as the mosh crypto layer won't allow a second client to connect to an already running server. There's no way to extract the current nonce from the first client and no way to give it to the second client. #394 has some more details on this.

[tlhonmey]

Yeah, I noticed that. I'll have to look to see if there's a convenient place to allow punching in a different IP address since it does no name resolution itself (which is why this isn't a trivial feature I'm sure.)

If the trouble is getting the correct key to use, it could also dump the latest key to stdout on an unclean shutdown. (I'm assuming it rotates keys periodically like SSH does, but I haven't looked yet.)

[tlhonmey]

In src/network/network.cc you'll find a line (varies significantly with which revision you're looking at, but it's tagged with the comment, "only client can roam") where it explicitly checks if it's the server before considering whether or not to update the remote IP address.

Allowing the client to update the remote IP address as well doesn't seem to break anything, but it also doesn't seem to help (I suspect because the client isn't actually listening for incoming connections, but I haven't gotten that far yet.)

I'm still puzzling out how to plumb an ip change through from an escape character to the network layer, but I think it's possible.

Somebody who's a better C++ coder than I am might see how difficult it would be to crib the network communication methods out of the Syncthing project. STUN and relay support would make this work in more than just the best case of having control over the firewalls and NATs that are in the way.

[ChaiTRex]

@tlhonmey The server gets the client's new address from new packets sent to it by the client. It's not going to work to have the client get the server's address that way, since the server isn't going to be sending data forever to a client that doesn't seem to be there anymore, so there's probably not any packets from the server to receive.

For example, the client loses network access, then the server switches addresses, then five hours later, the client gets network access again. You wouldn't expect the server to be broadcasting for five hours to the last-known address of the client because the idea of Mosh is that the client can change addresses at any moment, so why keep the server sending to an address that may be dead or may belong to someone else now?

Even if Mosh was going to be wasteful and rude to some stranger who got the client's old address, it won't work if both the server and client have switched addresses, since the server isn't sending anything to the new client address and the client isn't sending anything to the new server address, so a solution that addresses all these basic cases will have to be something else.

[cetex]

To support the server changing IP the client would have to resolve DNS again and then just try to reconnect on both ipv4 and/or ipv6 to see if the same server is still there.

[ChaiTRex]

@cetex One important thing is that there should be two classes (as in classification, not object-oriented programming) of addresses it should try to reconnect to:

• [Only one address] The address on the packet of the last received transmission that was verified using cryptography to come from the server
• [Multiple addresses] The addresses (we can get more than one) DNS is giving now

The only ways to update these classes are, respectively:

• We receive a transmission that is verified using cryptography to come from the server
• DNS gives yet more new addresses (note especially that if DNS isn't responding or is saying that there aren't any addresses, we keep the last addresses DNS gave us)

Neither of the two classes can erase or modify the addresses in the other class.

---

Note that the second class (addresses that DNS gives us) should be stored even from the time we initially connect to the Mosh server. That way, if we lose the connection and DNS is flaky later on, perhaps a different IP address we initially got from DNS will work.

---

When trying the two classes of addresses, we should take a bit of care to ensure that, if the address in the first class is also in the addresses in the second class, we don't try to connect to it double the number of times as other addresses in the second class.

[tlhonmey]

@ChaiTRex Yeah, it didn't work. Adding a server -> client heartbeat is more than I know how to do currently, but might make it work as long as only one address changes at a time (which is probably the most common use case.)

If the server address is going to change, then the user of the client will need some way to know what the current address is in order to even make the connection. Normally this is probably DNS.

Unfortunately the mosh client doesn't actually do name resolution, it relies on its wrapper to look up the IP for it. So it's not as simple as just having it re-resolve the name and reconnect. The other solution I will try to look into when I have more time is to add an escape key option that will tell the client to grab a new server address from somewhere. But this project is too complex for me to load it into my head without having bigger chunks of time to allot to it than I currently can spare, so that will probably take me a while.

[unode]

For those of you landing on this question, if your mosh server changed address but the new address is reachable you can circumvent the issue with firewall/routing changes.

In my case I had a connection to an external IP address. When inside the network the IP was not reachable. Since DNS to the same hostname resolved to an internal address and I could mosh to this IP without problems, using:

iptables -t nat -A OUTPUT -d IP.used.by.mosh -j DNAT --to-destination new.IP.to.use

allowed me to recover my mosh sessions.

The same principle can be used for most cases where IP changes.

[fungs]

@unode, that's a nice hack. It should also be possible to do this in userspace without needing root using netcat/socat or something like NUSE and always connecting to a static local ip which is redirected by one of these tools. A clever wrapper would then achieve what we want. Of course, it would be much nicer to have proper hostname resolution in mosh :)

[cespedes]

Is there a roadmap to try to add this feature, or at least are the developers open to patches to fix this issue?

[fayeshine]

@keithw @cgull please help, we need this feature

[mlambie]

For me this is an issue connecting to mosh-servers that are available on the LAN and via WAN with port forwarding. I use the following function in zsh to bounce through a bastion server, regardless of my location (internal to the LAN or external). Mosh to the bastion which will keep the connection open, then regular SSH from the bastion to the target host.

mosh () {
        case $@ in
                (lifeline)  ;&
                (lifeline.domain.com) command mosh dialtone.domain.com -- bash -c 'echo "Bouncing via dialtone..." && echo && ssh lifeline.domain.com' ;;
                (*) command mosh "$@" ;;
        esac
}

[PenelopeFudd]

I'd solve this a different way:

• Currently the server accepts packets from anywhere, and if they pass authentication, the server replies to whatever ip address sent them.
  • This allows the client's address to change due to roaming, and it works perfectly.
• If the server changes ip address, then the client should do the same thing; if packets come in from a new ip address and it passes authentication, then it should start replying to it.
  • This would allow the server's address to change due to roaming too.

The thing preventing this from working 100% of the time is that the client may be behind a NAT firewall: if the server changes ip addresses, the firewall isn't going to forward packets from the new address.

There are a couple of workarounds for this developed for VOIP applications: UDP hole punching and the STUN protocol.

Actually, because this protocol uses UDP, there's another way to make it work:

• For every outgoing packet (on client or server), send two packets instead: one with the correct source but a fake destination, and one with the fake source but the correct destination:
  • The one with the fake destination will "punch a hole" for incoming packets from that fake destination.
  • The one with the real destination but fake source will then go in through that hole (at the other end).
  • There will be a hole in the client's NAT, and a hole in the server's NAT; the ip addresses of the two holes can be the same or different; it might confuse that NAT if they're the same.

Once the holes are set up, they may stay up without having to keep sending more hole-punching packets, depending on the NAT's settings

• This works even if there are no NATs
• The fake ip addresses should probably be reserved ip addresses
• A non-reserved ip address could probably be used if the TTL on the hole-punching packet was large enough to punch the hole but small enough to not reach the real owner of that ip.
• Once the two machines can talk again, they can figure out what the real ip addresses are and just send directly.
• For that matter, they could talk using only fake source addresses, once they know what the real destination addresses are.

Essentially, this is UDP hole-punching but without the third-party server.

[albertz]

@keithw @cgull @bbarenblat @achernya I'm familiar with C++ and I assume I can implement this. But there have been no response by any of the maintainers (you?) whether this is a wanted feature, and which of the suggestions should be implemented.

I see two suggestions here:

• If domain name is given, try to resolve it, check for difference. -> Should be very simple to do, and solve the problem in most cases here.
• UDP hole punching. -> More complex. It has some advantages but also disadvantages over the first approach.

I think both suggestion could be implemented independently. For now, I would just implement the first proposed suggestion, as it should be simple, and already solve the problem for most users.

So, should I submit a PR which implements that?

I also saw that there is already the very related PR #600.

There was one concern about getting a different host behind a round-robin DNS load balancer. But I see some solution to that problem: Check temporarily for a new IP, and only keep it if we can make the connection, otherwise don't keep it.

There was another concern about clogging a slow connection with frequent re-resolving the domain. I don't think this is a problem, if you only do this every N seconds or so.

[bbarenblat]

This is a feature I want, and I know this is a feature @achernya wants too. We believe re-resolving DNS is the way to go, at least for now. I will happily review a PR that implements this, either a modified #600 (cc: @nogy) or a new one.

Ideally, DNS re-resolution would happen automatically, whenever the connection to the server is down for a few seconds. It’s important that it not block the UI. I don’t remember what the state of asynchronous DNS resolution on POSIX systems is these days, so this might involve spinning up a new thread.

Additionally, any implementation should be off by default and behind a flag, at least for now. Once we’ve gathered enough user experience reports to verify that it works in the real world, we can turn it on for everyone.

[M0rtenB]

I too am looking for this!