Mosh, VPN, and MTU

[cgull]

Tonight I'm riding home on an Acela Express train. I started OS X's L2TP VPN client to access a server (running an IRC client). After connecting to the Acela Express wifi and starting VPN, my Mosh connection was not coming back up. I've actually seen issues with Mosh on my last 2 or 3 rides, but not figured out what was going on.

Looking at the network connection showed that mosh-server is sending packets at its max MTU (IRC scrollback no doubt), but the client is not receiving them. Running various tests with 'ping' suggests that maximum-MTU pings to my VPN server's external address succeed, but large pings over VPN to the server fail. The failure threshold appears to be about 1320 bytes, which is smaller than the the 1380-byte MTU in mosh master. Switching to phone tethering and bringing up the VPN there results in a working connection.

I suspect Amtrak's Wifi/hotspot provider is simply dropping large packets in known (and maybe unknown) VPN encapsulation, because they can't manage/shape that traffic and need to control users attempting to use VPN to evade bandwidth limits.

Ultimately, the battle between Acela VPN abusers and Amtrak's Wifi provider could become a game of whack-a-mole, and we're left trying to figure out how to get Mosh's transport to work in this environment.

We can reduce the IPv4 MTU to work around this specific case, but that's probably not a general solution. Some ideas I have include falling back to a small MTU on evidence of persistent one-way connectivity, or occasionally/randomly falling back to a small MTU to break MTU logjams.

In my specific example the client has chosen a 1280-byte MTU for its VPN tunnel interface (though the VPN server is using 1500 bytes). I don't believe the client MTU is related to this issue, but if mosh-server and mosh-client could learn MTUs and tell the other side, that might also help with other MTU/fragmentation issues.
#299 has several reports of VPN-related issues which may or may not resemble what I've seen here.

[cgull]

Neither Mosh nor VPN UDP packets seem to have DF set. Almost at destination, will examine pcaps later.

[The-Compiler]

Whoever @mosh-ci is, do you intend to post comments as "Mosh continuous integration"? 😉

[keithw]

Sigh. Ultimately in the long term, I think we're going to end up doing our own PMTU search (taking successful/acknowledged receipt as the positive signal, and not depending on ICMP / the socket error message).

[keithw]

Sorry about that, those were from me (trying to reply via email).

[mponomarenko]

I have similar issue, but on corporate Cisco VPN and home WiFi (over 2 internet providers)

I can see that my MTU is 1200

[keithw]

mponomarenko: Can you please test with mosh-client and mosh-server 1.2.4 and see if you have the same problem? I am confused if we have a regression here, or if there is just a rise in nonconforming VPN software.

[cgull]

Agreed that ICMP-less PMTU discovery is probably the long-term solution. But not for 1.2.6.

[cgull]

If you're running into VPN problems and are looking at this issue, please look at the request for information on #299-- if you can help us with a detailed report, that'll help us solve the problem.