Tailscale requires MTU <= 1280 for relayed traffic from non-tailscale subnets

[vikmik]

What is the issue?

Hello!

I have a 3-machine setup, where 1 machine doesn't have tailscale:

A (no tailscale) -> B ("subnet router", relay to tailscale network) -> C (exit node) -> (internet)

Browsing the internet on A, everything works fine... except some websites like https://duckduckgo.com . (google, youtube... etc work fine)
Inspecting the issue for problematic websites, I noticed the TLS Server Hello from the internet getting dropped by the exit node - and the exit node replying with ICMP 3,4 "MTU is 1280, please fragment" to the website, but they probably disappear in a PMTU black hole and get ignored - classic.
In summary: A's interface has an MTU of 1500, so the client on A announces an MSS of 1460 bytes - and the website consequently replies back with IP packets that don't fit in the tailscale link. And PMTU breaks because it's the internet wild west.

I'm able to make things work by changing the MTU on A to 1280 via DHCP option (I do not control the device, but I can tweak the DHCP server).
I can also fix things by clearing the DF flag in very despicable ways on the exit node (clearing the DF flag on forwarded packets from internet to tailscale will allow IP fragmentation to happen instead of getting dropped)

Steps to reproduce

Replicating the setup might be a little annoying, but it seems logically equivalent to reproduce with 2 machines by doing:

Machine A: use exit node B
Machine B: setup as exit node

Then, on machine A: sudo ifconfig tailscale0 mtu 1281 (1 higher than 1280)
At that point, browsing the web on machine A (with B as exit node) should work pretty seamlessly, but visiting https://duckduckgo.com will not work for example: the announced MSS will prevent the exit node to forward internet traffic back to the client through the tailscale link.

Manually changing the MTU on tailscale0 is obviously not a normal behavior, but I think it is equivalent (in terms of generated traffic) to the 3-node setup I was describing.

Given that:

1. tailscale introduces a link with low-ish MTU (1280)
2. Clients behind a tailscale "subnet router" and servers on the internet will both usually announce MSS around ~1460, too high to be compatible with tailscale link's default MTU

This means the traffic has high chances of requiring PMTU to work perfectly, which is often not the case (cf duckduckgo) - so in practice chances of traffic being affected by PMTU issues are higher than in normal, non-tailscale conditions.

=> It would be good to mention on the relevant page ( https://tailscale.com/kb/1019/subnets/ ) that that TCP clients behind tailscale "subnet routers" may want to announce an appropriate (low) MSS to avoid PMTU issues at the exit node (and practically, setting their interface MTU to tailscale's MTU will do the trick). This would help with the "principle of least astonishment", especially as the setup I am describing seems very simple.

Thanks!

Are there any recent changes that introduced the issue?

No response

OS

Linux, Android

OS version

Debian 12, OL 7.9

Tailscale version

1.20.2

Bug report

BUG-85d741a3f5ca3f3ba86a0dd9cb0767178750e75f9f2aa7941c08c0f725bd530f-20220128165433Z-2529369bd3d4f577

[vikmik]

https://twitter.com is another "problematic" website with which it is easy to repro, except it will not completely fail like duckduckgo - twitter will eventually try to fragment below the MSS after a few tries. It takes ~7-10 seconds for the TCP connection to be able to transmit data

[DentonGentry]

Duplicate of #311

[clouvet]

I recently ran into this issue with a similar setup. I configured a Raspberry Pi (Machine B, to use the language from the above example) as a WiFi router that sent all its traffic through a separate Tailscale exit node (Machine C). Any client (Machine A) connected to the router worked mostly fine but ran into problems with some sites. Testing Machine A with a direct connection to the exit node resulted in no problems. It was only when connected through the router.

As described above, requests to both duckduckgo.com and twitter.com exhibited issues. Specifically though, I was able to determine that the primary problem seemed to be with any sites using Fastly. Like @vikmik I also saw the TLS handshake fall apart:

curl --verbose https://fastly.com
*   Trying 151.101.193.57:443...
* Connected to fastly.com (151.101.193.57) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):

That same outcome happened with any site using Fastly (reddit.com, nytimes.com, avatars.githubusercontent.com, amongst many others).

Between this issue and this forum post, I was able to solve the problem with an iptables rule on the Raspberry Pi (Machine B), the go-between for non-Tailscale devices and the Tailscale exit node:

iptables -t mangle -A FORWARD -i tailscale0 -o eth0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

In the event that it may prove useful information, I'm writing this comment now for anyone who runs into this issue in the future and also to add the detail about specifically websites that use Fastly causing problems.

[pcl]

I automated the application of the MTU limit and the iptables rule with a systemd unit. I put mine at /etc/systemd/system/tailscale-mtu-fix.service

[Unit]
Description=Fix Tailscale MTU
After=tailscaled.service
Requires=tailscaled.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c "/sbin/ip link set tailscale0 mtu 1280 && /sbin/iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu"
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

To enable:

sudo systemctl daemon-reload
sudo systemctl enable tailscale-mtu-fix.service
sudo systemctl enable tailscale-mtu-fix.service
sudo systemctl status tailscale-mtu-fix.service

I used systemd for this since tailscale wants to set its own iptables rules and NetworkManager does gods-know-what to the rules, so systemd seemed like an easy place to hack into without learning too much about the details.