TL;DR When mtu_fix is enabled mss side is only altered for incoming traffic but not outgoing. This affects clients with pmtu disabled.
The issue is quite difficult to reproduce as it requires specific conditions however it happens in real setups quite regularly.
The conditions are bellow:
OpenWrt router working in "router" mode with IPv4 traffic NATed towards Internet
Upstream is connected by means of something that has reduced MTU (PPPoE, DS-LITE, etc)
mtu_fix is applied to WAN interfaces
The issue:
MSS from a client towards a server in the internet is not reduced to fit reduced MTU if the client has PMTU disabled (one example of such a client - LineageOS 16.x). As a result the packets of maximums MSS towards the server will be dropped.
Solution:
Enable mss clamping form incoming SYN packets on the interface with mtu_fix enabled
iptables without the fix:
iptables-save -t mangle | grep clamp
-A FORWARD -o ds-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
iptables with the fix:
iptables-save -t mangle | grep clamp
-A FORWARD -o ds-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i ds-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
TCP session initiation without the fix (WAN mtu is 1452):
Note ** mss 1420 ** in the syn packet towards the client. This make the client consider 1420 being the maximum possible mss however of the client tries to send such a packet with DF flag set it will be silently dropped.
For the record, it's not that obvious that the fix works: it depends on the behaviour of ''--clamp-mss-to-pmtu''. If it only looks at the PMTU towards the destination IP of the packets, it won't work for incoming packets.
It turns out that ''--clamp-mss-to-pmtu'' looks at both source and destination IP and that's why it works:
--clamp-mss-to-pmtu
Automatically clamp MSS value to (path_MTU - 40 for IPv4; -60 for IPv6). This may not
function as desired where asymmetric routes with differing path MTU exist — the kernel
uses the path MTU which it would use to send packets from itself to the source and
destination IP addresses. Prior to Linux 2.6.25, only the path MTU to the destination
IP address was considered by this option; subsequent kernels also consider the path MTU
to the source IP address.
Last time I saw this issue was when reviewing a wireguard setup. IP fragmentation
in the client->server direction was very frequent in the pppoe-wan interface. I think this is why the old way mostly works, fragmentation in one direction.
At that time I fixed it by setting a correct mtu value (16 multiples) on the local openwrt side, and having mss clamp in both directions on the remote side.
giner:
TL;DR When mtu_fix is enabled mss side is only altered for incoming traffic but not outgoing. This affects clients with pmtu disabled.
The issue is quite difficult to reproduce as it requires specific conditions however it happens in real setups quite regularly.
The conditions are bellow:
The issue:
Solution:
iptables without the fix:
iptables-save -t mangle | grep clamp
-A FORWARD -o ds-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
iptables with the fix:
iptables-save -t mangle | grep clamp
-A FORWARD -o ds-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i ds-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
TCP session initiation without the fix (WAN mtu is 1452):
tcpdump -ni wlan0 "(tcp[tcpflags] & (tcp-syn) != 0) and (host www.twitter.com)"
11:05:25.531925 IP 192.168.1.141.35040 > 104.244.42.129.80: Flags [S], seq 2878102236, win 64240, options [mss 1460,sackOK,TS val 2138517727 ecr 0,nop,wscale 7], length 0
11:05:25.536034 IP 104.244.42.129.80 > 192.168.1.141.35040: Flags [S.], seq 180174328, ack 2878102237, win 28960, options [mss 1420,sackOK,TS val 2685758451 ecr 2138517727,nop,wscale 8], length 0
Note ** mss 1420 ** in the syn packet towards the client. This make the client consider 1420 being the maximum possible mss however of the client tries to send such a packet with DF flag set it will be silently dropped.
TCP session initiation the fix (WAN mtu is 1452):
tcpdump -ni wlan0 "(tcp[tcpflags] & (tcp-syn) != 0) and (host www.twitter.com)"
11:02:42.062918 IP 192.168.1.141.34954 > 104.244.42.129.80: Flags [S], seq 4079972940, win 64240, options [mss 1460,sackOK,TS val 2138354260 ecr 0,nop,wscale 7], length 0
11:02:42.067116 IP 104.244.42.129.80 > 192.168.1.141.34954: Flags [S.], seq 416388380, ack 4079972941, win 28960, options [mss 1412,sackOK,TS val 920483133 ecr 2138354260,nop,wscale 8], length 0
Note ** mss 1412 ** in the syn packet towards the client. This is a real maximum for mss.
The text was updated successfully, but these errors were encountered: