docs: fastd: recommend better default mtu

[mweinelt]

It is known that a fastd MTU of 1280 bytes will cause excessive fragmentation for full packets on networks that advertise to transport payloads of 1280 bytes, because it neglects the headers that are being added by the batman-adv encapsulation.

The following graphic is widely recognized as factually correct and it depicts this issue quite well.

I therefore think the default documentation should reflect this by at least recommending a MTU value that allows for unfragmented IPv6 communication to pass the network, decreasing the burden on all of our embedded routers.

This will in turn allow new communities, who certainly heavily rely on gluons documentation, to make better choices when setting up their networks.

Thanks to @ohrensessel and others who didn't let the MTU discussion die, so we can build better networks.

[jplitza]

What is the point of aiming for unfragmented 1280 byte packets? Yes, that's what IPv6 mandates as minimum MTU. But no client, no device actually uses 1280 byte MTU, because most ignore what they are told in DHCP replies and RAs, what the MTU should be.

[mweinelt]

That may be true, but some still do. Even then carefully calculated MSS Clamping reduces payload sizes to match the target MTU.

Currently clients that adhere to the advertised MTU set their link to 1280 and send full frames that need fragmentation every time. Setting it lower than that breaks IPv6 connectivity. Setting the transport MTU higher makes the fragmentation unnecessary.

[jplitza]

I'd really like to know which clients adhere to the advertised MTU. And MSS Clamping works for the current value, too, doesn't it?

[T-X]

@jplitza: What about ICMPv6 Packet Too Big messages? At least for IPv6 I would expect it to work reliably as IPv6 dumped the intermediate fragmentation support, making Packet Too Big messages essential for any IPv6 communication to work reliable.

[mweinelt]

@jplitza I can see this behaviour on my N5X running Stock Android 7.1.2 for example.

Yes, and MSS clamping works solely for TCP.

[jplitza]

@T-X So you're suggesting that the gateway sends a Packet Too Big reply for each new connection as soon as the client sends a packet thats above 1280 byte? That sounds like it could work.

@mweinelt Hm, interesting. Last time I checked, neither Android nor Ubuntu nor Windows seemed to care about it, but I don't remember which versions that were.

And because it seems relevant to the whole MTU and fragmentation discussion: IP Fragmentation is broken (by Cloudflare)

[ohrensessel]

Windows 7 and 10 respect the MTU given by RAs afaik, but only for v6. Can check again if required. To force a MTU of 1280 you also set this mtu for the uplink interface of the gateway, e.g the tunnel. Then even non-tcp packets would not exceed 1280 byte. Am 20.08.2017 14:14 schrieb "Jan-Philipp Litza" <notifications@github.com>:

…

@T-X <https://github.com/t-x> So you're suggesting that the gateway sends a Packet Too Big reply for each new connection as soon as the client sends a packet thats above 1280 byte? That sounds like it could work. @mweinelt <https://github.com/mweinelt> Hm, interesting. Last time I checked, neither Android nor Ubuntu nor Windows seemed to care about it, but I don't remember which versions that were. And because it seems relevant to the whole MTU and fragmentation discussion: IP Fragmentation is broken (by Cloudflare) <https://blog.cloudflare.com/ip-fragmentation-is-broken/> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1210 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAvWTIdP1TDBkdg4dUClKN9b1_3zeZolks5saCM2gaJpZM4O8eaR> .

[mweinelt]

Apple iOS 10.3.3 does not adhere to the advertised MTU. Happily sends SYN packets with a 1460 MSS, unfortunately there does not seem to be an option to directly view the MTU on these devices.

[mweinelt]

@jplitza I can see this behaviour on my N5X running Stock Android 7.1.2 for example.

Still valid for Android 8.0 (Oreo), which was released yesterday.

[T-X]

From @ohrensessel:

To force a MTU of 1280 you also set this mtu for the uplink interface of the gateway, e.g the tunnel. Then even non-tcp packets would not exceed 1280 byte.

Just in case anyone is wondering why this works: This is exactly the ICMPv6 Packet Too Big message case :-).

(The kernel gets a "large" packet, sees it does not fit through the 1280 bytes MTU tunnel, drops it and sends an "ICMPv6 Packet Too Big" back to the sender which retransmits with a smaller packet size, generally the one stated in the previous ICMPv6 message. By default, this ICMPv6 PTB behaviour happens automatically on a Linux router.)

[Adorfer]

I do not get the point of restarting the complete MTU-fragemtation discussion the n-th time again.
with all the arguments already well evaluated the last time.

is there any doubt about the conclusion "fragmented packets should be avoided if possible"?
(and if not possible on certain cases, it does not help to divert to OS wars of any kind)

what about the initial question: proposing an MTU which
a) leaves IPv6 1280 payload packets unfragmented and
b) does not fragment on CGN/DSlite when passing fastd via servers reachable only by IPv4

Since i saw a lot of communities starting with fastd-mtu like 1280 or 1486 in their site.conf (and other rather unfortunate values), i would support any documentation rework in putting a resonable value like 1312, plus a warning lowering it (breaking considerable features for IPv6) and rising it above 1354 may reduce speed for certain dialup connections using VPN.

[mweinelt]

When we last visited that topic there were very few devices, if any, that adhered to the MTU announced through DHCP and Router Advertisments. These device are imposing an additional load on our routers by sending full packets that require fragmentation.

With the OS/device situation changing we can now properly inform those devices so they adjust their outgoing packets accordingly. And Android makes up a big part of our network, probably yours as well.

This is why "OS wars", as you call them, are a necessary argument in this discussion to get the documentation updated.

[Adorfer]

@mweinelt

1. there is a sweet spot for the fastd-mtu somewhere between 1312 and 1364 (ymmd), preferrably at the low end, as your PR states (which i fully support)
2. there is a number of devices which do not operate as we consider it as optimal, even if we apply the exhausitive lists of tweaks (dhcp-options, radvd-options, mss/clamping on the router etc.pp), which all may have one or the other downturns.

But whatever happens in section 2, i do not see why this would influence section 1.

selection of 1280 as fastd-mtu makes things even worse on for "non cooperative devices", but it may make things better if at least 1280 can be send unfragmented. .

(the only argument against selection of a good MTU could be "punish bad designed clients". But this is nothing i would seriously consider.

[christf]

@mweinelt while I already like the commit in this PR I do also see some benefit in explaining 1312 a little.

[rotanid]

so, 1312 works great (better than 1280) here. first in experimental and since 2 weeks on the stable branch.
we measured around 30% performance increase for every device type, even WR841N

@mweinelt maybe add a short explanation for the suggested mtu value like @christf asked ?

[A-Kasper]

Our Tunnel have an MTU of 1374. (Fastd) radvd and dhcpd advertise an MTU of 1342 no problems. This Setup works for over 6 Month without Problems. I took all known providers in Bochum and asked People to measure their maxmtu for ipv6 and ipv4. after that i calculated the overheads and came to these values. attention: this maybe different with other isps... 2017-09-12 18:13 GMT+02:00 Andreas Ziegler <notifications@github.com>:

…

so, 1312 works great (better than 1280) here. first in experimental and since 2 weeks on the stable branch. we measured around 30% performance increase for every device type, even WR841N @mweinelt <https://github.com/mweinelt> maybe add a short explanation for the suggested mtu value like @christf <https://github.com/christf> asked ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1210 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIA7Uvj7Jm9BX8IyTiS9oZckrAX1EFhjks5shojkgaJpZM4O8eaR> .

[mweinelt]

Rough idea of how to document that chapter as follows. Some terms are still a bit wonky and I want you to give me feedback on things that look wrong. Actually I believe that collaborative editing would yield a better result faster, but let's start with this:

---

Transport MTU

Setting the MTU on the transport interface requires careful consideration, as setting it too low will cause excessive fragmentation and setting it too high may leave peers with a broken tunnel due to packet loss.

Consider these key values:

• Payload: Allow for the transport of IPv6 packets, by adhering to the minimum MTU of 1280 Byte [RFC 2460]
  • and configure MSS clamping accordingly,
  • and announce your link MTU via Router Advertisments and DHCP
• Encapsulation: Account for the overhead created by the configured mesh protocol encapsulating the payload, which is
  • up to 32 Byte (14 Byte Ethernet + 18 Byte batadv) for batman-adv compat v15 (v2014.0 and later)
  • up to 28 Byte (14 Byte Ethernet + 14 Byte batadv) for batman-adv compat v14 (v2011.3.0 until and including v2013.4.0)
• PMTU: What MTU does the path between your gateway and each of its peers allow?

Minimum MTU

Calculcate the minimum transport MTU by adding the encapsulation overhead to the minimum payload MTU required. This is the lowest recommended value, since going lower would cause unnecessary fragmentation for clients which respect the announced link MTU.

Example: Our network currently uses batman-adv v15 and therefore requires up to 32 Bytes of encapsulation overhead on top of the minimal link MTU required for transporting IPv6:

1312              1294          1280                                 0
  +-----------------+-------------+----------------------------------+
  |    batadv v15   |   Ethernet  |            Payload               |
  +-----------------+-------------+----------------------------------+

MTU_LOW = 1280 Byte + 14 Byte + 18 Byte = 1312 Byte

Maximum MTU

Calculating the maximum transport MTU is interesting, because it increases the throughput, by allowing larger payloads to be transported, but also more difficult as you have to take into account the tunneling overhead and each peers PMTU, which varies between providers. The underlying reasons are mostly PPPoE, Tunneling and IPv6 transition technologies like DS-Lite.

Example: The peer with the smallest MTU on your network is behind DS-Lite and can transport IPv4 packets up to 1436 Bytes in size. The tunnel uses IPv4 (20 Byte), UDP (8 Byte), Fastd (24 byte) and you require TAP (14 Byte) for Layer 2 (Ethernet) Tunneling.

1436                 1416     1408                    1384          1370
  +-------------------+--------+-----------------------+-------------+
  |        IP         |  UDP   |         Fastd         |     TAP     |
  +-------------------+--------+-----------------------+-------------+

MTU_HIGH = 1436 Byte - 20 Byte - 8 Byte - 24 Byte - 14 Byte = 1370 Byte

Conclusion

Determining the maximum MTU can be a tedious process, especially since the PMTU of peers could change at any time. The general recommendation for maximized compatibility is therefore the minimum MTU of 1312 Byte, which works well with all combinations of IPv4, IPv6, batman-adv compat v14 and v15.

[lemoer]

@mweinelt Maybe something like this :)?

\   1312              1294          1280                                 0
 \----+-----------------+-------------+----------------------------------+
  \   |    batadv v15   |   Ethernet  |            Payload               |
   \--+-----------------+-------------+----------------------------------+
    \

MTU_LOW = 1280 Byte + 14 Byte + 18 Byte = 1312 Byte

1436                 1416     1408                    1384          1370 \
  +-------------------+--------+-----------------------+-------------+----\
  |        IP         |  UDP   |         Fastd         |     TAP     |     \
  +-------------------+--------+-----------------------+-------------+------\
                                                                             \
MTU_HIGH = 1436 Byte - 20 Byte - 8 Byte - 24 Byte - 14 Byte = 1370 Byte

[mweinelt]

Basically this. Not sure how good this works with the horizontal scroll here vs readthedocs.

1436                 1416     1408                    1384          1370 <-----> 1312              1294          1280                                 0
  +-------------------+--------+-----------------------+-------------+---  ...  ---+-----------------+-------------+----------------------------------+
  |        IP         |  UDP   |         Fastd         |     TAP     |     ...     |    batadv v15   |   Ethernet  |            Payload               |
  +-------------------+--------+-----------------------+-------------+---  ...  ---+-----------------+-------------+----------------------------------+

[lemoer]

Hm. This looks more as if there are some bytes between TAP and batman layer.

[lemoer]

\        1312              1294          1280                                 0
 \---------+-----------------+-------------+----------------------------------+
  \TAP     |    batadv v15   |   Ethernet  |            Payload               |
   \-------+-----------------+-------------+----------------------------------+
    \      ^
           |

        MTU_LOW = 1280 Byte + 14 Byte + 18 Byte = 1312 Byte

1436                1416     1408                    1384          1370    \
  +-------------------+--------+-----------------------+-------------+------\
  |        IP         |  UDP   |         Fastd         |     TAP     |    bat\
  +-------------------+--------+-----------------------+-------------+--------\
                                                                     ^         \
                                                                     |

     MTU_HIGH = 1436 Byte - 20 Byte - 8 Byte - 24 Byte - 14 Byte = 1370 Byte

[mweinelt]

Yeah, this makes it much clearer.

[mweinelt]

PR updated.