Wrong / confusing MSS clamping behaviour with PPPoE MTU 1492

[DollarSign23]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

My ISP uses PPPoE, so my WAN MTU must be 1492 instead of 1500.
With this setup, some websites (for example Azure CDN–hosted sites over IPv6) do not open or time out. A packet capture shows that this is caused by incorrect MSS clamping on OPNsense.

To Reproduce

1. Configure WAN as PPPoE with MTU 1492.

2. Open some IPv6 sites behind Azure CDN – the connection hangs / times our. Use http://pmtud.enslaves.us to check.

3. Capture traffic on the WAN interface: you will see MSS values of 1460 (IPv4) and 1440 (IPv6), which do not fit into an MTU of 1492.

4. Now change the MSS setting so that you enter 1492 in the MSS field. The resulting effective MSS becomes 1452 (IPv4) and 1432 (IPv6), and the problem disappears.

Expected behavior

• OPNsense should automatically derive the correct MSS from the interface MTU (e.g. MTU – 40 for IPv4 and MTU – 60 for IPv6) and clamp accordingly, especially for PPPoE interfaces with MTU 1492; or

• The field should clearly be an MTU field, or there should be a checkbox like “Clamp MSS to interface MTU” (enabled by default) so the user does not have to do the header-size calculation manually.

Additional context

Check the original Reddit Post for more Details

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 25.7.7 (amd64).

[AdSchellevis]

The MSS clamping setting in the interface is a remnant from legacy choices I'm afraid, last time we touched that code we already made a note that it should be better if it would fully dissapear.

core/src/etc/inc/filter.inc

Lines 553 to 567 in f31afb4

	foreach ($FilterIflist as $scrubcfg) {
	if (is_numeric($scrubcfg['mss'] ?? '')) {
	/**
	* Legacy MSS clamping on interface expects outbound packets to be scrubbed in order to work.
	* https://github.com/pfsense/pfsense/commit/7c382a8
	*
	* In a future release we might want to consider to move the MSS option from the interface into a
	* manual scrubbing rule, this is a bit intransparant.
	*/
	$mssclampv4 = 'max-mss ' . (intval($scrubcfg['mss'] - 40));
	$mssclampv6 = 'max-mss ' . (intval($scrubcfg['mss'] - 60));
	$scrubrules .= "scrub on {$scrubcfg['if']} inet all {$scrub_gen_opts} {$mssclampv4}\n";
	$scrubrules .= "scrub on {$scrubcfg['if']} inet6 all {$scrub_gen_opts} {$mssclampv6}\n";
	}
	}

In reality, when mss clamping is needed, there's almost always somewhere another mtu misconfiguration leading to misalignments of packets (or not being able to fragment them properly).

With MSS you can more or less fix TCP, but then when other types of traffic are being used (such as UDP), you will be left with an unsolvable problem as mss doesn't apply there.

[DollarSign23]

Thanks for your answer and the link to the code.

I fully agree that MSS clamping is a legacy bandaid and that most of the time it is hiding a broken MTU/PMTUD setup somewhere in the path.

However, in this particular case there are two separate problems, and one of them is on OPNsense itself:

1. WAN MTU is correct, but OPNsense advertises an impossible MSS

• WAN is plain PPPoE, ifconfig pppoe0 clearly shows mtu 1492.
• With no MSS value configured on the interface, the SYNs leaving the PPPoE interface still carry
• MSS 1460 for IPv4
• MSS 1440 for IPv6
  which correspond to an MTU of 1500, not 1492.
• That means OPNsense is happily sending TCP segments that cannot fit into its own WAN MTU (1500-byte IP packets over a 1492 interface). This is visible in the packet capture and confirmed by pmtud.enslaves.us: MSS 1460/1440 fails, MSS 1452/1432 works.
  So even if the rest of the path were perfectly configured, these MSS values are already inconsistent with the PPPoE MTU at the firewall edge.

2. The MSS field behaves like an MTU field, but the user has to do the math.
   When I set the “MSS” field on the WAN interface to 1492, the resulting SYNs have:

• IPv4 MSS = 1452 (1492 – 40)
• IPv6 MSS = 1432 (1492 – 60)
  This matches the help text in the GUI:
  “If you enter a value in this field, then MSS clamping for TCP connections to the value entered above minus 40 (IPv4) or 60 (IPv6) will be in effect (TCP/IP header size).”

In other words: the field is actually treated as MTU input, not as a raw MSS value.
Once I put 1492 there, Azure CDN over IPv6 and the PMTUD test page both work fine.
From a UX and configuration perspective that is very confusing:

• The GUI calls it “MSS”, but internally it expects “MTU” and subtracts header sizes.
• Without touching it, OPNsense derives MSS values that ignore the PPPoE MTU.
• With PPPoE 1492 and LAN 1500 this leads to real-world breakage (Azure CDN over IPv6 timing out) unless the user manually enters the MTU into this field.

3. Yes, there is likely also a PMTUD black-hole on the Azure side
   I agree there is probably an additional misconfiguration in the path (ICMPv6 “Packet Too Big” filtered somewhere). As an end user I cannot fix Azure or any intermediate ISP. MSS clamping is the only realistic mitigation on my side.
   But regardless of that, OPNsense should not generate MSS values that can’t even traverse its own PPPoE MTU. That part is entirely under OPNsense’s control.

If MSS clamping is considered legacy and you’d like to get rid of it, that’s fine in the long term. But as long as:

• PPPoE WAN MTU defaults to 1492, and
• LAN usually runs at 1500,

it would be great if OPNsense at least:

1. Automatically derived MSS from the interface MTU (MTU–40 for IPv4, MTU–60 for IPv6) when the WAN MTU < 1500; or
2. Clearly labelled this as an MTU-based clamp, e.g. “Clamp TCP MSS to interface MTU”, with an optional override field, so the user doesn’t have to do header-size calculations by hand.

Right now the combination of:

• correct PPPoE MTU (1492),
• default MSS values (1460/1440), and
• a field called “MSS” that actually expects “MTU”

is what makes the behaviour questionable and confusing.

[AdSchellevis]

However, in this particular case there are two separate problems, and one of them is on OPNsense itself:

Well that depends on the rest of the configuration, mtu sizes of upper interfaces for example. If it's an (upstream) bug is hard to assess with the data known here, our forum might be a better place to discuss setup challenges, I believe quite some people use PPPoE.

In other words: the field is actually treated as MTU input, not as a raw MSS value.

It's a calculated field as explained in the help text (not being confused with mtu input), a strict mss setting can easily be offered via normalization (Firewall: Settings: Normalization)

[fichtner]

@DollarSign23 I've noticed today that we do not allow certain ICMPv6 types that may be required for PMTU to work properly. Maybe you can give 7824ce5 a try.

[JuggerReviry]

Had this same Issue, was very difficult for me to figure out what was causing it, since most services were able to handle the requests anyways.
Only by sheer coincidence I ran into Two Services using the same Content Delivery Network which didn't work with IPv6 + HTTPS.
Leaving them here for search ability:

• https://www.netze-bw.de/ (German Energy Provider)
• nProtect Gameguard Anticheat used in many games for example Helldivers 2 (https://gameguard.thehelldiversgame.com)

[engels0n]

Hi,
I had the same error and spent ages trying to find out what was causing it.

After entering 1492 in the MSS field on the WAN interface just for fun, everything seems to be running smoothly so far.

The @DollarSign23 test page now also shows the correct value of 1452 for IPv4.

I think it makes sense to take another look at the MSS field and check how and why it is being calculated incorrectly...

ISP: Deutsche Glasfaser / nordischnet
OPNsense VM on Proxmox

Thanks in advance ;)

[funtowne]

Confirming everything I've read in this chat with my own experiences w/ DNSMASQ and M-Net in Germany. (PPPoE over fiber)

edit: confirming the "MSS fix" from @engels0n appears to work as expected! I'm back on DNSMASQ away from kea and RADVD. If this causes any other downstream issues I'll post here.

Update ~2 weeks on -- no issues to report. Everything seems stable!

[funtowne]

Confirming everything I've read in this chat with my own experiences w/ DNSMASQ and M-Net in Germany. (PPPoE over fiber)

edit: confirming the "MSS fix" from @engels0n appears to work as expected! I'm back on DNSMASQ away from kea and RADVD. If this causes any other downstream issues I'll post here.

Update ~2 weeks on -- no issues to report. Everything seems stable!

Some IPv6 UDP traffic did not work stably with this "fix". No surprise since MSS only works for TCP.

I wiped out the MSS value in the WAN interface and forced DNSMASQ to announce a MTU of 1492 for my IPv6 assignments. This has made the connection more reliable. It'd be great to get DNSMASQ to recognize non-standard MTUs for IPv6 RAs as RADVD had done. ...or the other fix of sorting IPv6 PMTU to eliminate little bodges.

[funtowne]

@DollarSign23 I've noticed today that we do not allow certain ICMPv6 types that may be required for PMTU to work properly. Maybe you can give 7824ce5 a try.

Applied to 26.1.4. PMTU still fails for IPv6 when testing via https://pmtud.enslaves.us/ - client sends MSS of 1440, expects 1432.

[fichtner]

7824ce5 has been added to some 25.7.x version in the past. If it didn't help you probably already knew.

Cheers,
Franco

[funtowne]

7824ce5 has been added to some 25.7.x version in the past. If it didn't help you probably already knew.

Cheers, Franco

Missed that, been poking around to understand this MTU situation and had a bit of a play with the patch. Thanks for confirming!