19.1 upgrade sets WAN MTU to 576 bytes

[mimugmail]

I followed the reports about strange behavior with some guys after updating. When you check forums or Reddit it seems the WAN MTU was set to 576 bytes and reconfigure fixes it. I cant reproduce, perhaps it's only related to Realtek as most users reporting had them.

I also had 2 long Teamviewer sessions with one of them and I'm quite sure it's not a local configuration issue.

Any ideas?

[immanuelfodor]

Setting it to 1500 via web gui on the wan interface (previously that box was empty = default) solved all my issues after 18.07->19.1 upgrade. I have Intel NICs though.

[mimugmail]

Ah, you are the reddit guy :) Thanks for reporting. I myself am not affected, I'll update some more boxes ...

[immanuelfodor]

Yeah, I looked for the issue right away to subscribe if I can help with something to the team if there are any questions.

[mimugmail]

So, the user yesterday and you use DHCP as WAN. I only have static or PPPoE devices here,, that could be a reason not everyone's affected. What happens when you remove the entry and reboot? Again 576 bytes? Could you check DHCP log?

[mimugmail]

https://redmine.pfsense.org/issues/8507

[immanuelfodor]

Removed the custom 1500 MTU setting from the GUI (empty box), applied it and all my issues came back, slow page load speeds, inaccessible websites, etc. I didn't even had to reboot, the evil 576 appeared instantly.

ifconfig # of WAN interface
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 576
        options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether xxxxxxxx
        hwaddr xxxxxxxx
        inet6 xxxxxxxx%vtnet1 prefixlen 64 scopeid 0x2
        inet xxxxxxxxx netmask 0xfffffc00 broadcast 255.255.255.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active

netstat -4rnW
Routing tables

Internet:
Destination        Gateway            Flags       Use    Mtu      Netif Expire
default            xxxxxxxxx     UGS         181    576     vtnet1
127.0.0.1          link#4             UH        45217  16384        lo0
xxxxxxxx    link#2             U           690    576     vtnet1
xxxxxxxx      link#2             UHS         285  16384        lo0
xxxxxxxx     link#1             U      29315153   1500     vtnet0
xxxxxxxx        link#1             UHS           0  16384        lo0
xxxxxxxx     link#8             U             0   1500 vtnet0_vlan4
xxxxxxxx        link#8             UHS           0  16384        lo0

grep mtu /var/log/dhcpd.log
# empty, also with MTU

vtnet1 is facing the ISP, 0 is the LAN, 4 is VLAN for IoT devices.

When I set back the 1500 over the gui, everything is fine again.

ifconfig
# only change is the 1500 mtu

netstat -4rnW
Routing tables

Internet:
Destination        Gateway            Flags       Use    Mtu      Netif Expire
default            xxxxxxxx     UGS         218    576     vtnet1
127.0.0.1          link#4             UH        45553  16384        lo0
xxxxxxxx    link#2             U          1085   1500     vtnet1
xxxxxxxx      link#2             UHS           1  16384        lo0
xxxxxxxx     link#1             U      29546475   1500     vtnet0
xxxxxxxx        link#1             UHS           0  16384        lo0
xxxxxxxx     link#8             U             0   1500 vtnet0_vlan4
xxxxxxxx        link#8             UHS           0  16384        lo0

# still no output for greps

Here I can still see an 576 value for the ISP but the other one became 1500 (3rd line).

[mimugmail]

In Interface Config under DHCP, Client Config, Advanced, set:

supersede interface-mtu 0

in Option Modifiers ...

Then try again .. seems to be the solution for this. @fichtner and @marjohn56 know better if there's a chance to get this in per default

[jamiew0w]

In Interface Config under DHCP, Client Config, Advanced, set:

supersede interface-mtu 0

in Option Modifiers ...

Then try again .. seems to be the solution for this. @fichtner and @marjohn56 know better if there's a chance to get this in per default

Can confirm the above works. Thanks @mimugmail !

[marjohn56]

issue is not there on my live router after upgrade to 19.1 ( PPPoE static ). nor is it there on my test router running 19.1,rc2 ( DHCP ) not had chance to update it yet, ). Both units are Qotoms using Intel NICs. Need more reports to try and isolate what the issue is.

[marjohn56]

@jamiew0w - Can you confirm if this is a clean install or upgrade and what the firmware is?

[immanuelfodor]

@mimugmail okay, so I added this and now in netstat -4rnW the first line is also 1500.

[marjohn56]

By default in FreeBSD it should be set to 1500 however.....
https://forums.freebsd.org/threads/my-isp-giving-a-mtu-of-576.67860/

[immanuelfodor]

@mimugmail now I tried removing the manual MTU 1500 setting, and left only the supersede interface-mtu 0 advanced setting there, and ifconfig and netstat -4rnW are both reporting mtu=1500 everywhere, so this is the only thing we need to set, you nailed it! 😎

[jamiew0w]

@marjohn56 was an upgrade from 18.7 - the NICs are;

2x Broadcom NetXtreme II BCM5716 (default gigabit nics on dell r310) - only needs to be set on WAN though.

Tried setting the MTU to 1500 via the gui but didn't work, was still getting the 576 mtu error afterwards

edit; Like @immanuelfodor mentioned above, the manual setting of 1500 doesn't need to be there, only the fix suggested by @mimugmail and everything is rosy afterwards.

[marjohn56]

OK, I'm doing a clean install on my test unit to see what happens.

[patrickceg]

I'm the poster who mentioned it in the support forum https://forum.opnsense.org/index.php?topic=11425.0 .

My machine was an upgrade from 18.7 (not upgrade from before the 18.x series): no MTU issues ever on 18.7.

I will add that the "factory reset" if you have the 576 MTU doesn't have any effect.

Let me know if you need a test on my odd laptops, which have an Intel 82579LM and a Realtek 8168. (I mentioned laptops plural because I have two of the same laptop with the Intel LAN and two funny Realtek 8168 ExpressCards.)

[mimugmail]

It's related to an update from 11.1 to 11.2 dhclient like in the pfSense redmine issue described. Initial thought was Realtek, but it isnt

[marjohn56]

So until it's fixed upstream Franco needs to force it it then on install. Is that what they have done over the road?

[mimugmail]

I have no idea if every dhcp Interface config is affected, and if not, what happens when this value is set as default. Let's wait for Franco :)

[marjohn56]

It’s not affecting my test system which is dhcp to the live system and it’s not affecting at least one of my testers who is using dhcp to Sky UK ISP. That said we are both using Qotom with intel i211 NICs.

[patrickceg]

I read the upstream posts, and my reason is indeed the ISP is sending an improper interface MTU of 576 in the DHCP ACK. Unfortunately it looks like a "fix" means seeing if ISP is actually lying to you on the WAN side.

Capturing the DHCP packet I received back from my ISP (Rogers Canada for reference), the MTU in their packet is (incorrectly) 576.

[immanuelfodor]

Can one easily check what mtu setting an ISP provides? I'd happily run a/some bash commands to see mine.

[marjohn56]

I've done a patch for this. basically I've added an option in the dhclient set-up on the WAN page that gives the option to override the ISP supplied MTU buy adding the 'supersede...' line to the client conf file.

Tested and working but as the dev files are not 19.1 then I'll need to post them as patches on my repo. I'll do that in the morning and issue a PR for the dev too,

[fichtner]

Thanks to @marjohn56 we'll fix this in 19.1.1 and change the upgrade path to 19.1.1 directly so major upgrades won't suffer from this. There may be new images coming as well but not as early as 19.1.1.

[Julien-nl]

Dies it mean we can update now to 19.1 ? i was affect but afterwaard i moved back to 18.7

thank you

[fichtner]

Manual workarounds exist. 19.1.1 will fix it, 18.7 upgrade will be moved to 19.1.1 directly skipping the bad 19.1 after confirmation.

[fichtner]

PS: 19.1.1 planned for Tuesday. Upgrade path switch maybe on Wednesday.

[Julien-nl]

@fichtner thank you for your quick reply, i'll wait those couple of days untill the 19.1.1 is released.

[immanuelfodor]

When we will upgrade to 19.1.1 or above, should the advanced config (the workaround, "supersede...") be removed from the GUI by hand?

[marjohn56]

Yes. By default the workaround will be on, you'll need to tick the box to disable it.

[mimugmail]

I think he wonders if you get in trouble when you update and already have supersede option in Advanced

[fichtner]

Try to add it twice in advanced to see what happens.

[yhap2003]

supersede interface-mtu 0

In Interface Config under DHCP, Client Config, Advanced, set:

supersede interface-mtu 0

in Option Modifiers ...

Then try again .. seems to be the solution for this. @fichtner and @marjohn56 know better if there's a chance to get this in per default

Thank you!

This works for my reported issue #3184

[Northguy]

@fichtner

Just wondering...

While researching my own issue #3200 I came across this patch on FreeBSD which is also all about MTU and DHCP https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229432

Shouldn't this then be already in 11.2 by default, or is OPNsense Hardened BSD not on same patch level?

[fichtner]

@Northguy yes, it's in 11.2 and our 19.1

[fichtner]

Sorry, it's not in 11.2 but we have it in 19.1 opnsense/src@4ab5484a8