Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PPPoE MTU #3328

Closed
djhau opened this issue Mar 13, 2019 · 22 comments
Closed

PPPoE MTU #3328

djhau opened this issue Mar 13, 2019 · 22 comments
Labels
support Community support

Comments

@djhau
Copy link

djhau commented Mar 13, 2019

I have noticed recently that with OPNSense I don't get ICMP packet too big messages when sending DF marked packets out the WAN interface.

Setup as follows:

Screen Shot 2019-03-14 at 5 50 10 am

Client on LAN side sending ping tests out via PPPoE interface.

*** Firewall passing traffic as expected at 1492 MTU **
dhunt@Debian:~$ ping -s 1464 -M do 150.101.32.161
PING 150.101.32.161 (150.101.32.161) 1464(1492) bytes of data.
1472 bytes from 150.101.32.161: icmp_req=1 ttl=249 time=6.65 ms
1472 bytes from 150.101.32.161: icmp_req=2 ttl=249 time=9.91 ms
1472 bytes from 150.101.32.161: icmp_req=3 ttl=249 time=6.70 ms
1472 bytes from 150.101.32.161: icmp_req=4 ttl=249 time=6.72 ms
1472 bytes from 150.101.32.161: icmp_req=5 ttl=249 time=7.14 ms
1472 bytes from 150.101.32.161: icmp_req=6 ttl=249 time=6.79 ms

*** No ICMP packet too big messages received for packets that are too big to pass***
dhunt@Debian:~$ ping -s 1465 -M do 150.101.32.161
PING 150.101.32.161 (150.101.32.161) 1465(1493) bytes of data.
^C
--- 150.101.32.161 ping statistics ---
14 packets transmitted, 0 received, 100% packet loss, time 13103ms

dhunt@Debian:~$

Expect to get ICMP messages from OPNSense back to client reflecting correct MTU of the interface.

OPNSense 19.1.3
VMWare ESXI 6.7.0
VMSwitch and interface set to MTU 1500
CPU E5-2430 0 @ 2.20GHz
NIC - Broadcom BCM 5720

@AdSchellevis
Copy link
Member

which icmp response would that be for IPv4? for IPv6 there's a type response containing it's mtu (https://tools.ietf.org/html/rfc4443#section-3.2) used for PMTU, I don't know of a version for IPv4.

@AdSchellevis AdSchellevis added the support Community support label Mar 13, 2019
@djhau
Copy link
Author

djhau commented Mar 13, 2019

As per my example IPv4 - https://tools.ietf.org/html/rfc1191

This is the ICMP Type it should send back - https://tools.ietf.org/html/rfc792
4 = fragmentation needed and DF set;

@AdSchellevis
Copy link
Member

maybe this https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220351 give you some pointers. I'm not entirely sure this can work the way you expect.

@djhau
Copy link
Author

djhau commented Mar 15, 2019

I am pretty sure that PFSense does this, I will spin up a new instance and try. This in all honesty is a basic network mechanism.

@djhau
Copy link
Author

djhau commented Mar 23, 2019

Verified that this works as expected in pFsense 2.4.4p1 (No other change to my network other than the VM running pfSense instead of OPNSene).

dhunt@Debian:~$ ping -s 1464 -M do 192.231.203.132
PING 192.231.203.132 (192.231.203.132) 1464(1492) bytes of data.
1472 bytes from 192.231.203.132: icmp_req=1 ttl=57 time=8.56 ms
1472 bytes from 192.231.203.132: icmp_req=2 ttl=57 time=8.77 ms
1472 bytes from 192.231.203.132: icmp_req=3 ttl=57 time=8.84 ms
1472 bytes from 192.231.203.132: icmp_req=4 ttl=57 time=8.56 ms

dhunt@Debian:~$ ping -s 1465 -M do 192.231.203.132
PING 192.231.203.132 (192.231.203.132) 1465(1493) bytes of data.
From 192.168.201.201 icmp_seq=1 Frag needed and DF set (mtu = 1492)
From 192.168.201.201 icmp_seq=1 Frag needed and DF set (mtu = 1492)
From 192.168.201.201 icmp_seq=1 Frag needed and DF set (mtu = 1492)
From 192.168.201.201 icmp_seq=1 Frag needed and DF set (mtu = 1492)
From 192.168.201.201 icmp_seq=1 Frag needed and DF set (mtu = 1492)
From 192.168.201.201 icmp_seq=1 Frag needed and DF set (mtu = 1492)

13:59:17.263941 IP 192.168.201.201 > 192.231.203.2: ICMP echo request, id 15236, seq 1, length 1472
13:59:17.274151 IP 192.231.203.2 > 192.168.201.201: ICMP echo reply, id 15236, seq 1, length 1472
13:59:18.265853 IP 192.168.201.201 > 192.231.203.2: ICMP echo request, id 15236, seq 2, length 1472
13:59:18.276537 IP 192.231.203.2 > 192.168.201.201: ICMP echo reply, id 15236, seq 2, length 1472
13:59:19.266977 IP 192.168.201.201 > 192.231.203.2: ICMP echo request, id 15236, seq 3, length 1472
13:59:19.276947 IP 192.231.203.2 > 192.168.201.201: ICMP echo reply, id 15236, seq 3, length 1472
13:59:20.268521 IP 192.168.201.201 > 192.231.203.2: ICMP echo request, id 15236, seq 4, length 1472
13:59:20.279004 IP 192.231.203.2 > 192.168.201.201: ICMP echo reply, id 15236, seq 4, length 1472
13:59:29.832532 IP 192.168.201.201 > 192.231.203.2: ICMP echo request, id 15237, seq 1, length 1473
13:59:29.832906 IP 192.168.201.222 > 192.168.201.201: ICMP 192.231.203.2 unreachable - need to frag (mtu 1492), length 36

Note the ICMP message when I up the ping size to 1465 .. and the tcpdump output reflecting the message I get back from my pfSense firewall VM (192.168.201.222).

Can you fix this on OPNSense?

@AdSchellevis
Copy link
Member

can you compare mtu sizes on all interfaces on both systems and compare the output of grep -r scrub /tmp/rules.debug? It's likely a setting, pfSense sometimes sets values different then configured, which we don't do for obvious reasons.

@djhau
Copy link
Author

djhau commented Mar 23, 2019

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:ad:14:b9
hwaddr 00:0c:29:ad:14:b9
inet xxx.xxx.xxx.xxx netmask 0xffffff00 broadcast 192.168.201.255
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
inet6 fe80::1:1%vmx0 prefixlen 64 scopeid 0x1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active

vmx3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:ad:14:af
hwaddr 00:0c:29:ad:14:af
inet6 fe80::20c:29ff:fead:14af%vmx3 prefixlen 64 scopeid 0x4
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
pppoe1: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
inet6 fe80::20c:29ff:fead:14b9%pppoe1 prefixlen 64 scopeid 0x9
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64 autoconf
inet xxx.xxx.xxx.xxx --> 10.20.25.153 netmask 0xffffffff
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

[2.4.4-RELEASE][dhunt@fw1.local]/home/dhunt: grep -r scrub /tmp/rules.debug
scrub on $NBN all fragment reassemble
scrub on $LAN all fragment reassemble
scrub on $AIRSTREAM_OPT2 all fragment reassemble


vmx1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:13:2d:c3
hwaddr 00:0c:29:13:2d:c3
inet xxx.xxx.xxx.xxx netmask 0xffffff00 broadcast 192.168.201.255
inet6 fe80::20c:29ff:fe13:2dc3%vmx1 prefixlen 64 scopeid 0x2
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:13:2d:e1
hwaddr 00:0c:29:13:2d:e1
inet6 fe80::20c:29ff:fe13:2de1%vmx0 prefixlen 64 scopeid 0x1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
inet6 fe80::20c:29ff:fe13:2de1%pppoe0 prefixlen 64 scopeid 0x9
inet6 fe80::20c:29ff:fe13:2dd7%pppoe0 prefixlen 64 scopeid 0x9
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64 autoconf
inet xxx.xxx.xxx.xxx --> 10.20.21.38 netmask 0xffffffff
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

dhunt@fw1:~ % grep -r scrub /tmp/rules.debug
scrub on lo0 all
scrub on vmx3 all max-mss 1420
scrub on pppoe0 all max-mss 1420
scrub on vmx1 all max-mss 1420
dhunt@fw1:~ %

@AdSchellevis
Copy link
Member

a bit of context might probably help, settings are obviously differ between both, but I'm not going to guess what is what here. you can configure scrubbing at Firewall -> Settings -> Normalisation

I don't expect anything needs fixing other then using the proper configuration, but if you can tell us what we should fix where, I'll happily look into it. I can't spare more time in figuring our why your specific setup doesn't work.

Same message in similar circumstances (lan package doesn't fit wan's size and not using mss) from my machine using OPNsense:

36 bytes from opnsense.localdomain (x.x.x.x): frag needed and DF set (MTU 1450)

@djhau
Copy link
Author

djhau commented Mar 23, 2019

Context:
pFSense config:

Internal interface where client is pinging from MTU 1500 by default for Ethernet:
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:ad:14:b9
hwaddr 00:0c:29:ad:14:b9
inet xxx.xxx.xxx.xxx netmask 0xffffff00 broadcast 192.168.201.255
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
inet6 fe80::1:1%vmx0 prefixlen 64 scopeid 0x1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active

External interface supporting PPPoE session:
vmx3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=60009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0c:29:ad:14:af
hwaddr 00:0c:29:ad:14:af
inet6 fe80::20c:29ff:fead:14af%vmx3 prefixlen 64 scopeid 0x4
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active

Routed interface where ping target is with MTU 1492 set by default:
pppoe1: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
inet6 fe80::20c:29ff:fead:14b9%pppoe1 prefixlen 64 scopeid 0x9
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64 autoconf
inet xxx.xxx.xxx.xxx --> 10.20.25.153 netmask 0xffffffff
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Output as requested from pfSense
[2.4.4-RELEASE][dhunt@fw1.local]/home/dhunt: grep -r scrub /tmp/rules.debug
scrub on $NBN all fragment reassemble
scrub on $LAN all fragment reassemble
scrub on $AIRSTREAM_OPT2 all fragment reassemble


OPNSense config:

vmx1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:13:2d:c3
hwaddr 00:0c:29:13:2d:c3
inet xxx.xxx.xxx.xxx netmask 0xffffff00 broadcast 192.168.201.255
inet6 fe80::20c:29ff:fe13:2dc3%vmx1 prefixlen 64 scopeid 0x2
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active

Internal interface where client is pinging from MTU 1500 by default for Ethernet:
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:13:2d:e1
hwaddr 00:0c:29:13:2d:e1
inet6 fe80::20c:29ff:fe13:2de1%vmx0 prefixlen 64 scopeid 0x1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active

Routed interface where ping target is with MTU 1492 set by default:
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
inet6 fe80::20c:29ff:fe13:2de1%pppoe0 prefixlen 64 scopeid 0x9
inet6 fe80::20c:29ff:fe13:2dd7%pppoe0 prefixlen 64 scopeid 0x9
inet6 2001:xxx:xxx:xxx:xxx prefixlen 64 autoconf
inet xxx.xxx.xxx.xxx --> 10.20.21.38 netmask 0xffffffff
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Output as requested from OPNSense - this MSS is not set by me!
dhunt@fw1:~ % grep -r scrub /tmp/rules.debug
scrub on lo0 all
scrub on vmx3 all max-mss 1420
scrub on pppoe0 all max-mss 1420
scrub on vmx1 all max-mss 1420
dhunt@fw1:~ %

What I expect to be fixed is that when the system assumes PPPoE is running at 1492 (which is totally fair and what pFSense does), that once the LAN side client attempt to send a packet out the PPPoE interface the system responds back with the ICMP message (too big) I have presented in the packet trace previously by default, i.e it is part of the default configuration of the firewall, as it stands this is a broken implementation. I have no idea where the 1420 MSS came from either, I have 1460 set in my config.

@taospartan
Copy link

Hi djhau,
Did you get a normalisation config working? ICMP replies do look like they are not occurring. I’m having the same fault with pppoe and without.

@djhau
Copy link
Author

djhau commented May 10, 2019

No this never got resolved! I gave up fighting this and went back to pFSense, which "just works".

@taospartan
Copy link

taospartan commented May 10, 2019 via email

@mimugmail
Copy link
Member

https://forum.opnsense.org/index.php?topic=11993.msg58684;topicseen#msg58684

Is this the same issue? I can have a look next week in a quite moment.

@taospartan
Copy link

taospartan commented May 11, 2019 via email

@AdSchellevis
Copy link
Member

@mimugmail it's quite easy to test/check, the "this MSS is not set by me" is explained in the help text at the interface (value set minus 40), fiddling in detail or totally disabling scrubbing can be done from Firewall -> Settings -> Normalisation, the fragment reassemble is set by default if I'm not mistaken.

core/src/etc/inc/filter.inc

Lines 579 to 595 in ab75fbb

if (empty($config['system']['scrub_interface_disable'])) {
foreach ($FilterIflist as $scrubcfg) {
if (isset($scrubcfg['virtual']) || empty($scrubcfg['descr'])) {
continue;
}
$mssclamp = '';
if (!empty($scrubcfg['mss']) && is_numeric($scrubcfg['mss']) &&
!in_array($scrubcfg['if'], array('pppoe', 'pptp', 'l2tp'))) {
$mssclamp = 'max-mss ' . (intval($scrubcfg['mss'] - 40));
}
$scrubnodf = !empty($config['system']['scrubnodf']) ? 'no-df' : '';
$scrubrnid = !empty($config['system']['scrubrnid']) ? 'random-id' : '';
$scrubrules .= "scrub on {$scrubcfg['if']} all {$scrubnodf} {$scrubrnid} {$mssclamp}\n";
}
}

It just works in this case means, defaults are different and settlings might be too in this case.

@logicethos
Copy link

Don't know if this is relevant, but after doing to 19.1.9, PPPoE Jumbo Frames seem to be broken. 1508 (1500).

@logicethos
Copy link

I fixed mine by un-ticking "Prevent interface removal"

@AdSchellevis
Copy link
Member

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

@monogatron
Copy link

monogatron commented Mar 10, 2021

Can this issue please reopened? We have the same problem, that no icmp packet too big messages is send by opnsense if the mtu of a client in the network is bigger than 1492. So I have to set the mtu at every client manually.

@pallebone
Copy link

pallebone commented Nov 24, 2021

I have same issue as well on OPNsense 21.7.5-amd64. Disabling interface scrub did not help.

@fichtner
Copy link
Member

@pallebone Hi Peter, should be the same as #5285 -- can you check?

@pallebone
Copy link

pallebone commented Nov 24, 2021

Hi there,

I am also affected by that issue, however I cant seem to fix it by following the instructions in #5285. It is not clear to me why editing the parent interface mtu worked for him but not for me. I am continuing to check things but havent found a solution yet. They do seem related this issue and that one.

Pete

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
support Community support
Development

No branches or pull requests

7 participants