Route-based IPsec / big packets / fragmentation not working correct

[jroehler23]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[X] I have searched the existing issues and I'm convinced that mine is new.

Describe the bug
I have changed a working policy-based VPN connection to a route-based. After the change ping, normal traffic, DNS, ... everything is working fine, except SIP and bigger packets. The SIP router on the remote end network sends for call signalization a big packet (1500 bytes). In the policy-based configuration everything worked fine, so here is not the problem.

I've checked the firewall log on the remote end and there you can see lots of normal packet directed to the vti interface from the SIP router (192.168.202.202). These packets are reaching me on the local side (192.168.200.20) - fine. But you can also see also some fragmented UDP packets directed to the IPsec interface. This also happens when I try to send pings with 1400 or 1450 bytes.

When I try to ping with DF flag, the ping is rejected. But without it is lost.

I have tried to change the MTU of the vti interface. This has an effect to the DF flag of my ping. But does not help with the fragmentation problem.

To Reproduce
Connect two host over the internet with route based IPsec and try to ping over the VPN with 1450 bytes.

Expected behavior
Packtes should be fragmented and put correct together again.

Screenshots

Relevant log files
If applicable, information from log files supporting your claim.

Additional context
Add any other context about the problem here.

Environment

OPNsense 19.1.9-amd64
FreeBSD 11.2-RELEASE-p10-HBSD
OpenSSL 1.0.2s 28 May 2019

[jroehler23]

[AdSchellevis]

can you try from the firewall itself first and ping both tunnel endpoints, there are many places where this can go wrong in the network.

From my end, using a basic setup sending 1500 bytes:

[jroehler23]

Something strange happens: from my firewall I can ping with 1700 bytes the remote firewall and host in the remote network. But from the remote firewall I can only ping with 1700 bytes the local firewall. Hosts are not pingable.

[jroehler23]

The big pings are all logged with IPsec as interface. The smaller ping to the same destinations are logged with the vti_ interface.

[jroehler23]

Any suggestions where I can dig deeper into?

I have done further test from my local LAN to several remote LANs. All connected via VPN and all gateways are OPNsense machines. In every policy-base connection the big packet 1700 bytes is written als one packet in the log with 1728 bytes as input on the IPsec interface and with 1728 bytes as output on the LAN interface. And I get response from the destination machine.

With all route-based connections I can see the input fragmented on the IPsec interface as two packets and as one packet with 1728 bytes on the LAN interface as output. I don't know what form this packet finally has at the destination machine, but I do not get any answer from there.

I can ping with 1700 all remote OPNsense interfaces on all ends from all LANs and from all OPNsense. But not the LAN behind them in route-based configuration.

What is the difference in the packet forming between route- and policy-based IPsec?

[jroehler23]

When I deactivate the complete packet filtering in the "Advanced Settings" of the firewall, everything works. I have tried to deactivate all set rules, but this did not have the same effect.

Any idea where to look?

[AdSchellevis]

Anything set in Firewall -> Settings -> Normalisation? a copy of the ruleset /tmp/rules.debug (without private addresses) and bit of context about the interfaces might help

[jroehler23]

No normalisation set. I have tried to switch off scrub. But this didn't help with SIP communication.

The rules.debug is nearly 300 lines long. May I send you this via Mail? Too many IP addresses to remove...

[AdSchellevis]

ok, let's focus on settings and scrub first then:

grep -E '^(set|scrub)' /tmp/rules.debug

this is likely a configuration issue, I rather keep the exchanged content online within community support time.

When disabling scrub, did the ping work as expected? there could be multiple issues, so best to step by step here.

[jroehler23]

set limit table-entries 1000000
set ruleset-optimization profile
set optimization normal
set timeout { adaptive.start 0, adaptive.end 0 }
set limit states 805000
set limit src-nodes 805000
set loginterface ix0
set skip on pfsync0
scrub on lo0 all
scrub on igb3 all
scrub on ix0 all
scrub on igb4 all
scrub on igb1_vlan400 all
scrub on igb0 all
scrub on igb2 all
scrub on igb1_vlan40 all
scrub on igb5 all
scrub on ipsec7000 all
scrub on ipsec8000 all

[jroehler23]

Yes, when scrub was disabled I could ping with 1700 through the local firewall

[AdSchellevis]

ok, when considering this situations (with scrub enabled as it is now):

[client A]  ---> [firewall A] --- tunnel --- [firewall B] ---> [client B]

can client A ping large packets to the tunnel IP of Firewall A ?

[jroehler23]

Yes, [client A] can ping [firewall A] end of the tunnel with 1700

And I can see the incoming icmp packet in the firewall log of firewall b end of the tunnel. But client a gets a timeout and no response.

[AdSchellevis]

can you share ifconfig output (with obfuscated addresses)?

[jroehler23]

VTI of firewall A

ipsec8000: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
        tunnel inet XXX.59 --> YYY
        inet6 fe80::ae1f:6bff:fe69:f6ae%ipsec8000 prefixlen 64 scopeid 0x13
        inet 10.0.202.1 --> 10.0.202.2 netmask 0xffffffff
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        reqid: 8000
        groups: ipsec

[AdSchellevis]

from all related interfaces (lan, wan, tunnel)

[jroehler23]

LAN:

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether ac:1f:6b:69:f6:ae
        hwaddr ac:1f:6b:69:f6:ae
        inet 192.168.200.201 netmask 0xffffff00 broadcast 192.168.200.255
        inet6 fe80::ae1f:6bff:fe69:f6ae%ix0 prefixlen 64 scopeid 0x1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active

WAN:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether ac:1f:6b:69:f3:86
        hwaddr ac:1f:6b:69:f3:86
        inet XXX.59 netmask 0xfffffff8 broadcast XXX.63
        inet XXX.58 netmask 0xfffffff8 broadcast XXX.63
        inet6 XXX%igb0 prefixlen 64 scopeid 0x3
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

[AdSchellevis]

the 1400 on ipsec8000 is very likely an issue now without mss clamping configured

[jroehler23]

These were the standard values from the system. Any suggestions for MTU and MSS fix?

[AdSchellevis]

can you try to set mss on ipsec8000 to anything lower then 1400?

[mimugmail]

MSS only works for TCP, normally MTU should be received from parent interface, but no idea how to realize.

[AdSchellevis]

@mimugmail you're right, my mistake. just out of curiosity, could you reproduce the ping large packet issue on your end? Personally I would expect that switching off scrubbing should keep the normal behaviour. Maybe the better question to @jroehler23 is if the large packet ping works when scrub is disabled.

[mimugmail]

I can test next week. Normally, when ICMP is allowed in every direction correctly, PMTU discovery should take care of (when DF bit not set)

[AdSchellevis]

ok, thanks in advance.

[mimugmail]

#3327 could be related. Worth a test to disable pf completely and check If it works. Perhaps some auto rules etc. prohibit ICMP messages (or sysctl)

[AdSchellevis]

I think @jroehler23 mentioned it worked with pf disabled, which leaves scrubbing or filtering as most likely candidates. The new version (19.7) should also show these internal rules in the overview, for now they should generate log entries when logging is enabled if I'm not mistaken.

[jroehler23]

I can confirm that large ping and the SIP signalization from the remote LAN is working when pf is deactivated.

With scrubbing disabled I can ping with large packet, but not really stable. When I try to SIP call the remote location from outside (public phone number), neither the ping is working the same time I am calling, nor the SIP is passed from the remote LAN to the telephone server in the local LAN. But it seams to use and block to VPN tunnel with its packets.

[jroehler23]

New experience:
I have checked once again the VPN configuration at the local OPNsense and I have found one policy-base tunnel configuration is still active. After I deactivated this one, I have only two route-based IPsec tunnels active.

After a reboot the large pings are working! BUT: not from the beginning! I have to ping at least once with a normal size ping (e.g. 32 bytes). When I try to ping direct with 1700 bytes it is still not working.

[jroehler23]

And I have to keep pinging with small packets. I have waited now for 40 minutes and cannot ping with 1700 bytes. I have to ping first with a small packet, then the big ping is working again.

The tunnel is up and routed all the time.

It is getting stranger and stranger...

[jroehler23]

Next watching: after 24 hours the LAN at the remote site is not reachable any more. Even via ssh connection to the remote OPNsense - no ping to any host in the remote LAN is possible. I have tried to reload services or to deactivate the complete pf. No change.

But the tunnel is still active, routes are set and both endpoints are reachable. Also the LAN address of the remote OPNsense is reachable. Clients in the WLan or other networks there are reachable via ping from the remote OPNsense.

After a reboot of the remote sense, the remote LAN is reachable again. Any hint where to look, to get rid of this behavior? I am confused...

[AdSchellevis]

Unfortunately not, it could be some other network or configuration issue, it's a complex world. Within the boundaries of community support I'm afraid I can't help you at the moment.

If someone can isolate a specific issue which we can reproduce on our end, I'll gladly take a look.

[GurliGebis]

I'm seeing a similar problem with my roadwarrior setup.
After a while, no traffic is able to go through the tunnel, but disconnecting and reconnecting works, until it stops again after a while, and the cycle repeats.

[jroehler23]

So this evening I started a test with a complete new installation and did the complete configuration by hand from the scratch. A lot of stupid work, but it was worth it: now the fragmentation issue is gone. I have absolute no idea why this happend.

From my side we can close this.

[xEn64]

Same here, no Idea what can help.
Switching to policy-based setup might help hopefully.

[mimugmail]

On 23.7? Freebsd 13.2 adds PMTU support for ipsec, maybe it helps to frag packets before loosing them

[xEn64]

Hi Michael,

its 23.1.6.
I will try to upgrade, keep fingers crossed.

[xEn64]

upgraded to 23.7, behaviour still same.

[AdSchellevis]

https://docs.opnsense.org/manual/vpnet.html#path-mtu-discovery