IPsec traffic stalling on 20.7

[fraenki]

Describe the bug
After upgrading from 20.1.4 to 20.7.2 IPsec phase 2 tunnels will randomly stall (IKEv1, mode tunnel IPv4). Only restarting strongswan seems to fix this issue (temporarely).

Several other people have reported the same issue on 20.7. More reports, details, hardware specs, etc. are available in the forums:
https://forum.opnsense.org/index.php?topic=18918.0

To Reproduce
Steps to reproduce the behavior:

1. Establish an IPsec tunnel between two physical OPNsense 20.7 boxes
2. It was better reproducable with IKEv2 (at least in my case)
3. Send enough traffic through the tunnel (200-400 MB in my case)
4. The tunnel completely stalls, no traffic gets through

Downgrading to OPNsense 20.1 immediately fixes this issue (according to other reporters in the forum).

However, @mimugmail tried to reproduce this issue on OPNsense virtual machines, but did not succeed unfortunately. So this issue is probably related to specific vendors/hardware or drivers. Until now all reporters were using Intel NICs.

Expected behavior
Traffic should get through the IPsec tunnel, it should not stall after transferring a specific amount of data.

Relevant log files
When using IKEv2 strongswan is actually able to detect the problem and restart the SA after a while:

Sep  8 17:03:05 charon[62985]: 05[IKE] <con5|1> giving up after 10 path probings
Sep  8 17:03:05 charon[62985]: 05[IKE] <con5|1> restarting CHILD_SA con5 

Note that with IKEv1 strongswan will still think that the tunnel is active and alive, so with IKEv1 you need to restart strongswan.
Of course, in both cases this does not really help, because the tunnel will get stuck again after very little traffic.

Additional context
I've manually built packages for strongswan 5.8.3 and 5.9.0 and tested them on OPNsense 20.7.2, but this did not change anything.

Other reporters and myself tested several different IPsec settings, but this did not change anything (details of the various settings can be found in the Forum). Changing the options in Interfaces: Settings did not help either.

An interesting observation: when one box was still running OPNsense 20.1 and only the other box was on OPNsense 20.7, this issue did not occur at all. Only after upgrading both boxes to 20.7 we first experienced this issue. Also this issue does not IPsec connections between OPNsense and other firewalls – those are 100% stable. Only OPNsense-to-OPNsense connections are affected.

Environment
My setup:
OPNsense 20.7.2 / 20.7.3 (amd64, OpenSSL).
Supermicro A2SDi-2C-HLN4F (Intel C3338 CPU, Intel C3000 NIC)

Other reporters:
Landitec scope7-1510 (Intel Atom C3558)
Supermicro A2SDi-4C-HLN4F (almost the same as mine)

[sjansen1]

I have the same issues after Updating to 20.7 between two OPNSense systems (KVM + a SuperMicro A2SDi-H-TP4F) using IKEv1 and IKEv2. Tunnel. Connection stall after around 1-2 minutes and i can see no more IPSEC Traffic on the WAN interface. Restarting the IPSEC connection bring it alive again for 1-2 minutes. Another Tunnel between OPNsense (A2SDi-H-TP4F) and Sophos UTM using IKEv1 works without stalls.

I now switched to Zerotier to connect this two OPNsense boxed and it works fine without any stalls.

[mimugmail]

@sjansen1 which errors do you have in log?

[sjansen1]

@sjansen1 which errors do you have in log?

Sadly i cannot provide more information because this issue startet right after release of 20.7 and i had to use an alternative to IPSEC. I had no errors in the main log, ipsec was shown as "up", connection went dead after 1-2 minutes.

Yes i know this is not really helpful, this issue startet right after updating both boxed to 20.7 at the same time without any other modification. Reboots or recreating the whole ipsec config didnt help.

[mimugmail]

I had a call to Landitec and they told me that they dont have any feedback from customers about such a problem and they sold hundreds of those 1510's the last two months. Probably they'll send me two devices so I can try to reproduce.

[mimugmail]

I set up a real world VPN mesh:

Unit 1: Supermicro Cluster in Munich, 20.7.2
Unit 2: Jetway Appliance (Core i5) in Berlin, 20.7.3
Unit 3: VM on IONOS, KVM, Frankfurt, 20.7.3
Unit 4: VM on VMware, Munich, 20.7.3
Unit 5: Jetway Appliance (Celeron J3600), Munich, 20.7.2

They are full-meshed, so every device has 4 active VPNs. I'll watch the traffic and logs.
On Monday I'll receive a 1510 from Landitec and install it also to the mesh

[mimugmail]

OK, it took me quite some time to set this up. The Supermicro device has problems pinging via VPN when mangling source address via ping, all other devices don't have these and when I use a device behind the Supermirco everything is fine. Every VPN to every neighbor is currently up and stable.

I need some more facts:

1. Does traffic in both directions stall, so does a ping from A to B stall but B to A works?
2. You experience this always with traffic going THROUGH the firewall, correct?

[mimugmail]

VPN still stable, if anyone wants to join this mesh for testing just ping me.

[mimugmail]

Just added a scope7 1510 to the VPN and pushed 8GB of traffic over VPN, no problems.
On this test I used default settings and the scope is behind NAT, other side is respond-only (IKEv2).

[mimugmail]

VPN still stable between all hosts, I'm treating this as "works-for-me".

These were the hardware details of some appliances:

https://bsd-hardware.info/?probe=8088bf7e60
https://bsd-hardware.info/?probe=630ace9406

[fraenki]

I'll perform new tests when OPNsense 20.7.5 is released.

[mimugmail]

I'm still testing with equipment and other ppl affected.
For one case it was reproducable that a new install of 20.7 (compared to an update of 20.1) fixed the issue.

Any chance to test this?

[fraenki]

Not in my case, these are all-remote machines, I can't risk a reinstallation – the location is 6 hours away.

[fraenki]

For one case it was reproducable that a new install of 20.7 (compared to an update of 20.1) fixed the issue.

I took the risk and reinstalled the primary firewall. However, even after completing this procedure IPsec traffic is still stalling.

However, I've noticed one thing: in order to reinstall firewall A I had to perform a (CARP) failover to firewall B. When firewall B was the master, then the IPsec traffic was not stalling.

That's extremely strange, because they use the same hardware (Supermicro A2SDi-2C-HLN4F) that was purchased on the same date.

EDIT: In a later test firewall B had the same IPsec issues, so this was an incorrect conclusion.

I have no idea how to proceed.

[fraenki]

We have been able to workaround this issue by changing the following setting...
System: Settings: Miscellaneous -> Hardware acceleration
...from AES-NI CPU-based to none. (Be sure to reboot the firewall afterwards.)

It looks like this is either a FreeBSD/HardenedBSD 12.x bug or related to my hardware platform (Intel Atom C3338, "Denverton" SoC) – in any case it was triggered by the upgrade from FreeBSD 11.2 to 12.1.