WireGuard endpoints no Internet access unless service manually restarted (NAT not being applied)

[Kinerg]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [*] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [*] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

After OPNsense boot, WireGuard endpoints don't have Internet access. Traffic passes through OPNsense out to WAN interface and nothing appears to be blocked, but it seems NAT is not being applied so nothing comes back from the Internet. Last known working OPNsense version was 23.1.11_2. Updated to 23.7.5 and noticed the issue.

Forum topic about the issue. At least one other user reported what appears to be the same issue.

To Reproduce

Steps to reproduce the behavior:

1. Restart OPNsense
2. Connect WireGuard endpoint
3. Try to access Internet or ping any domain/IP, no answer
4. Restart WireGuard service
5. Everything works

Expected behavior

WireGuard clients should have Internet access after OPNsense boot.

Relevant log files

OPNsense fresh boot, before manual WG restart:

wan			2023-10-01T15:43:55	10.101.80.1	1.1.1.1	icmp	let out anything from firewall host itself
WGxInternet		2023-10-01T15:43:55	10.101.80.1	1.1.1.1	icmp	Allow WGxInternet to Internet

After manual WG restart:

wan			2023-10-01T15:44:56	192.168.61.10	1.1.1.1	icmp	let out anything from firewall host itself (force gw)
WGxInternet		2023-10-01T15:44:56	10.101.80.1	1.1.1.1	icmp	Allow WGxInternet to Internet

10.101.80.1 -> WireGuard endpoint IP
192.168.61.10 -> WAN interface IP (upstream gateway 192.168.61.1)

Additional context

Communication between WireGuard endpoints and OPNsense works without issue. It is only when WG traffic has to go out through the WAN interface that the issue occurs.

Environment

OPNsense 23.7.5-amd64
FreeBSD 13.2-RELEASE-p3
OpenSSL 1.1.1w 11 Sep 2023

[AdSchellevis]

Could be a state issue, assuming between restarts of Wireguard nothing changes in the ruleset itself (you can check that by comparing the contents of /tmp/rules.debug before and after the reload). If you kill the state (icmp 10.101.80.1 --> 1.1.1.1) and restart the ping, does that lead to the same result?

[Kinerg]

I've compared the contents of /tmp/rules.debug before and after the reload. These four lines are added after reload and that is the only change:

nat on vtnet1 inet from (wg2:network) to any port 500 -> (vtnet1:0) static-port # Automatic outbound rule
nat on vtnet1 inet from (wg1:network) to any port 500 -> (vtnet1:0) static-port # Automatic outbound rule

nat on vtnet1 inet from (wg2:network) to any -> (vtnet1:0) port 1024:65535 # Automatic outbound rule
nat on vtnet1 inet from (wg1:network) to any -> (vtnet1:0) port 1024:65535 # Automatic outbound rule

Killing the states had no effect.

[AdSchellevis]

ok, thanks, that might be a race condition then. For openvpn we construct the [tun] devices in an earlier stage if I'm not mistaken. Is your problem also fixed by restarting the filter using /usr/local/etc/rc.filter_configure?

[Kinerg]

Yes, restarting the filter fixes the issue.

[AdSchellevis]

ok, thanks, that offers some direction on where to look.

[AdSchellevis]

this opnsense/plugins@a7a94cc should fix the issue.

[Kinerg]

Great! Should I run # opnsense-patch a7a94cc?

[AdSchellevis]

@Kinerg I think you need to select the plugins in the command (it's not part of core yet), something like the following likely does the trick:

opnsense-patch -c plugins a7a94cc

(but I haven't tried it on my end)

[Kinerg]

Then I'll wait till the patch gets integrated. Thank you for the quick support!

[fichtner]

opnsense-patch is risk free in this regard. If not you can always use opnsense-revert.

[Kinerg]

I've tried and got the following:

# opnsense-patch -c plugins a7a94cc
Fetched a7a94cc via https://github.com/opnsense/plugins
No such line 226 in input file, ignoring
1 out of 1 hunks failed while patching opnsense/scripts/Wireguard/wg-service-control.php

[fichtner]

Ok, I think the CARP VHID feature is interfering. But as I said it’s risk free as it simply won’t modify files in that case.

[TheekshanaA]

I am experiencing the same issue on latest opnsense

[AdSchellevis]

@TheekshanaA you can either wait for the next update or build the plugin locally. Reporting what we already know (there's something to fix) when it has already been pushed to git is rather unproductive.

[Westie]

I've put together a janky but lightweight workaround here. The workaround essentially attempts to reload all rules after a minute has passed, and works rather well on my environment.

[Kinerg]

@AdSchellevis @fichtner
So, 24.1.1 is here and I still have the same issue. This gets added to WireGuard log after every boot:

/usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.101.1.2/30' -interface 'wg2'' returned exit code '1', the output was ''

Restarting WireGuard service fixes the issue.