Port forwarding via wireguard interface not working

[goofus]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[x] I have searched the existing issues and I'm convinced that mine is new.

Describe the bug
Port forwarding doesn't work through wireguard interface to lan.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce
Steps to reproduce the behavior:

1. Establish wireguard connection
2. Forward a tcp port from the wireguard(WAN) network to LAN network
3. Open port with ncat on host in LAN
4. Try to connect to forwarded port from WAN
5. Follow packets with tcpdump on OPNsense firewall

Expected behavior
Establishing a tcp connection between WAN host (xxx.de) via wireguard interface (wg0) to LAN host.

Screenshots

https://i.imgur.com/x9hBagG.png
https://i.imgur.com/dJs9l38.png
https://i.imgur.com/Ylx9J3L.png

Relevant log files
tcpdump on LAN interface on OPNsense

19:28:57.774733 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79282081 ecr 0,nop,wscale 7], length 0
19:28:57.775498 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133438719 ecr 79282081,nop,wscale 7], length 0
19:28:58.776502 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79283084 ecr 0,nop,wscale 7], length 0
19:28:58.777248 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133439721 ecr 79282081,nop,wscale 7], length 0
19:28:59.779374 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133440723 ecr 79282081,nop,wscale 7], length 0
19:29:00.792129 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79285100 ecr 0,nop,wscale 7], length 0
19:29:00.792870 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133441737 ecr 79282081,nop,wscale 7], length 0
19:29:02.979382 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133443923 ecr 79282081,nop,wscale 7], length 0
19:29:04.888799 IP xxx.de.23023 > 192.168.20.101.11526: Flags [S], seq 1802740233, win 64240, options [mss 1380,sackOK,TS val 79289196 ecr 0,nop,wscale 7], length 0
19:29:04.888967 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133445833 ecr 79282081,nop,wscale 7], length 0
19:29:08.952726 IP 192.168.20.101.11526 > xxx.de.23023: Flags [S.], seq 1252543376, ack 1802740234, win 65160, options [mss 1460,sackOK,TS val 1133449897 ecr 79282081,nop,wscale 7], length 00

tcpdump on wg0 interface on OPNsense

19:34:19.395412 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79603694 ecr 0,nop,wscale 7], length 0
19:34:20.408152 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79604707 ecr 0,nop,wscale 7], length 0
19:34:22.424482 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79606723 ecr 0,nop,wscale 7], length 0
19:34:26.682679 IP xxx.de.23023 > 10.65.68.45.11526: Flags [S], seq 2533116111, win 64240, options [mss 1380,sackOK,TS val 79610980 ecr 0,nop,wscale 7], length 0`

SYN ACK doesn't get forwarded

Environment

OPNsense 20.7.3-amd64

Forum Report
https://forum.opnsense.org/index.php?topic=18013
https://forum.opnsense.org/index.php?topic=18062
https://forum.opnsense.org/index.php?topic=19409
https://forum.opnsense.org/index.php?topic=17973

[Tuxist]

same issue on my system

[Optic00]

my 2 cents: its not just port forwarding via WireGuard, its generally all VIP that don't behave like a physical interface. The same problem is with GRE and OpenVPN Tunnels. Last time I researched this I suspected reply-to to be the problem. What I had working was NAT through WireGuard tunnel after hours fiddling around with this. I would like to use a WireGuard Interface with a public ipv4 and run HA proxy or just NAT it but it doesn't work.

[rt-outofservice]

Same issue on the latest 20.7.5. Port forwarding through OpenVPN tunnel in exactly same configuration works just fine, so this is Wireguard specific.

[mimugmail]

You can play around with reply-to Rules to figure it out. I dont have a lab to test

[amonhk]

Same issue on WireGuard Port forwarding, in my opinion the problem is in the routes, the traffic does not return for the same input route.
Forcing a static route works but doesn't fix the problem.

[mimugmail]

Correct, there is a mismatch in the reply-to rules somewhere

[calvinbui]

Port forwarding works for me on OpenVPN but not WireGuard.

[TheLinuxGuy]

I am also trying to use wireguard + port forwarding and a public IP from the other endpoint and it is not working. Seems related to routing table.

I wrote about it here before I found this bug: https://forum.opnsense.org/index.php?topic=21006.0

[TheLinuxGuy]

You can play around with reply-to Rules to figure it out. I dont have a lab to test

Any links / references on how to do these test or play around / sample commands etc?

[H4R0]

@TheLinuxGuy don't waste time on it, I tried everything already.

pfsense-devel now has wireguard using the kernel implementation, opnsense will probably follow with 21.1.x/21.7

The kernel implementation will boost performance again and hopefully fix the routing problem.

[fichtner]

I'm curious why should it solve routing issues? Kernel WireGuard has everyone giddy, but it won't live up to the hype for sure.

[akront]

not sure if this is relevant, but managed to make outbound NAT and port forwarding work on WG after weeks of trial and error

the key component missing was WG only works on default WAN IP not WAN VIPs that you may have, and also you need outbound NAT rules for the port forward traffic coming in and going over the tunnel and also manual MSS value set for all to work properly

more references here https://forum.opnsense.org/index.php?topic=21445.15

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

[H4R0]

From the forum discussion port forwarding works with the kernel implementation.

The package is based on the freebsd rework by jason and not the horrible netgate implementation.

Install with (experimental and still in work):
pkg install wireguard-kmod

[fichtner]

Please only install wireguard-kmod instead to avoid future side effects with the wireguard meta package.

[frankyw]

Can still not get this to work, no matter what I try.

[H4R0]

Can still not get this to work, no matter what I try.

I did a quick test and it indeed still routes over wan.

[trunet]

I'm also unable to make port forwarding works. This ticket should be reopened, if not I can open a new one.

In my case, I installed wireguard-kmod, rebooted. tcpdumps below, you can see packet arriving fine on wg1 but being replied back on wan directly.

OpnSense wg1 tcpdump:

13:12:46.987457 IP [REDACTED_PUBLIC_IP].46256 > 10.13.128.89.10000: Flags [S], seq 3380801657, win 29200, options [mss 1380,sackOK,TS val 3306454498 ecr 0,nop,wscale 7], length 0

OpnSense ix1_vlan34 tcpdump (my WAN interface):

13:12:46.987713 IP 10.13.128.89.10000 > [REDACTED_PUBLIC_IP].46256: Flags [S.], seq 3870681174, ack 3380801658, win 65160, options [mss 1460,sackOK,TS val 3841074193 ecr 3306454498,nop,wscale 7], length 0
13:12:46.987814 IP 10.13.128.89.10000 > [REDACTED_PUBLIC_IP].46256: Flags [S.], seq 3870681174, ack 3380801658, win 65160, options [mss 1460,sackOK,TS val 3841074193 ecr 3306454498,nop,wscale 7], length 0
...... more TCP SYN/ACK retries

[fichtner]

It would be better to take this to the forum where people can actually tell you it works and help diagnose. Also, kmod is unsupported at this point.

Cheers,
Franco

[amonhk]

I noticed this commit 286000d

Firewall - allow manual reply-to configuration

Refactor firewall edit page to allow selection of a reply-to gateway in stead of the single "disablereplyto" option. Since underscores aren't valid for gateway names,
we should be able to use disable here to mark the previous "disablereplyto" setting which we can unravel when saving settings again.

just tested 21.7.b with the new reply-to option and it looks like WORK

[trunet]

@amonhk I think this is still not available for general public, right? But when we create a NAT port-forward, the firewall rule is added automatically and associated, and we don't have access to change this. If this makes it to work, the NAT rule should handle the firewall rule automatically as well.

[frankyw]

It would be better to take this to the forum where people can actually tell you it works and help diagnose. Also, kmod is unsupported at this point.

Already tried that, still does not work: https://forum.opnsense.org/index.php?topic=22856.0

[trunet]

Is this a misconfig by all of us, or this is a bug? if it's a bug, or this issue needs to be reopened, or we need to open a new one.

[H4R0]

Is this a misconfig by all of us, or this is a bug? if it's a bug, or this issue needs to be reopened, or we need to open a new one.

Since the exact same setup works with openvpn and ipsec I don't see how this is a misconfig from us.

Site-to-Site works, it's just port forwarding where replies get routed over wan instead of the wg interface.

Disabling reply-to and several other settings have no effect on this. The only workaround is to masquerade on the client side.

It would be interesting if pfsense has the same issue though.

[amonhk]

@trunet I also don't understand why the same openvpn logic applied to wireguard doesn't work (I think it's a bug)
@fraenki I made it work that way:
with the 21.7.b version (type development)

firewall ingress rule on the wireguard interface

Filter rule association set to None

I hope it will be useful to someone

[mimugmail]

Thx for testing, I plan to write an Advanced example section for WG docs

[trunet]

same port forward rule for both openvpn and wireguard interfaces. for openvpn it adds reply-to automatically, for wireguard it doesn't.

pass in quick on ovpnc4 reply-to (ovpnc4 10.XX.X.XX) inet proto tcp from any to <myserver> port = 10000 flags S/SA keep state label "514698ad49eba95b7d5b82502e1c3e65"
pass in quick on ovpnc4 reply-to (ovpnc4 10.XX.X.XX) inet proto udp from any to <myserver> port = 10000 keep state label "514698ad49eba95b7d5b82502e1c3e65"
pass in quick on wg1 inet proto tcp from any to <myserver> port = 10000 flags S/SA keep state label "514698ad49eba95b7d5b82502e1c3e65"
pass in quick on wg1 inet proto udp from any to <myserver> port = 10000 keep state label "514698ad49eba95b7d5b82502e1c3e65"

[jortan]

It would be interesting if pfsense has the same issue though.

pfsense doesn't (or at least didn't previously) have the same issue. I migrated from pfsense not too long after they introduced wireguard (due to the wg implementation drama / pfsense announcing they were going to remove wireguard temporarily)

I'm certain I had port forwarding working on wireguard interfaces in pfsense, but I also can't get them working in opnsense.

[trunet]

I did some $this->log to debug the problem.

wireguard interface doesn't contain a gateway on the interfacemapping, although I have a gateway defined on "Single"... openvpn contains a gateway there.

(
    [if] => wg1
    [descr] => [MY INTERFACE]
    [enable] => 1
    [lock] => 1
    [spoofmac] =>
    [ipaddrv6] =>
    [ipaddr] =>
    [gateway] =>
    [gatewayv6] =>
    [ifconfig] => Array
        (
            [ipv4] => Array
                (
                    [0] => Array
                        (
                            [ipaddr] => [MY VPN IP]
                            [subnetbits] => 24
                        )

                )

            [ipv6] => Array
                (
                )

        )

)

Therefore, https://github.com/opnsense/core/blob/master/src/opnsense/mvc/app/library/OPNsense/Firewall/FilterRule.php#L150 is false and it doesn't add reply-to wireguard interfaces.

[Nikotine1]

@trunet I also don't understand why the same openvpn logic applied to wireguard doesn't work (I think it's a bug)
@fraenki I made it work that way:
with the 21.7.b version (type development)

firewall ingress rule on the wireguard interface

Filter rule association set to None

I hope it will be useful to someone

I can confirm that manually creating the NAT port forward firewall rule, including the reply-to, fixed the port forwarding issue via Wireguard!

I could not figure out why Transmission kept saying my listening port was closed. It worked via WAN, but stayed closed via the WG interface. I even used Wireshark and could see Transmission communicating back an forth with 87.98.162.88. There was just some traffic that didn't happen when using the WG interface. I'm so lucky I stumbled upon this github issue!