Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IPv6_rpfilter=yes breaks WireGuard VPN over IPv6 #1203

Open
toreanderson opened this issue Aug 31, 2023 · 15 comments
Open

IPv6_rpfilter=yes breaks WireGuard VPN over IPv6 #1203

toreanderson opened this issue Aug 31, 2023 · 15 comments
Labels
needinfo Needs more information before action can be taken. triage Issue needs triaged.

Comments

@toreanderson
Copy link

What happened

When using a WireGuard VPN that tunnels all traffic towards an IPv6 VPN server, nothing works if firewalld is enabled with IPv6_rpfilter=yes. The WireGuard traffic from the VPN server is being dropped:

rpfilter_DROP: IN=wlan0 OUT= MAC=<snip> SRC=<wireguard-server-ipv6-address> DST=<wlan0-ipv6-address> LEN=256 TC=2 HOPLIMIT=56 FLOWLBL=0 PROTO=UDP SPT=51820 DPT=48993 LEN=216

This impacts both IPv4 and IPv6 traffic that is routed through the tunnel.

Setting IPv6_rpfilter=no and restarting firewalld.service makes things work.

What you expected to happen

The VPN tunnel should have worked fine with IPv6_rpfilter=yes (or this setting should have been disabled by default).

How to reproduce it (as minimally and precisely as possible)

  1. Connect to a network offering IPv6 connectivity
  2. Connect to an IPv6 WireGuard server that redirects all IPv6 traffic to itself (AllowedIPs = ::/0)
  3. Try to access anything on the Internet through the tunnel.

I used NetworkManager for the first two steps, for what it is worth.

Anything else we need to know?

This appears to be a regression, see #603.

Firewalld Version

firewalld-1.3.1-1.fc38.noarch

Firewalld Backend

nftables

Linux distribution

Fedora Linux 38 (Workstation Edition)

Linux kernel version

6.4.12-200.fc38.x86_64

Other information

No response
@toreanderson toreanderson added the triage Issue needs triaged. label Aug 31, 2023
@erig0
Copy link
Collaborator

erig0 commented Aug 31, 2023

Hrm. The filter has not changed since #603.

Can you share some details about your topology? Any chance of asymmetric routing?

@toreanderson
Copy link
Author

toreanderson commented Aug 31, 2023 •
edited

Pretty standard and uncomplicated, just my laptop connected over WiFi (dual-stacked), with a single WireGuard VPN-connection (dual-stacked WG server) that claims all IPv4/IPv6 traffic (AllowedIPs = 0.0.0.0/0, ::/0). Both the WiFi and the WireGuard VPN was set up with NetworkManager, nothing special as far as I can tell.

@toreanderson
Copy link
Author

Here's the output from the routing tables and so on (anonymised by doing global search&replace of the IPv6 prefixes):

[tore:~] $ ip route
default via 192.168.1.1 dev wlp0s20f3 proto dhcp src 192.168.1.182 metric 600
192.168.1.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.1.182 metric 600
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

[tore:~] $ ip -6 route
2001:db8::1 dev wireguard-vpn proto kernel metric 50 pref medium
2001:db8:ab:cd::a60 dev wlp0s20f3 proto kernel metric 600 pref medium
2001:db8:ab:cd::/64 dev wlp0s20f3 proto ra metric 600 pref medium
2001:db8:ab:cd::/56 via fe80::1a0f:76ff:fe39:bee8 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium
fe80::/64 dev wireguard-vpn proto kernel metric 1024 pref medium
default via fe80::1a0f:76ff:fe39:bee8 dev wlp0s20f3 proto ra metric 600 pref medium

[tore:~] $ ip rule
0:      from all lookup local
31558:  from all lookup main suppress_prefixlength 0
31559:  not from all fwmark 0xcb8e lookup 52110
32766:  from all lookup main
32767:  from all lookup default

[tore:~] $ ip -6 rule
0:      from all lookup local
31558:  from all lookup main suppress_prefixlength 0
31559:  not from all fwmark 0xcb8e lookup 52110
32766:  from all lookup main

[tore:~] $ ip route show table 52110
default dev wireguard-vpn proto static scope link metric 50

[tore:~] $ ip -6 route show table 52110
default dev wireguard-vpn proto static metric 50 pref medium

[tore:~] $ sudo wg
interface: wireguard-vpn
  public key: (hidden)
  private key: (hidden)
  listening port: 48993
  fwmark: 0xcb8e

peer: (hidden)
  preshared key: (hidden)
  endpoint: [2001:db8::ffff]:51820
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 21 seconds ago
  transfer: 747.20 MiB received, 59.47 MiB sent

@toreanderson
Copy link
Author

Here's the output from nft list ruleset when IPv6_rpfilter=yes:

table inet firewalld {
	ct helper helper-netbios-ns-udp {
		type "netbios-ns" protocol udp
		l3proto ip
	}

	chain mangle_PREROUTING {
		type filter hook prerouting priority mangle + 10; policy accept;
		jump mangle_PREROUTING_ZONES
	}

	chain mangle_PREROUTING_POLICIES_pre {
		jump mangle_PRE_policy_allow-host-ipv6
	}

	chain mangle_PREROUTING_ZONES {
		iifname "wireguard-vpn" goto mangle_PRE_FedoraWorkstation
		iifname "wlp0s20f3" goto mangle_PRE_FedoraWorkstation
		goto mangle_PRE_FedoraWorkstation
	}

	chain mangle_PREROUTING_POLICIES_post {
	}

	chain nat_PREROUTING {
		type nat hook prerouting priority dstnat + 10; policy accept;
		jump nat_PREROUTING_ZONES
	}

	chain nat_PREROUTING_POLICIES_pre {
		jump nat_PRE_policy_allow-host-ipv6
	}

	chain nat_PREROUTING_ZONES {
		iifname "wireguard-vpn" goto nat_PRE_FedoraWorkstation
		iifname "wlp0s20f3" goto nat_PRE_FedoraWorkstation
		goto nat_PRE_FedoraWorkstation
	}

	chain nat_PREROUTING_POLICIES_post {
	}

	chain nat_POSTROUTING {
		type nat hook postrouting priority srcnat + 10; policy accept;
		jump nat_POSTROUTING_ZONES
	}

	chain nat_POSTROUTING_POLICIES_pre {
	}

	chain nat_POSTROUTING_ZONES {
		oifname "wireguard-vpn" goto nat_POST_FedoraWorkstation
		oifname "wlp0s20f3" goto nat_POST_FedoraWorkstation
		goto nat_POST_FedoraWorkstation
	}

	chain nat_POSTROUTING_POLICIES_post {
	}

	chain nat_OUTPUT {
		type nat hook output priority -90; policy accept;
		jump nat_OUTPUT_POLICIES_pre
		jump nat_OUTPUT_POLICIES_post
	}

	chain nat_OUTPUT_POLICIES_pre {
	}

	chain nat_OUTPUT_POLICIES_post {
	}

	chain filter_PREROUTING {
		type filter hook prerouting priority filter + 10; policy accept;
		icmpv6 type { nd-router-advert, nd-neighbor-solicit } accept
		meta nfproto ipv6 fib saddr . mark . iif oif missing log prefix "rpfilter_DROP: " drop
	}

	chain filter_INPUT {
		type filter hook input priority filter + 10; policy accept;
		ct state { established, related } accept
		ct status dnat accept
		iifname "lo" accept
		ct state invalid log prefix "STATE_INVALID_DROP: "
		ct state invalid drop
		jump filter_INPUT_ZONES
		log prefix "FINAL_REJECT: "
		reject with icmpx admin-prohibited
	}

	chain filter_FORWARD {
		type filter hook forward priority filter + 10; policy accept;
		ct state { established, related } accept
		ct status dnat accept
		iifname "lo" accept
		ct state invalid log prefix "STATE_INVALID_DROP: "
		ct state invalid drop
		ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 addr-unreachable
		jump filter_FORWARD_ZONES
		log prefix "FINAL_REJECT: "
		reject with icmpx admin-prohibited
	}

	chain filter_OUTPUT {
		type filter hook output priority filter + 10; policy accept;
		ct state { established, related } accept
		oifname "lo" accept
		ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 addr-unreachable
		jump filter_OUTPUT_POLICIES_pre
		jump filter_OUTPUT_POLICIES_post
	}

	chain filter_INPUT_POLICIES_pre {
		jump filter_IN_policy_allow-host-ipv6
	}

	chain filter_INPUT_ZONES {
		iifname "wireguard-vpn" goto filter_IN_FedoraWorkstation
		iifname "wlp0s20f3" goto filter_IN_FedoraWorkstation
		goto filter_IN_FedoraWorkstation
	}

	chain filter_INPUT_POLICIES_post {
	}

	chain filter_FORWARD_POLICIES_pre {
	}

	chain filter_FORWARD_ZONES {
		iifname "wireguard-vpn" goto filter_FWD_FedoraWorkstation
		iifname "wlp0s20f3" goto filter_FWD_FedoraWorkstation
		goto filter_FWD_FedoraWorkstation
	}

	chain filter_FORWARD_POLICIES_post {
	}

	chain filter_OUTPUT_POLICIES_pre {
	}

	chain filter_OUTPUT_POLICIES_post {
	}

	chain filter_IN_FedoraWorkstation {
		jump filter_INPUT_POLICIES_pre
		jump filter_IN_FedoraWorkstation_pre
		jump filter_IN_FedoraWorkstation_log
		jump filter_IN_FedoraWorkstation_deny
		jump filter_IN_FedoraWorkstation_allow
		jump filter_IN_FedoraWorkstation_post
		jump filter_INPUT_POLICIES_post
		meta l4proto { icmp, ipv6-icmp } accept
		log prefix "filter_IN_FedoraWorkstation_REJECT: "
		reject with icmpx admin-prohibited
	}

	chain filter_IN_FedoraWorkstation_pre {
	}

	chain filter_IN_FedoraWorkstation_log {
	}

	chain filter_IN_FedoraWorkstation_deny {
	}

	chain filter_IN_FedoraWorkstation_allow {
		ip6 daddr fe80::/64 udp dport 546 accept
		tcp dport 22 accept
		udp dport 137 ct helper set "helper-netbios-ns-udp"
		udp dport 137 accept
		udp dport 138 accept
		ip daddr 224.0.0.251 udp dport 5353 accept
		ip6 daddr ff02::fb udp dport 5353 accept
		udp dport 1025-65535 accept
		tcp dport 1025-65535 accept
	}

	chain filter_IN_FedoraWorkstation_post {
	}

	chain nat_POST_FedoraWorkstation {
		jump nat_POSTROUTING_POLICIES_pre
		jump nat_POST_FedoraWorkstation_pre
		jump nat_POST_FedoraWorkstation_log
		jump nat_POST_FedoraWorkstation_deny
		jump nat_POST_FedoraWorkstation_allow
		jump nat_POST_FedoraWorkstation_post
		jump nat_POSTROUTING_POLICIES_post
	}

	chain nat_POST_FedoraWorkstation_pre {
	}

	chain nat_POST_FedoraWorkstation_log {
	}

	chain nat_POST_FedoraWorkstation_deny {
	}

	chain nat_POST_FedoraWorkstation_allow {
	}

	chain nat_POST_FedoraWorkstation_post {
	}

	chain filter_FWD_FedoraWorkstation {
		jump filter_FORWARD_POLICIES_pre
		jump filter_FWD_FedoraWorkstation_pre
		jump filter_FWD_FedoraWorkstation_log
		jump filter_FWD_FedoraWorkstation_deny
		jump filter_FWD_FedoraWorkstation_allow
		jump filter_FWD_FedoraWorkstation_post
		jump filter_FORWARD_POLICIES_post
		log prefix "filter_FWD_FedoraWorkstation_REJECT: "
		reject with icmpx admin-prohibited
	}

	chain filter_FWD_FedoraWorkstation_pre {
	}

	chain filter_FWD_FedoraWorkstation_log {
	}

	chain filter_FWD_FedoraWorkstation_deny {
	}

	chain filter_FWD_FedoraWorkstation_allow {
	}

	chain filter_FWD_FedoraWorkstation_post {
	}

	chain nat_PRE_FedoraWorkstation {
		jump nat_PREROUTING_POLICIES_pre
		jump nat_PRE_FedoraWorkstation_pre
		jump nat_PRE_FedoraWorkstation_log
		jump nat_PRE_FedoraWorkstation_deny
		jump nat_PRE_FedoraWorkstation_allow
		jump nat_PRE_FedoraWorkstation_post
		jump nat_PREROUTING_POLICIES_post
	}

	chain nat_PRE_FedoraWorkstation_pre {
	}

	chain nat_PRE_FedoraWorkstation_log {
	}

	chain nat_PRE_FedoraWorkstation_deny {
	}

	chain nat_PRE_FedoraWorkstation_allow {
	}

	chain nat_PRE_FedoraWorkstation_post {
	}

	chain mangle_PRE_FedoraWorkstation {
		jump mangle_PREROUTING_POLICIES_pre
		jump mangle_PRE_FedoraWorkstation_pre
		jump mangle_PRE_FedoraWorkstation_log
		jump mangle_PRE_FedoraWorkstation_deny
		jump mangle_PRE_FedoraWorkstation_allow
		jump mangle_PRE_FedoraWorkstation_post
		jump mangle_PREROUTING_POLICIES_post
	}

	chain mangle_PRE_FedoraWorkstation_pre {
	}

	chain mangle_PRE_FedoraWorkstation_log {
	}

	chain mangle_PRE_FedoraWorkstation_deny {
	}

	chain mangle_PRE_FedoraWorkstation_allow {
	}

	chain mangle_PRE_FedoraWorkstation_post {
	}

	chain filter_IN_policy_allow-host-ipv6 {
		jump filter_IN_policy_allow-host-ipv6_pre
		jump filter_IN_policy_allow-host-ipv6_log
		jump filter_IN_policy_allow-host-ipv6_deny
		jump filter_IN_policy_allow-host-ipv6_allow
		jump filter_IN_policy_allow-host-ipv6_post
	}

	chain filter_IN_policy_allow-host-ipv6_pre {
	}

	chain filter_IN_policy_allow-host-ipv6_log {
	}

	chain filter_IN_policy_allow-host-ipv6_deny {
	}

	chain filter_IN_policy_allow-host-ipv6_allow {
		icmpv6 type nd-neighbor-advert accept
		icmpv6 type nd-neighbor-solicit accept
		icmpv6 type nd-router-advert accept
		icmpv6 type nd-redirect accept
	}

	chain filter_IN_policy_allow-host-ipv6_post {
	}

	chain nat_PRE_policy_allow-host-ipv6 {
		jump nat_PRE_policy_allow-host-ipv6_pre
		jump nat_PRE_policy_allow-host-ipv6_log
		jump nat_PRE_policy_allow-host-ipv6_deny
		jump nat_PRE_policy_allow-host-ipv6_allow
		jump nat_PRE_policy_allow-host-ipv6_post
	}

	chain nat_PRE_policy_allow-host-ipv6_pre {
	}

	chain nat_PRE_policy_allow-host-ipv6_log {
	}

	chain nat_PRE_policy_allow-host-ipv6_deny {
	}

	chain nat_PRE_policy_allow-host-ipv6_allow {
	}

	chain nat_PRE_policy_allow-host-ipv6_post {
	}

	chain mangle_PRE_policy_allow-host-ipv6 {
		jump mangle_PRE_policy_allow-host-ipv6_pre
		jump mangle_PRE_policy_allow-host-ipv6_log
		jump mangle_PRE_policy_allow-host-ipv6_deny
		jump mangle_PRE_policy_allow-host-ipv6_allow
		jump mangle_PRE_policy_allow-host-ipv6_post
	}

	chain mangle_PRE_policy_allow-host-ipv6_pre {
	}

	chain mangle_PRE_policy_allow-host-ipv6_log {
	}

	chain mangle_PRE_policy_allow-host-ipv6_deny {
	}

	chain mangle_PRE_policy_allow-host-ipv6_allow {
	}

	chain mangle_PRE_policy_allow-host-ipv6_post {
	}
}

The following is the diff from the above, when I disable IPv6_rpfilter and restart firewalld:

--- rpfilter_enabled.txt        2023-08-31 15:24:02.751652491 +0200
+++ rpfilter_disabled.txt       2023-08-31 15:24:02.751652491 +0200
@@ -71,8 +71,6 @@
 
        chain filter_PREROUTING {
                type filter hook prerouting priority filter + 10; policy accept;
-               icmpv6 type { nd-router-advert, nd-neighbor-solicit } accept
-               meta nfproto ipv6 fib saddr . mark . iif oif missing log prefix "rpfilter_DROP: " drop
        }
 
        chain filter_INPUT {

@oxwivi
Copy link

oxwivi commented Dec 4, 2023

I'm on Fedora 39, and I'm positive I'm affected by this bug. Using Cloudflare's WARP service, it's impossible to establish connection so long as IPv6 is active on the primary interface. Disabling v6 forces v4 connection which goes through as expected. After finding this GitHub issue, I disabled IPv6_rpfilter and was able to successfully establish WG connection with IPv6 enabled for the first time.

Please let me know any information I can provide to debug this further.

@oxwivi oxwivi mentioned this issue Dec 4, 2023
Open
@erig0
Copy link
Collaborator

erig0 commented Dec 4, 2023

Can someone show iptables rules? I'm guessing wireguard is installing some iptables rules to set the fwmark.

@oxwivi
Copy link

oxwivi commented Dec 4, 2023

What command exactly do you want the output of? I got output of the tables filter, mangle and nat but they don't seem very useful:

$ sudo iptables -vL -t filter
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
$ sudo iptables -vL -t nat   
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
$ sudo iptables -vL -t mangle
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

@erig0
Copy link
Collaborator

erig0 commented Dec 5, 2023

Hrm. Nothing there. Are there nftables rules installed by wireguard? Check nft list ruleset.

Is firewalld started before or after wireguard?

@oxwivi
Copy link

oxwivi commented Dec 5, 2023 •
edited

I'm connecting to WireGaurd manually after logging into my DE. Also, do note that I'm using NetworkManager to initiate the connection. I'm not sure if OP used NM or the suite of wg tools. Just adding this info in case it's important context.

EDIT More context: this output is with the rpfilter disabled.

$ sudo nft list ruleset
table inet firewalld {
        ct helper helper-netbios-ns-udp {
                type "netbios-ns" protocol udp
                l3proto ip
        }

        chain mangle_PREROUTING {
                type filter hook prerouting priority mangle + 10; policy accept;
                jump mangle_PREROUTING_POLICIES
        }

        chain mangle_PREROUTING_POLICIES {
                iifname "B56473" jump mangle_PRE_policy_allow-host-ipv6
                iifname "B56473" jump mangle_PRE_FedoraWorkstation
                iifname "B56473" return
                iifname "enp4s0" jump mangle_PRE_policy_allow-host-ipv6
                iifname "enp4s0" jump mangle_PRE_FedoraWorkstation
                iifname "enp4s0" return
                iifname "wlp5s0" jump mangle_PRE_policy_allow-host-ipv6
                iifname "wlp5s0" jump mangle_PRE_FedoraWorkstation
                iifname "wlp5s0" return
                jump mangle_PRE_policy_allow-host-ipv6
                jump mangle_PRE_FedoraWorkstation
                return
        }

        chain nat_PREROUTING {
                type nat hook prerouting priority dstnat + 10; policy accept;
                jump nat_PREROUTING_POLICIES
        }

        chain nat_PREROUTING_POLICIES {
                iifname "B56473" jump nat_PRE_policy_allow-host-ipv6
                iifname "B56473" jump nat_PRE_FedoraWorkstation
                iifname "B56473" return
                iifname "enp4s0" jump nat_PRE_policy_allow-host-ipv6
                iifname "enp4s0" jump nat_PRE_FedoraWorkstation
                iifname "enp4s0" return
                iifname "wlp5s0" jump nat_PRE_policy_allow-host-ipv6
                iifname "wlp5s0" jump nat_PRE_FedoraWorkstation
                iifname "wlp5s0" return
                jump nat_PRE_policy_allow-host-ipv6
                jump nat_PRE_FedoraWorkstation
                return
        }

        chain nat_POSTROUTING {
                type nat hook postrouting priority srcnat + 10; policy accept;
                jump nat_POSTROUTING_POLICIES
        }

        chain nat_POSTROUTING_POLICIES {
                iifname "B56473" oifname "B56473" jump nat_POST_FedoraWorkstation
                iifname "B56473" oifname "B56473" return
                iifname "enp4s0" oifname "B56473" jump nat_POST_FedoraWorkstation
                iifname "enp4s0" oifname "B56473" return
                iifname "wlp5s0" oifname "B56473" jump nat_POST_FedoraWorkstation
                iifname "wlp5s0" oifname "B56473" return
                oifname "B56473" jump nat_POST_FedoraWorkstation
                oifname "B56473" return
                iifname "B56473" oifname "enp4s0" jump nat_POST_FedoraWorkstation
                iifname "B56473" oifname "enp4s0" return
                iifname "enp4s0" oifname "enp4s0" jump nat_POST_FedoraWorkstation
                iifname "enp4s0" oifname "enp4s0" return
                iifname "wlp5s0" oifname "enp4s0" jump nat_POST_FedoraWorkstation
                iifname "wlp5s0" oifname "enp4s0" return
                oifname "enp4s0" jump nat_POST_FedoraWorkstation
                oifname "enp4s0" return
                iifname "B56473" oifname "wlp5s0" jump nat_POST_FedoraWorkstation
                iifname "B56473" oifname "wlp5s0" return
                iifname "enp4s0" oifname "wlp5s0" jump nat_POST_FedoraWorkstation
                iifname "enp4s0" oifname "wlp5s0" return
                iifname "wlp5s0" oifname "wlp5s0" jump nat_POST_FedoraWorkstation
                iifname "wlp5s0" oifname "wlp5s0" return
                oifname "wlp5s0" jump nat_POST_FedoraWorkstation
                oifname "wlp5s0" return
                iifname "B56473" jump nat_POST_FedoraWorkstation
                iifname "B56473" return
                iifname "enp4s0" jump nat_POST_FedoraWorkstation
                iifname "enp4s0" return
                iifname "wlp5s0" jump nat_POST_FedoraWorkstation
                iifname "wlp5s0" return
                jump nat_POST_FedoraWorkstation
                return
        }

        chain nat_OUTPUT {
                type nat hook output priority -90; policy accept;
                jump nat_OUTPUT_POLICIES
        }

        chain nat_OUTPUT_POLICIES {
                oifname "B56473" jump nat_OUT_FedoraWorkstation
                oifname "B56473" return
                oifname "enp4s0" jump nat_OUT_FedoraWorkstation
                oifname "enp4s0" return
                oifname "wlp5s0" jump nat_OUT_FedoraWorkstation
                oifname "wlp5s0" return
                jump nat_OUT_FedoraWorkstation
                return
        }

        chain filter_PREROUTING {
                type filter hook prerouting priority filter + 10; policy accept;
        }

        chain filter_INPUT {
                type filter hook input priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                ct state invalid drop
                jump filter_INPUT_POLICIES
                reject with icmpx admin-prohibited
        }

        chain filter_FORWARD {
                type filter hook forward priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                ct state invalid drop
                ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 addr-unreachable
                jump filter_FORWARD_POLICIES
                reject with icmpx admin-prohibited
        }

        chain filter_OUTPUT {
                type filter hook output priority filter + 10; policy accept;
                ct state { established, related } accept
                oifname "lo" accept
                ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 addr-unreachable
                jump filter_OUTPUT_POLICIES
        }

        chain filter_INPUT_POLICIES {
                iifname "B56473" jump filter_IN_policy_allow-host-ipv6
                iifname "B56473" jump filter_IN_FedoraWorkstation
                iifname "B56473" reject with icmpx admin-prohibited
                iifname "enp4s0" jump filter_IN_policy_allow-host-ipv6
                iifname "enp4s0" jump filter_IN_FedoraWorkstation
                iifname "enp4s0" reject with icmpx admin-prohibited
                iifname "wlp5s0" jump filter_IN_policy_allow-host-ipv6
                iifname "wlp5s0" jump filter_IN_FedoraWorkstation
                iifname "wlp5s0" reject with icmpx admin-prohibited
                jump filter_IN_policy_allow-host-ipv6
                jump filter_IN_FedoraWorkstation
                reject with icmpx admin-prohibited
        }

        chain filter_FORWARD_POLICIES {
                iifname "B56473" oifname "B56473" jump filter_FWD_FedoraWorkstation
                iifname "B56473" oifname "B56473" reject with icmpx admin-prohibited
                iifname "B56473" oifname "enp4s0" jump filter_FWD_FedoraWorkstation
                iifname "B56473" oifname "enp4s0" reject with icmpx admin-prohibited
                iifname "B56473" oifname "wlp5s0" jump filter_FWD_FedoraWorkstation
                iifname "B56473" oifname "wlp5s0" reject with icmpx admin-prohibited
                iifname "B56473" jump filter_FWD_FedoraWorkstation
                iifname "B56473" reject with icmpx admin-prohibited
                iifname "enp4s0" oifname "B56473" jump filter_FWD_FedoraWorkstation
                iifname "enp4s0" oifname "B56473" reject with icmpx admin-prohibited
                iifname "enp4s0" oifname "enp4s0" jump filter_FWD_FedoraWorkstation
                iifname "enp4s0" oifname "enp4s0" reject with icmpx admin-prohibited
                iifname "enp4s0" oifname "wlp5s0" jump filter_FWD_FedoraWorkstation
                iifname "enp4s0" oifname "wlp5s0" reject with icmpx admin-prohibited
                iifname "enp4s0" jump filter_FWD_FedoraWorkstation
                iifname "enp4s0" reject with icmpx admin-prohibited
                iifname "wlp5s0" oifname "B56473" jump filter_FWD_FedoraWorkstation
                iifname "wlp5s0" oifname "B56473" reject with icmpx admin-prohibited
                iifname "wlp5s0" oifname "enp4s0" jump filter_FWD_FedoraWorkstation
                iifname "wlp5s0" oifname "enp4s0" reject with icmpx admin-prohibited
                iifname "wlp5s0" oifname "wlp5s0" jump filter_FWD_FedoraWorkstation
                iifname "wlp5s0" oifname "wlp5s0" reject with icmpx admin-prohibited
                iifname "wlp5s0" jump filter_FWD_FedoraWorkstation
                iifname "wlp5s0" reject with icmpx admin-prohibited
                oifname "B56473" jump filter_FWD_FedoraWorkstation
                oifname "B56473" reject with icmpx admin-prohibited
                oifname "enp4s0" jump filter_FWD_FedoraWorkstation
                oifname "enp4s0" reject with icmpx admin-prohibited
                oifname "wlp5s0" jump filter_FWD_FedoraWorkstation
                oifname "wlp5s0" reject with icmpx admin-prohibited
                jump filter_FWD_FedoraWorkstation
                reject with icmpx admin-prohibited
        }

        chain filter_OUTPUT_POLICIES {
                oifname "B56473" jump filter_OUT_FedoraWorkstation
                oifname "B56473" return
                oifname "enp4s0" jump filter_OUT_FedoraWorkstation
                oifname "enp4s0" return
                oifname "wlp5s0" jump filter_OUT_FedoraWorkstation
                oifname "wlp5s0" return
                jump filter_OUT_FedoraWorkstation
                return
        }

        chain filter_IN_FedoraWorkstation {
                jump filter_IN_FedoraWorkstation_pre
                jump filter_IN_FedoraWorkstation_log
                jump filter_IN_FedoraWorkstation_deny
                jump filter_IN_FedoraWorkstation_allow
                jump filter_IN_FedoraWorkstation_post
                meta l4proto { icmp, ipv6-icmp } accept
        }

        chain filter_IN_FedoraWorkstation_pre {
        }

        chain filter_IN_FedoraWorkstation_log {
        }

        chain filter_IN_FedoraWorkstation_deny {
        }

        chain filter_IN_FedoraWorkstation_allow {
                ip6 daddr fe80::/64 udp dport 546 accept
                tcp dport 22 accept
                udp dport 137 ct helper set "helper-netbios-ns-udp"
                udp dport 137 accept
                udp dport 138 accept
                ip daddr 224.0.0.251 udp dport 5353 accept
                ip6 daddr ff02::fb udp dport 5353 accept
                tcp dport 1714-1764 accept
                udp dport 1714-1764 accept
                udp dport 1025-65535 accept
                tcp dport 1025-65535 accept
        }

        chain filter_IN_FedoraWorkstation_post {
        }

        chain filter_OUT_FedoraWorkstation {
                jump filter_OUT_FedoraWorkstation_pre
                jump filter_OUT_FedoraWorkstation_log
                jump filter_OUT_FedoraWorkstation_deny
                jump filter_OUT_FedoraWorkstation_allow
                jump filter_OUT_FedoraWorkstation_post
        }

        chain filter_OUT_FedoraWorkstation_pre {
        }

        chain filter_OUT_FedoraWorkstation_log {
        }

        chain filter_OUT_FedoraWorkstation_deny {
        }

        chain filter_OUT_FedoraWorkstation_allow {
        }

        chain filter_OUT_FedoraWorkstation_post {
        }

        chain nat_OUT_FedoraWorkstation {
                jump nat_OUT_FedoraWorkstation_pre
                jump nat_OUT_FedoraWorkstation_log
                jump nat_OUT_FedoraWorkstation_deny
                jump nat_OUT_FedoraWorkstation_allow
                jump nat_OUT_FedoraWorkstation_post
        }

        chain nat_OUT_FedoraWorkstation_pre {
        }

        chain nat_OUT_FedoraWorkstation_log {
        }

        chain nat_OUT_FedoraWorkstation_deny {
        }

        chain nat_OUT_FedoraWorkstation_allow {
        }

        chain nat_OUT_FedoraWorkstation_post {
        }

        chain nat_POST_FedoraWorkstation {
                jump nat_POST_FedoraWorkstation_pre
                jump nat_POST_FedoraWorkstation_log
                jump nat_POST_FedoraWorkstation_deny
                jump nat_POST_FedoraWorkstation_allow
                jump nat_POST_FedoraWorkstation_post
        }

        chain nat_POST_FedoraWorkstation_pre {
        }

        chain nat_POST_FedoraWorkstation_log {
        }

        chain nat_POST_FedoraWorkstation_deny {
        }

        chain nat_POST_FedoraWorkstation_allow {
        }

        chain nat_POST_FedoraWorkstation_post {
        }

        chain filter_FWD_FedoraWorkstation {
                jump filter_FWD_FedoraWorkstation_pre
                jump filter_FWD_FedoraWorkstation_log
                jump filter_FWD_FedoraWorkstation_deny
                jump filter_FWD_FedoraWorkstation_allow
                jump filter_FWD_FedoraWorkstation_post
        }

        chain filter_FWD_FedoraWorkstation_pre {
        }

        chain filter_FWD_FedoraWorkstation_log {
        }

        chain filter_FWD_FedoraWorkstation_deny {
        }

        chain filter_FWD_FedoraWorkstation_allow {
        }

        chain filter_FWD_FedoraWorkstation_post {
        }

        chain nat_PRE_FedoraWorkstation {
                jump nat_PRE_FedoraWorkstation_pre
                jump nat_PRE_FedoraWorkstation_log
                jump nat_PRE_FedoraWorkstation_deny
                jump nat_PRE_FedoraWorkstation_allow
                jump nat_PRE_FedoraWorkstation_post
        }

        chain nat_PRE_FedoraWorkstation_pre {
        }

        chain nat_PRE_FedoraWorkstation_log {
        }

        chain nat_PRE_FedoraWorkstation_deny {
        }

        chain nat_PRE_FedoraWorkstation_allow {
        }

        chain nat_PRE_FedoraWorkstation_post {
        }

        chain mangle_PRE_FedoraWorkstation {
                jump mangle_PRE_FedoraWorkstation_pre
                jump mangle_PRE_FedoraWorkstation_log
                jump mangle_PRE_FedoraWorkstation_deny
                jump mangle_PRE_FedoraWorkstation_allow
                jump mangle_PRE_FedoraWorkstation_post
        }

        chain mangle_PRE_FedoraWorkstation_pre {
        }

        chain mangle_PRE_FedoraWorkstation_log {
        }

        chain mangle_PRE_FedoraWorkstation_deny {
        }

        chain mangle_PRE_FedoraWorkstation_allow {
        }

        chain mangle_PRE_FedoraWorkstation_post {
        }

        chain filter_IN_policy_allow-host-ipv6 {
                jump filter_IN_policy_allow-host-ipv6_pre
                jump filter_IN_policy_allow-host-ipv6_log
                jump filter_IN_policy_allow-host-ipv6_deny
                jump filter_IN_policy_allow-host-ipv6_allow
                jump filter_IN_policy_allow-host-ipv6_post
        }

        chain filter_IN_policy_allow-host-ipv6_pre {
        }

        chain filter_IN_policy_allow-host-ipv6_log {
        }

        chain filter_IN_policy_allow-host-ipv6_deny {
        }

        chain filter_IN_policy_allow-host-ipv6_allow {
                icmpv6 type nd-neighbor-advert accept
                icmpv6 type nd-neighbor-solicit accept
                icmpv6 type nd-router-advert accept
                icmpv6 type nd-redirect accept
        }

        chain filter_IN_policy_allow-host-ipv6_post {
        }

        chain nat_PRE_policy_allow-host-ipv6 {
                jump nat_PRE_policy_allow-host-ipv6_pre
                jump nat_PRE_policy_allow-host-ipv6_log
                jump nat_PRE_policy_allow-host-ipv6_deny
                jump nat_PRE_policy_allow-host-ipv6_allow
                jump nat_PRE_policy_allow-host-ipv6_post
        }

        chain nat_PRE_policy_allow-host-ipv6_pre {
        }

        chain nat_PRE_policy_allow-host-ipv6_log {
        }

        chain nat_PRE_policy_allow-host-ipv6_deny {
        }

        chain nat_PRE_policy_allow-host-ipv6_allow {
        }

        chain nat_PRE_policy_allow-host-ipv6_post {
        }

        chain mangle_PRE_policy_allow-host-ipv6 {
                jump mangle_PRE_policy_allow-host-ipv6_pre
                jump mangle_PRE_policy_allow-host-ipv6_log
                jump mangle_PRE_policy_allow-host-ipv6_deny
                jump mangle_PRE_policy_allow-host-ipv6_allow
                jump mangle_PRE_policy_allow-host-ipv6_post
        }

        chain mangle_PRE_policy_allow-host-ipv6_pre {
        }

        chain mangle_PRE_policy_allow-host-ipv6_log {
        }

        chain mangle_PRE_policy_allow-host-ipv6_deny {
        }

        chain mangle_PRE_policy_allow-host-ipv6_allow {
        }

        chain mangle_PRE_policy_allow-host-ipv6_post {
        }
}

@brownan
Copy link

brownan commented Feb 22, 2024

I'm guessing wireguard is installing some iptables rules to set the fwmark.

Just ran into this issue. You're close, @erig0, they're not iptables rules but rather a routing policy database rule. I hope I can provide some more useful information on this.

Background: Wireguard interfaces can be set to apply a fwmark to the encrypted tunnel packets it sends. I guess this is so those packets can be identified in the routing database or iptables rules. The wg-quick script uses this feature when tunneling all traffic (when setting allowed ips to 0.0.0.0/0 or ::/0) so that it can install a new route for all packets not marked with the fwmark which routes the packet via the wireguard interface. So it can route all traffic except the wireguard packets themselves, without touching the main routing table.

This is described more in the section "Improved Rule-based Routing" on https://www.wireguard.com/netns/

Here's the commands wg-quick executes to bring up my vpn:

[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.0.8.2 dev wg0
[#] ip -6 address add fd9d:767f:77ec::2 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] wg set wg0 fwmark 51820
[#] ip -6 route add ::/0 dev wg0 table 51820
[#] ip -6 rule add not fwmark 51820 table 51820
[#] ip -6 rule add table main suppress_prefixlength 0
[#] ip6tables-restore -n
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] sysctl -q net.ipv4.conf.all.src_valid_mark=1
[#] iptables-restore -n
[#] resolvectl dns wg0 1.1.1.1; resolvectl domain wg0 "~."

Here's the ipv6 routing policy:

$ ip -6 rule
0:	from all lookup local
32764:	from all lookup main suppress_prefixlength 0
32765:	not from all fwmark 0xca6c lookup 51820
32766:	from all lookup main

And the ipv6 routing table number 51820:

$ ip -6 route show table 51820
default dev wg0 metric 1024 pref medium

The main routing table is unchanged from the routes installed by other network management software.

The effect of this is that outgoing packets will hit the routing policy rule 32765 and thus hit the alternate routing table with the single default route to the interface wg0.

Wireguard will then generate an encrypted outgoing packet with the fwmark on it. That skips the routing policy rule 32765 and thus goes to the normal routing table and gets routed normally.

I'm not sure I completely understand what the rpfilter does or why it interferes, but after I found this bug and turned it off, my tunnel started working over ipv6.

Happy to help debug further.

@erig0
Copy link
Collaborator

erig0 commented Feb 29, 2024 •
edited

So on the return path, i.e. packets coming into the node running firewalld, there will not be a fwmark. Firewalld will do a rpfilter (fib lookup) on this packet (no fwmark present) and this yields wg0. However the packet is being RX'd on a different interface, e.g. eth0. Thus the rpfilter check fails.

I bet if you add an iptables or nftables rule that adds fwmark 0xca6c to RX'd wireguard traffic before it hits the firewalld rules, then it'll start working. You probably have to do this in the mangle/prerouting hooks.

Can anyone test that?

@erig0 erig0 added the needinfo Needs more information before action can be taken. label Feb 29, 2024
@oxwivi
Copy link

oxwivi commented Mar 17, 2024 •
edited

I can try, but you need to teach me how to add the fwmark 0xca6c rule. (Also, if you want to replicate my setup for yourself, simply generate a WG conf file using wgcf; it's to connect and use Cloudflare's free WARP service)

@brownan
Copy link

brownan commented Mar 17, 2024

With a wireguard tunnel set up, the command wg set <interface> fwmark <fwmark> will set the fwmark for all wireguard outgoing packets.

I haven't had a chance to try out the suggested test, since I'm no longer traveling and thus not using my vpn. I'll still try to test this if I can find the time though.

@erig0
Copy link
Collaborator

erig0 commented Mar 19, 2024 •
edited

Creating a new nftables table with this rule should do the trick. It'll apply the mark for both IPv4 and IPv6.

Apply it with this:

# nft -f - <<HERE
> table inet wg-mark {
        chain mangle {
                type filter hook prerouting priority mangle; policy accept;
                udp dport 51820 meta mark set 0x0000ca6c
        }
}
HERE

@oxwivi
Copy link

oxwivi commented Apr 28, 2024

Sorry for the extremely late response. I was not in a position to test things out, and come next month, it'll probably the same again.

Anyhow, the fwmark thing (using the nft command, not wg) didn't work. As soon I restart firewalld after IPv6_rpfilter=yes, the connection stops working. (Side note, WG is supposed to be stateless, but changes like these requires me to disconnect and reconnect the connection on NetworkManager to make it work again)

I also don't know if it's related to the RP filter issue, but the WG connection sometimes temporarily stops working for a few minutes (never timed, but I'd hazard guess around 10 min or so), before simply working as expected. Stopping and reconnecting always fixes the issue, and sometimes suspending and resuming after a few minutes fixes it quickly than it automatically recovering. I mention this because this behavior I've never seen on Android or Windows with the same service and WG profile. During the WG downtime, I'm sure it's not network or Internet issue because I can still continue to access LAN resources (router web console, Android scrcpy, Windows Sunshine remote desktop, etc) without a hitch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needinfo Needs more information before action can be taken. triage Issue needs triaged.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@brownan @erig0 @toreanderson @oxwivi