Both virtual CARP IPv6 addresses of the Cluster Machines stay in Master State

[bewue]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[x] I have searched the existing issues and I'm convinced that mine is new.

Describe the bug
Both virtual CARP IPv6 addresses of the cluster machines stay in master state.
With tcpdump i can not see any IPv6 CARP traffic on that interfaces.

To Reproduce
Steps to reproduce the behavior:

1. Add an IPv4 address to the corresponding interface on both cluster machines
2. On the master machine add a virtual IPv4 CARP address on that interface (XMLRPC on)
   ...Until here CARP is functional.
3. Do the same for IPv6 addresses but with a different vhid. Thus there is an IPv4, an IPv6, a virtual CARP IPv4 and a virtual CARP IPv6 address on both interfaces
   Problem: Both virtual CARP IPv6 addresses of the cluster machines stay in master state. (Firewall: Virtual IPs: Status)

Expected behavior
I also expect a correct virtual CARP address master/backup status negotiation with IPv6

Additional context
Tried CARP "enable/disable", CARP maintenance mode "enter/leave" and reboots with both machines. Also tried adding an IPv6 Alias with same vhid (virtual CARP IPv4 vhid) instead of an virtual CARP IPv6.

Environment
OPNsense 19.1.6 (amd64, OpenSSL).
Running on Xen (XCP-ng)

[ppmathis]

I am experiencing the same issue on my OPNsense cluster running version 19.1.6 on amd64. I noticed that disabling the firewall completely on the master host (pfctl -d) immediately results in outgoing IPv6 CARP packets being visible:

# tcpdump -i <carp-iface> -ttt -n proto CARP -T carp
00:00:00.800016 IP6 fe80::xxx > ff02::12: ip-proto-112 36

Adding a floating rule which allows IPv6 CARP from "any" to "any" over "any interface" doesn't seem to work here, only completely disabling the firewall does.

As long as pf isn't disabled, I am only seeing IPv4 CARP advertisements and neither outgoing nor incoming CARP traffic for the configured IPv6 addresses. After searching the forums, I found the following threads which were never resolved but show exactly the same issue / pattern:

• https://forum.opnsense.org/index.php?topic=6290.0 (german, newest update from 2019)
• https://forum.opnsense.org/index.php?topic=11691.0 (english, newest update from 2019)
• https://forum.opnsense.org/index.php?topic=6228.0 (english, last update in 2018, same issue reporter)

Could you try/verify if pfctl -d also "resolves" the issue in your environment? FWIW, you can re-enable the firewall with pfctl -e after doing so, which should make the CARP advertisements disappear again.

I already tried checking for dropping packets on the firewall by running tcpdump -n -e -ttt -i pflog0 proto CARP, but no dropped outgoing CARP advertisements are visible there either. As I am certainly no expert with FreeBSD, I did not get further up to this point.

[bewue]

Ok, didnt know theres an different tcpdump syntax for IPv6 CARP package filtering compared to IPv4.
Anyway there is no header parsing for IPv6 CARP packages in tcpdump (for IPv4 CARP there is!)?

tcpdump -i <carp-iface> carp -----> only IPv4 CARP packages are shown
tcpdump -i <carp-iface> proto CARP ------> IPv4 and IPv6 CARP packages are shown

root@opnsense:~ # tcpdump -i <carp-iface> proto CARP -n
13:00:20.153815 IP6 fe80::xxx > ff02::12: ip-proto-112 36
13:00:21.177256 IP6 fe80::xxx > ff02::12: ip-proto-112 36

Why the link local address is used? My IPv6 address on the interface is fd00:1::1/64 and the virtual CARP IP is fd00:1::5/64. Maybe this behavoir is intended?!

As you can see there are outgoing IPv6 CARP packages.
PF Firewall is enabled and there is only one rule on that interface to allow IPv4 traffic from any to any.

This tests i had to run on another OPNsense machine.
I still have to test the behavior with the HA cluster from my first post.

[ppmathis]

@bewue Thank you for your findings so far. I am awaiting your test results on the cluster, as I can also see those outgoing CARP v6 packets on a standalone machine, but not on my cluster - and others from the forum encounter the same issue.

In case you do see those outgoing packets though, I'll open a new issue and we are most likely experiencing a different bug.

[ppmathis]

FWIW: While I still do not know if our issues are related, I spent some further time today with trying to track down this specific issue. Reducing the ruleset to a single rule which permits any incoming and outgoing traffic did not work either, which is why I started to analyze global pf flags.

It turned out that disabling Firewall -> Settings -> Advanced -> Multi-WAN -> Use shared forwarding between packet filter, traffic shaper and captive portal immediately restored full CARP functionality for both IPv4 and IPv6.

Behind the scenes, this will disable the two sysctl variables net.pf.share_forward and net.pf.share_forward6, which are being configured here: https://github.com/opnsense/core/blob/master/src/etc/inc/filter.inc#L469-L471

[bewue]

@ppmathis Good news, I will test that too. (at the earliest next tuesday)

[bewue]

For CARP IPv6 address functionality the deactivation of shared forwarding was the solution for me as well!

[andreas-p]

The answer is probably in #3468.
Firewall -> Settings -> Advanced -> Multi-WAN -> Use shared forwarding was never enabled on my firewalls, so net.pf.share_forward is 0.

[mimugmail]

@bewue @ppmathis which switches are between your firewalls? IPv4 and IPv6 are all static IP interfaces? Something else different from default install?

[ppmathis]

@mimugmail The setup involves two OPNsense firewalls running on KVM instances using Proxmox and are directly connected through an usual Linux Bridge with VLAN support - as offered by Proxmox through its webinterface. Connection path looks like this:

FW1 with CARP <-> OPNsense VLAN interface <-> KVM TAP interface <-> Linux Bridge <-> KVM TAP interface <-> OPNsense VLAN interface <-> FW2 with CARP

The issue also occurs with both VMs running on a single Proxmox hypervisor instead of a cluster. No specific deviation from default config aside from interface configuration and firewall rules. I also already tested both available vNIC models, namely virtio and e1000.

[andreas-p]

Try
opnsense-patch c5d6b6c , see discussion in #3468

[ppmathis]

@andreas-p @mimugmail Thanks for the hint! I just re-enabled shared forwarding, waited for the cluster to break apart (had to reboot the systems to get the same effect as in the past), applied the patch to both cluster systems and finally ensured by running pfctl -s rules that the responsible firewall rule has been updated on both systems:

Old: pass in log quick proto carp all keep state
New: pass log quick proto carp all keep state

I can confirm that patch c5d6b6c fixed the issue mentioned earlier on my system. I tried various situations and all of them worked just fine - I was not able to produce a duplicate CARP master state as before and both nodes seem to properly receive CARP traffic from each other.

@bewue Can you attempt the same fix on your cluster?

[mimugmail]

Really nice, thx :)

[fichtner]

Closing as per https://github.com/opnsense/core/blob/master/CONTRIBUTING.md