CARP using wrong multicast MAC dst addr

[andreas-p]

I've been experiencing carp problems since the backup firewall didn't receive the carp packets sent by the master firewall, unless the firewall was disabled with pfctl -d. This was ultimately fixed for me by disabling IGMP snooping on the switches connecting both firewalls.
Might be the same problem as described in #3411

Later on, I performed another test with 19.1.x to check what's different when pf is enabled or disabled; turns out that the Multicast MAC address is different:

While the source address is 00:00:5e:00:01: as expected in both cases, the destination MAC will be 01:00:5e:7e:78:61 with pf enabled, and 01:00:5e:00:00:12 with pf disabled.

According to RFC1112 6.4, the destination MAC address should be 00:01:5e:00:00:12, since the destination Multicast IP address is 224.0.0.18, so I consider the dest mac used with pf enabled incorrect (which seems to be derived from the default gateway IP addr). Apparently, some IGMP capable switches don't like this.

amendment: Some LAN interfaces show correctly addressed multicasts, and some don't (those which have a gateway in the subnet)

[mimugmail]

This was ultimately fixed for me by disabling IGMP snooping on the switches connecting both firewalls.

I don't really see the problem here? Does it work now for when igmp snooping is disabled?

[andreas-p]

It works for me at the moment (until a firmware update on those switches I don't control make them filter again), but the MCast addresses are clearly wrong.

[mimugmail]

Some months ago I had a replacement of customer Firewalls with OPN and also new DELL switches, also including CARP. Funny thing is, only the interface where a gateway was present (WAN) the VIP was master on the correct device. All other interfaces had split brain. The only difference was that internal interfaces were on a switch with multiple uplink switches and via port mirror I could see that the Multicast packet was sent to the wrong switches.
In my situation it was working again by setting ip igmp snooping unregistered floodall
Also when you search for such problems they exist in pfSense and vanilla FreeBSD.
Maybe it's worth to set up a vanilla FreeBSD because it might be upstream problem which OPNsense devs have too few ressources fixing theirself.

[andreas-p]

I've noticed reports about similar carp problems as well, one other indication towards pfctl -d/-e, but noone noticed the wrong multicast address. I bet you'd dianose the same broken packets on your customer's firewalls.
Setting up vanilla FreeBSD myself is out of my scope, but I already posted the bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234472 a while ago; just updated subject and comment with newest information there. So this issue will probably get the status 'waiting for upstream fix'.

[mimugmail]

Related to this post https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234472#c10 ... which policy routes are you using? On my test system without any PBR there are no route-to rules and so I'm not affected.

@AdSchellevis @fichtner When carp enabled there's a auto filter rule
pass in log quick proto carp from {any} to {any} label
Maybe we also need somthing for outgoing packets as well.

@andreas-p Can you check if a pass out with proto carp instead of multicast would also fix your issue?

[AdSchellevis]

@mimugmail outgoing should already be free

core/src/etc/inc/filter.lib.inc

Lines 519 to 525 in 4615085

	// out from this Firewall
	$fw->registerFilterRule(
	5,
	array('direction' => 'out', 'statetype' =>'keep', 'allowopts' => true,
	'quick' => false, "descr" => "let out anything from firewall host itself"),
	$defaults['pass']
	);

[andreas-p]

@AdSchellevis , that rule is not quick, so the route-to is applied anyway.
Finally, the fix is
pass out quick carp from {(self)} to 224.0.0.18

[AdSchellevis]

@andreas-p you're right, the route-to rule seems to take precedence, we should add the outbound rule indeed.

[AdSchellevis]

c5d6b6c should make sure carp avoids route-to.

opnsense-patch c5d6b6c

[andreas-p]

I tested and can confirm that the patch fixes the issue.

[AdSchellevis]

@andreas-p thanks, so we can close this one?

[andreas-p]

Yes, done.

[mimugmail]

Thx @AdSchellevis for fixing and @andreas-p for reporting :)