net/frr: ospfd generated rules should be extended

[cultcom]

The generated rules for OSPF do not include "inet6" rules and do not allow unicast updates between the peers.

Oct 27 15:37:07 fw-d-test filterlog: 3,,,0,vmx0_vlan1451,match,block,in,4,0xc0,,1,13002,0,none,89,ospf,208,x.y.z.65,x.y.z.94,datalength=188

As it appears some updates are send directly between the peers by unicast.

[fabianfrz]

I added a dev branch to the repository with an additional firewall rule for unicast:
You may try it using
opnsense-patch -c plugins c0c1b0f

You can see the diff here: https://github.com/opnsense/plugins/compare/quagga_fw_rule?expand=1

If it works, I will merge it to master and it will go to stable soon.

[fabianfrz]

@cultcom Please also add some block samples for IPv6 so I can add IPv6 rules too.

[cultcom]

@fabianfrz what I am missing is an outgoing rule for the multicast traffic.

Here the init state is clearly described as that the process receives hello packets but they do not contain the router id of the host itself:
https://www.cisco.com/c/en/us/support/docs/ip/open-shortest-path-first-ospf/13685-13.html

For me that means, that the other routers do not see the hello packets from the opnsense at all. So outgoing packets are dropped somewhere.

After applying the patch I see a new pf rule but nothing changes so far.

[cultcom]

foreach ($ospf->networks->network->__items as $network)

Only the interface network addresses on which the ospfd is talking should be allowed here. The network statements are for route summarizations at ABRs when I remember correctly.

[fabianfrz]

the networks are all addresses, which are registered to the OSPF daemon (it will search for neighbors and sends routing updates unless the interface is configured as a passive interface).

OPNsense should contain already a rule covering this:

[cultcom]

yes I do see that rule but something is wrong with it:

pass out log all flags S/SA keep state allow-opts label "let out anything from firewall host itself"

When I dump packets on the OSPF interface like this:

tcpdump -nnvi vmx0_vlan1451 proto ospf and host x.y.z.94

I get correctly looking hello packets:

16:42:33.250517 IP (tos 0xc0, ttl 1, id 55212, offset 0, flags [none], proto OSPF (89), length 92)
    x.y.z.94 > 224.0.0.5: OSPFv2, Hello, length 56
	Router-ID x.y.z.94, Area 0.0.0.4, Authentication Type: MD5 (2)
	Key-ID: 1, Auth-Length: 16, Crypto Sequence Number: 0x5a0325e9
	Options [External]
	  Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.224, Priority 1
	  Designated Router x.y.z.94
	  Neighbor List:
	    x.y.z.249
	    x.y.z.91
	    x.y.z.93

But on another host within the same segment nothing is coming in:

tcpdump -nnvi eth1.1451 proto ospf  and host x.y.z.94
tcpdump: listening on eth1.1451, link-type EN10MB (Ethernet), capture size 262144 bytes
<<<<< emptiness >>>>>

When I disable the firewall with "pfctl -d" almost instantly the packets arrive on the peer hosts.

For testing I installed a pfSense VM and configured quagga on it and I notice slightly different outgoing rules:

pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"

So there is definitely something blocking the packets from leaving the VM. Any suggestions on how to debug this further?

[mimugmail]

@AdSchellevis @fichtner I remember an isse (for forum post) the last days regarding CARP not sending outgoing packets and with pfctl -d working again. It think it was in combination with upgrade from 17.1 to 17.7.
Were there some changes in the default behavior so something could work with fresh install 17.7 but when upgrade and old value in DB not?

[AdSchellevis]

I don't think so, but if it is, the changes to the default config are here https://github.com/opnsense/core/commits/17.7.7/src/etc/config.xml.sample

[cultcom]

I'm testing on a freshly installed VM and did not upgrade yet.

[mimugmail]

@AdSchellevis https://forum.opnsense.org/index.php?topic=6228.0

[AdSchellevis]

oops, likely opnsense/core@632d887

opnsense-patch 632d887d
and reload filter to try it out.

[cultcom]

nope no change yet.
I even rebooted to be sure but the hello packets are still dropped.

[mimugmail]

Great, but doesn't solve this one with proto ospf .
Is default out direction pass only tcp/udp or also include other proto numbers?

[mimugmail]

@cultcom this would fix the bug in the Forum ... but we get closer to this one :)

[cultcom]

well, I pasted the rule 2 hours ago. That is all I have.

[cultcom]

Another fun fact: I have another production VM which is working, but that was upgraded from an earlier version. Maybe that gives you a hint.

[AdSchellevis]

Well, that's odd. you probably have to compare both pf rulesets to know for sure what's different there. The "let out anything from firewall host itself" on both machines can't be different, so there must be some other rule causing your issues.

[cultcom]

the out rule look exactly the same

pass out all flags S/SA keep state allow-opts label "let out anything from firewall host itself"

I do not really believe that it relates to the rules at all? Are there other possibilities like kernel flags or anything?

[AdSchellevis]

If it's generating filter log messages, I don't expect so.

Can you check the log again for the offending rule (first item in the log line)? then check the details of that rule using

pfctl -vvPnf /tmp/rules.debug

[cultcom]

I recently tried this without success:

root@fw-d-test:~ # pfctl -sr
pass in all flags S/SA keep state
pass out all flags S/SA keep state

[cultcom]

Unfortunately I do not see anything logged. I see the hello packets with tcpdump on the test machine but I do not see them arrive on the peers.

I do see them with "pfctl -d".

[AdSchellevis]

Maybe it's a user defined rule, when logging is enabled it should generate messages for the default rules. With the risk of generating a lot of log traffic, you could add log info to all your rules and see if anything pops out.

I can't help you any further here, maybe someone else steps in.

[cultcom]

Well, with those two rules that allow everything in and out I don't see the rules involved anymore. Maybe I find time to continue debugging on Friday. I'm happy for every input that might help to find a solution.

[AdSchellevis]

Last check, the settings in system_advanced_firewall.php, are they equal for the functional and non functional machine?

[mimugmail]

And diff /tmp/rules.debug on both machines please ...

[cultcom]

funny, both are 17.7.7_1 but the dialogs in system_advanced_firewall.php differ. The working machine has a section "Network Address Translation" between IPv6 and Bogon.

But NAT is disabled on both.

[cultcom]

any new ideas on this? I compared all the rules up and down, but even with a "pass all" rule it does not work. It has to be something different.

[cultcom]

I recently tried to this with current version 17.7.11 and still have the same issues. I am definitely sure it has nothing to do with the pf rules themselves. The OSPF exchange does not work with pf enabled even in "pass everything" mode.

[cultcom]

Today I set up a new 18.7.10_3 VM with OSPF enabled on the WAN side and noticed an issue with the generated rules:

# Generated catch-all rule
pass out log route-to (vmx0_vlan504 x.x.x.x) inet from x.x.x.136 to ! (vmx0_vlan504:network) flags S/SA keep state allow-opts label "let out anything from firewall host itself"
  
# Custom rule to allow outgoing OSPF traffic
pass out quick on vmx0_vlan504 reply-to (vmx0_vlan504 x.x.x.x) inet proto ospf all no state label "USER_RULE"

As you can see there is a "reply-to" included which stops OSPF sending HELLO packets to 224.0.0.5

Due to the fact that I need a static route before I can setup FRR this is a chicken-egg-problem.
With a static route on WAN the rule is generated with "replay-to" so OSPF will not work.
Without a static route I cannot install and setup OSPF.

So the frr package should always add a generic outgoing rule for OSPF packets, maybe like this:

pass out quick on vmx0_vlan504 inet proto ospf from (self) to 224.0.0.4/30 keep state label "USER_RULE"
pass out quick on vmx0_vlan504 inet proto ospf from (self) to (vmx0_vlan504:network) keep state label "USER_RULE"

[fabianfrz]

@cultcom this rules should not be a problem to add those two rules. I can probably get them into the next release.

[fabianfrz]

@cultcom your rules will be released soon.

[cultcom]

Did you read my comments on PR #1149