"reply-to" feature (enabled by default) breaks RFC1122 section 3.3.1.1 (local/remote decision)

[ctminime]

Connections from WAN network IP to WAN interface IP don't work when an allow rule is created and the reply-to option is not disabled.

In my scenario, I was testing OPNsense in a lab environment. After a lot of beating my head against the wall and forum searching and then back and forth in the forum, I discovered that this behavior actually breaks RFC1122 section 3.3.1.1. It has been mentioned that this is potentially an upstream bug, but there has been no evidence given that it is an upstream bug. This topic was discussed on the forum here:
https://forum.opnsense.org/index.php?topic=15900.0

In my my test case (as shown in Wireshark screenshots) other computer has IP 192.168.169.200 and OPNsense WAN interface has IP 192.168.169.191.

To Reproduce
Steps to reproduce the behavior:

1. Fresh install of OPNsense
2. WAN network has other IPs that will access the WAN interface (ie. SSH)
3. In my test setup WAN interface receives a private IP via DHCP
4. If WAN has private IP, allow Private IPs and Bogons
5. Gateway is stateful firewall (ie. OPNsense)
6. Create rule allowing something (ie, SSH,) from WAN network to WAN interface.
7. For my test... Enable SSH access.
8. Run tcpdump on WAN interface collecting for port 22 and write to file.
9. From another computer on WAN network SSH to WAN interface. This will time out.
10. In packet capture, you should see a SYN from other computer to WAN interface
11. The SYN ACK that the WAN interface responds with, sends to the gateway MAC address instead of the MAC address of the other computer.

NOTE: disabling reply-to on the firewall rule allows a computer on the WAN network to connect. However, since this behavior breaks RFC, it is my opinion that this needs to be addressed despite a checkbox work-around.

Expected behavior
Even with reply-to enabled (which is default), the test scenario should be functional instead of timing out.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
The version below was the only one tested.

Environment
Software version used and hardware type if relevant.
e.g.:

OPNsense 20.1.1-amd64
FreeBSD 11.2-RELEASE-p16-HBSD
OpenSSL 1.1.1d 10 Sep 2019

[mimugmail]

The problem is not that reply-to breaks RFC, the problem is that reply-to is default on.

But turning it off (checkbox enabled) will break more setups than it will help people fighting with this problem. If counted right you were the third guy asking about it and the support time this will produce when changing a default may not worth it, or better invested somewhere else (like API enabled firewall rules way more ppl ask for).

Just my personal opinion.

[pv2b]

Thanks for writing up a bug report for this. I'm the original poster of the forum thread, and I completely agree the way OPNsense does this is bogus. The fact that it's been bogus for a very very long time (way pre-fork from pfSense) doesn't change that. :-)

Just my two cents:

This is not an upstream bug. FreeBSD is doing exactly what the pf rule says. The rule says to send the reply traffic to the gateway, and that's exactly what's happening. You might argue that reply-to is a misfeature, and that there should be some handy flag to exclude replies to the same subnet, but it's certainly not a bug.

If this is a behaviour that should change, that change should happen in what pf rules are generated by OPNsense, not in how pf itself acts. For example, by generating one rule with reply-to for traffic coming from outside the WAN subnet, and one without reply-to set for traffic coming from inside the WAN subnet.

Fortunately, if you want this to happen, it's already possible to do this yourself in the OPNsense UI. It'll require duplicating all rules on the WAN, so it's super ugly, but it's possible. Which is useful if you actually want to do multi-WAN and use reply-to. IMO OPNsense should just do this behind the scenes. It won't break any reasonable setup.

[fichtner]

Misfeature, maybe, maybe not as the use case has been established many years ago. To make this discussion somewhat useful, I would like to make an actual suggestion: If we only have at most one gateway-capable interface (that includes user-created gateways) we disable reply-to as we don’t have a multi-wan setup anyway. I have no idea if this is feasible, but it’s certainly more useful than arguing do and don’t.

…

On 27. Feb 2020, at 13:04, pv2b ***@***.***> wrote:  Just my two cents: This is not an upstream bug. FreeBSD is doing exactly that the pf rule says. The rule says to send the reply traffic to the gateway, and that's exactly what's happening. You might argue that reply-to is a misfeature, but it's certainly not a bug. If this is a behaviour that should change, that change should happen in what pf rules are generated by OPNsense, not in how pf itself acts. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[mimugmail]

If it doesn't touch existing installs and only new ones, why not. Maybe need some small documentation updates.

I think the most clean way would be to negate the WAN net described here, but I have no idea if this is possible with pf:

https://forum.opnsense.org/index.php?topic=15900.msg73203#msg73203

[pv2b]

I think it's good to talk about solutions, @fitchner. My suggestion is to change OPNsense rule generator to apply reply-to for packets only coming from different subnets. Something like this.

Before change:

pass in quick on igb1 reply-to { igb1 192.0.2.1 } inet proto tcp from {any} to {{self}} port {443} keep state

After change:

pass in quick on igb1 inet proto tcp from {{igb1:network}} to {{self}} port {443} keep state
pass in quick on igb1 reply-to { igb1 192.0.2.1 } inet proto tcp from {any} to {{self}} port {443} keep state

This would not break any reasonable setups, while still doing what the feature is intended to do in the first place.

If you're still afraid of breaking existing setups, make it an optional checkbox only checked by default in new installs, not on existing ones.

Combine this with the idea of only enabling this feature once you actually have multi-wan and you won't double the number of firewall rules in the pf ruleset for most users.

[AdSchellevis]

Can we please stop with this? We're not planning to expand the problem by generating more rules auto-magically than what it already does (error-prone and has other visibility issues) and we're not changing the defaults now, maybe in time, you never know, but certainly not now.

How it works is documented behaviour on our end, as stated multiple times, we would have done it differently if we would have designed it from the start, but looking at our support desk, the number of questions flowing in regarding this is very low and would certainly be way more when changing the current behaviour.

[ctminime]

I was able to confirm this behavior does indeed exist in FreeBSD 11.3. So, I opened a bug report there as well.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244514

I will say that I totally understand that there are only a handful of use cases that will be affected by this. So, when determining where to invest DEV time, this probably falls to the bottom of the list. And now that I know about this issue and its work around, I am OK with with that. Especially since I am also a person who would love to see API enabled firewall rules.

During this whole thing I have had 2 major issues with this:

1. Just because this is the way it has worked for a decade+, doesn't mean it is right or should stay that way. Especially since that RFC was written in 1989.
2. I understand that Multi-WAN is an advanced configuration and since "reply-to" is meant for those scenarios, that is where the option is (in the advanced section in a firewall rule) and why the documentation for reply-to is in the Policy Based Routing section of the documentation. However, I was NOT doing any advanced setup NOR does routing come into play for a connection between 2 IPs on the subnet (or shouldn't). And I was extremely frustrated trying to figure out what the hell was going on. Especially since networking is what I do for a living.

So @AdSchellevis, when I have the time, I will do a PR to update the help dialog for the reply-to option. I will also see if I can come up with something for the OPNsense documentation as well. So that way, I am making "useful additions" instead of just complaining.

[ctminime]

The bug report at freebsd got closed almost immediately saying "works as intended" I have re-opened it requesting that it be left open for at least a week for commenters.

@pv2b @fichtner @AdSchellevis @mimugmail , it would be great if each of you could comment either for or against.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244514

[AdSchellevis]

I'm most definitely not going to comment on a this report, for reasons explained earlier.

Let's go back one step, if you (all) check "Disable force gateway" in Firewall --> Settings --> Advanced, is the behaviour as you would expect it to be? (Which has been my assumption)

[AdSchellevis]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

[Peter2121]

I would like just to mention a real problem introduced by this behavior:
https://forum.opnsense.org/index.php?topic=23144.0

I don't think that the problem is at FreeBSD level as pf rules are created by OPNsense. The fact that FreeBSD permits to overcome the 'standard' behavior is just a 'feature' of the OS. The fact that OPNsense generates the pf ruleset that uses this feature by default is the bug in OPNsense.

IMHO, this feature should be disabled by default and should be better commented in OPNsense web interface.

UPDATE: It seems that there is no way to get working the VPN between two peers placed in WAN network as the default rules for IKE/NAT-T traffic always enforce 'reply-to'. PLEASE FIX IT!!

[pv2b]

@Peter2121 If I'm not mistaken, user-specific rules come before the auto-generated IPsec rules. So just add a rule like this:

Interface: WAN
From: WAN Network
To: WAN Address
Protocol: UDP
Port: 4500
Reply-to: Disabled

Repeat for IKE traffic
This should then override the default WAN rule and get your stuff working... I've not tested this myself, but this should work around this bug.

[Peter2121]

@pv2b
That's the type of rules I've tried to use. I created two rules like this - for IKE and for NAT-T.
Unfortunately, there was no effect.

[pv2b]

Can you please post a copy of your /tmp/rules.debug here?

Make sure to redact any private information you don't want shared on a public forum.

[Peter2121]

@pv2b
I found the difference between the rule you proposed and my ones - I used 'Floating' section and 'Any' source/dest.
Changing the rules to describe exactly the traffic, I could get up the tunnel.
@pv2b, thanks a lot for your assistance!

[Peter2121]

Please, somebody, reopen this bug!

[AdSchellevis]

not planning to, no, #3952 (comment)

Whatever we do with reply-to/route-to, it will almost always lead to someones functional setup ending in a non functional state. If there's anything new / a proposed solution that would make sense, just open a new issue containing relevant details.

[Peter2121]

@AdSchellevis
I don't know if you've correctly read my posts, but disabling reply-to in advanced settings of 'Firewall' section has no effect on implicit rules for IKE/NAT-T.
Once again, the option does not work as expected. So it is a bug.
If you consider that this option should not affect implicit rules - OK, so we need another option, just for implicit rules.
In the actual state we have OPNsense that breaks RFC, and there is no evident option to workaround this behavior. I perfectly understand that changing defaults can break working installations, and this is NOT a way to go. We just need a option (maybe unchecked by default) to get things worked as 99.9% of network engineers are expecting.

[AdSchellevis]

?

In which case it's this ticket #4942

[Peter2121]

OK, indeed, #4942 makes sense.

[AdSchellevis]

just try to remember for next time, first check the tickets and read previous comments before shouting a ticket should be reopened....

[pv2b]

not planning to, no, #3952 (comment)

Whatever we do with reply-to/route-to, it will almost always lead to someones functional setup ending in a non functional state. If there's anything new / a proposed solution that would make sense, just open a new issue containing relevant details.

If you're interested in solving this, I think you should reconsider my proposed solution at #3952 (comment) - it's been over a year since we talked about this, and perhaps your perspective now is different than then. I hear your suggestion about opening a new ticket, but since I'm not really bringing a new solution to the table, I thought it's better to keep it here so we don't clutter the issue system.

I agree: This solution may be slightly hacky in terms of creating multiple rules on the back-end, but it has the advantage of actually doing what the feature is actually intended to do, which is to make sure reply traffic egresses on the same interface it comes in, and works properly even for same-subnet traffic.

So from the perspective of it being a solution to the actual problem, it is valid. I don't think you'll find another solution, unless somehow you add more functionality to FreeBSD's pf, which I really doubt will happen.

As for the argument about "magic" behind the scenes, it's IMO normal for an abstraction layer like OPNsense to hide some of the underlying complexity of pf.

If you're worried about breaking existing setups, it could perhaps be gated behind a configuration bit that would, once you're confident the solution makes sense, defaults to on on new installs, but defaults to off on importing/upgrading a configuration. I really don't think it's neccessary. Given my proposed change in behaviour it's very unlikely to break anything, but if you have a different perspective, that's a way to do it with no chance of breaking anything for anyone.

If you still think my proposed solution isn't worth doing, that's OK. You don't have to explain yourself if you still disagree for the same reasons as before.

But if you change your mind on this, I'm willing to put a few hours into getting a PR done for this issue, but not if the proposed approach is dismissed outright ahead of time, that would just be a waste of everyone's time.

[AdSchellevis]

sorry, my perspective hasn't changed. I do however think we should offer reply-to as an option in rules so one could create any variation they would like to (as suggested #4905 (comment)).

[ctminime]

I would like to know in what scenario would enforcing RFC1122 section 3.3.1.1, WITH with Reply-To checked, would break an existing OPNsense implementation? I personally can't think of anything.