Multiple Gateway Bug on OPNsense 18.7.5_1-amd64

[sachaz]

Dear all,

since last upgrade I have several issues on all my OPNsense firewalls with 2 PPPOE:

• OPNSense 18.7.5_1-amd64:
• 2 PPPOE links
• routing depending on firewall rule selecting a gateway (pppoe0 or pppoe1)
  All this worked fine between the upgrade :/

netstat -rn show me only pppoe0
public ips are on lo0

My ADSL provider gives me public ip, but they got the same gateway.

I cannot route traffic to the non default gateway with packet filter

This is a major issue from my point of view

[fichtner]

Here are candidates from the 18.7.5, assuming you were on 18.7.4 before:

o backend: fix CPU hogging when reading on already disconnected streams
o ports: mpd5 upstream MTU fix

You can switch back to older mpd5 via:

# opnsense-revert -r 18.7.4 mpd5

You can undo patch 942ddc9 for the backend via:

# opnsense-patch 942ddc996

Both require internet connectivity.

From my point of view it's unclear what caused this disruption so let's suspend judgement. :)

[fichtner]

PS: you can also revert to 18.7.4 core package directly

# opnsense-revert -r 18.7.4 opnsense

[sachaz]

Thanks for your fast answers, rollback will be useful !
I'm not sure I was on 18.7.4 is there somewhere any history of the updates ?

[fichtner]

there should be a file /usr/local/opnsense/version/opnsense.last but it has been deprecated with 18.7.5 now so it may not be accurate or available.

[sachaz]

why this issue is tagged as support and not a bug ?

[fichtner]

because you receive free support. if we know what to fix or add, we will change it to a bug or feature. :)

[sachaz]

sorry for me, the tagg of the ticket should be a bug once it is qualified, not a support it confusing somebody who would like to correct an issue. It's like you manage the ticketing as a private project :/

[mimugmail]

You receive the same gateway for both pppoe connections?

[sachaz]

yes this is the same provider with the same gateway but different public IP on each PPPOE.

[mimugmail]

And this worked with pre 18.7.5 or did you receive different gateways before perhaps?

[sachaz]

The configurations didn't changed nope

[mimugmail]

Yes, but the gateway you receive via LCP is a decision by the provider ...

[sachaz]

yes sure and ?

[mimugmail]

I just try to understand the problem.
So, when you revert back to 18.7.4 all works fine again?

[sachaz]

I just tested:
# opnsense-revert -r 18.7.4 opnsense
the route-to rules to pppoe[0-1] are working again

[fichtner]

There are two firewall changes in 18.7.4 -> 18.7.5:

5f5f8bf8
027d8fc8

Second one being the more likely candidate.

Can you grep and provide the route-to rules in the working state?

# grep route-to /tmp/rules.debug

[sachaz]

yes sure:

pass out log route-to ( pppoe0 $IP_GW ) from {pppoe0} to {!(pppoe0:network)} keep state allow-opts label "let out anything from firewall host itself"
pass out log route-to ( pppoe1 $IP_GW ) from {pppoe1} to {!(pppoe1:network)} keep state allow-opts label "let out anything from firewall host itself"
# pass in quick on igb0_vlan100 route-to ( pppoe1 $IP_GW ) inet from {(igb0_vlan100:network)} to $Video_Wildix keep state set prio (4, 4) label "USER_RULE" # e7509540f71ef91623f1a3c10cf06b72
pass in quick on igb0_vlan100 route-to ( pppoe0 $IP_GW ) inet from {(igb0_vlan100:network)} to {any} keep state label "USER_RULE: Default allow LAN to any rule" # 1a8e19dfe7c0b041baf28059c0db7cc2
# pass in quick on igb0_vlan30 route-to ( pppoe1 $IP_GW ) inet proto tcp from {(igb0_vlan30:network)} to {!(igb0_vlan30:network)} keep state label "USER_RULE" # af7c0fe3720b8275c375d4a5f3c75acb

[fichtner]

@sachaz is this still the case on 18.7.7 ?

[stevenfoong]

the problem remain at version 18.7.7 .

[sachaz]

An idea of where the issue is coming from ?

[mimugmail]

Firewall : Settings : Advanced : Disable force Gateway ... ticked or not?

[stevenfoong]

Disable force Gateway is unchecked.

[mimugmail]

Can you enable it for testing? I usually always enable it on Multi WAN setups, otherwise you might have problems with port forwards on both WANs

[fichtner]

Timeout.

[sachaz]

Disable force Gateway is checked
still not working
a rule with a specific gateway => flow outgoing but not incoming