Static route to route-based IPsec gateway does not get configured after reboot

[alexanderharm]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[X] I have searched the existing issues and I'm convinced that mine is new.

Describe the bug
We have setup route-based IPsec with the necessary gateway. We added a static route for our remote LAN that points to the IPsec gateway. After reboot the static route does not get applied anymore (still visible in Configuration but not in Status). Similar issue #2388.

To Reproduce
Steps to reproduce the behavior:

1. Setup a route-based IPsec (see https://wiki.opnsense.org/manual/how-tos/ipsec-s2s-route-azure.html)
2. Then reboot
3. Static route is still configured but not applied

Expected behavior
Route being set

Screenshots

Relevant log files
Cannot find anything in the logs

Environment
OPNsense 19.1.4-amd64
Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz (8 cores)
Network Intel® I210-AT

[fichtner]

So 9046727 should work for tunnels on static IPs... but it needs more work unfortunately. You can try it and let us know:

# opnsense-patch 9046727

Cheers,
Franco

[alexanderharm]

@fichtner Works for us in 19.1.4

Thanks for the quick fix.

[fichtner]

Thanks, I'll bring this workaround into 19.1.7 and will work on a better solution for 19.7.

[TP75]

Apparently, this issue is a burden to 19.7.4_1 in continuity. This is rather unfortunate as it makes more flexible VTI difficult and not as reliable as hoped.

Please be aware this may hinder deployment aims and seems to contradict the OPNsense own advertisement "to run dynamic routing protocols over the tunnel to create more redundant, or software-defined networks."

[mimugmail]

Only to clarify, dynamic routing is NOT affected by this issue, as these routes are learned dynamically by FRR and not configured via OPN itself, so the "advertisement" is still true :)

[TP75]

Only to clarify, dynamic routing is NOT affected by this issue, as these routes are learned dynamically by FRR and not configured via OPN itself, so the "advertisement" is still true :)

IMHO this may be technically correct but is quite misleading. Obviously, there is a lack of reliability to VTI and one cannot hope to ge a "more redundant" network.

Dynamic routing works (after some efforts and configuration) but apparently a VTI route fails after any reboot. The tunnel is up but routing and traffic is failing.

[mimugmail]

P.S.: my lab survived the reboot, no iusse with route-based ipsec and reboot.

[TP75]

P.S.: my lab survived the reboot, no iusse with route-based ipsec and reboot.

My findings are different. In a three peer scenario both VTI channels fail repeatedly after reboot. As a workaround we use a two hop IPsec standard tunnel, re-apply the route configuration, and only then the VTI comes up again.

[mimugmail]

Screenshots please, for you it's clear what a "three peer scenario" and a "two hop IPsec standard tunnel" means. I'd recommend you open a new issue with as much information as possible and screenshot of P1 and P2, error logs from system log and ipsec.log.

[TP75]

Screenshots please, for you it's clear what a "three peer scenario" and a "two hop IPsec standard tunnel" means. I'd recommend you open a new issue with as much information as possible and screenshot of P1 and P2, error logs from system log and ipsec.log.

Point taken. However, the core issue seems to be the same and IPSec.logs are quite exhaustive.

[fichtner]

I‘m waiting for users to help solve this by providing more test cases, not to hear how frustrating it is. The ticket is open and the issue is obvious. 😊

[dklimpel]

How can I support? I have the same problem.
My system OPNsense 19.7.4_1-amd64, OpenSSL.

I am use a VPN to Azure with static route. I have to press the "Apply"-Button (System -> Routes) after every reboot.

[TP75]

I am use a VPN to Azure with static route. I have to press the "Apply"-Button (System -> Routes) after every reboot.

Same here in the a.m. scenario but without Azure. OPNsense 19.7.4_1-amd64, LibreSSL
VTI route between two self administrated boxes. A third box allows the access to the down VTI after the daily reboot to 're-apply' the route config.

Do you need more details?

[mimugmail]

system.log while/after reboot from both machines

[dklimpel]

My system.log after reboot from my opnsense

system.log

[8191]

@fichtner I'd like to test this bugfix with a 20.7.7_1 installation. Is applying patch 4992c11 sufficient? Or were there other commits since 20.7.7 which might be required to solve the missing routes?

[fichtner]

No overlapping changes in the file, as easy as opnsense-patch 4992c11

[8191]

@fichtner I just tested a bit with the patch active and after a reboot the routes are now successfully applied (at least at two tries). But unfortunately after a WAN interface down/up even, the routes are still missing. This is now a slightly different situation. Is there a different issue already for that one?

I think an issue might be, that when the WAN IP does not change, that routes are not re-applied:

core/src/etc/rc.newwanip

Lines 151 to 165 in 6529ef7

	if (!is_ipaddr($cacheip) || $ip != $cacheip || !is_ipaddr($configip)) {
	@unlink($cacheip_file);
	system_routing_configure(false, $interface);
	plugins_configure('monitor');
	filter_configure_sync(false, isset($config['system']['ip_change_kill_states']));
	if (is_ipaddr($ip)) {
	@file_put_contents($cacheip_file, $ip);
	}
	plugins_configure('vpn', false, array($interface));
	plugins_configure('newwanip', false, array($interface));
	rrd_configure();
	}

[fichtner]

@8191 I'm aware the issue shifts to WAN reload now. Since I don't know what changes it's difficult to tell. You can try to delete the cache file and call rc.newwanip directly to force all the reloads. Although that only gives a 50% chance that it will work since system_routing_configure(false, **$interface**); may not call all the necessary steps here to fix a separate interface/VTI.

[peironem]

Hi all.
What i have to do in order to apply the patch?

[peironem]

Ok, no problem, just done.
I'll try it this evening to see if works fine.

[peironem]

@8191 Did the fix solved the problem for you?
@fichtner, with clean GRE problem is not solved.
Just applied the patch 4992c11.

I can see under root@opnsense:/usr/local/etc/rc.syshook.d/start following files

root@opnsense:/usr/local/etc/rc.syshook.d/start # ls
10-newwanip             20-freebsd              90-cron
10-newwanip.orig        90-carp                 95-beep

10-newwanip content is patched

root@opnsense:/usr/local/etc/rc.syshook.d/start # cat 10-newwanip
#!/bin/sh

REROUTE=

for IPV4 in $(find /tmp -type f -name "newwanip_*"); do
        INTERFACE=$(cat "${IPV4}")
        rm "${IPV4}"

        echo -n "Reconfiguring IPv4 on ${INTERFACE}: "
        configctl interface newip ${INTERFACE}

        REROUTE=yes
done

for IPV6 in $(find /tmp -type f -name "newwanipv6_*"); do
        INTERFACE=$(cat "${IPV6}")
        rm "${IPV6}"

        echo -n "Reconfiguring IPv6 on ${INTERFACE}: "
        configctl interface newipv6 ${INTERFACE}

        REROUTE=yes
done

if [ -n "${REROUTE}" ]; then
        # Following #3414 there is a missing link between VTI and
        # routing configuration reload so at least for reboot make
        # sure that it is properly executed even if no VTI exists

        echo -n "Reconfiguring routes: "
        configctl interface routes configure
fi

After sys reboot i can't see my GRE vti routes up, notice that they're up on GUI

root@opnsense:~ # route show 192.168.1.0/24
route: route has not been found
root@opnsense:~ # route show 172.16.10.0/24
route: route has not been found

I still have to manually disable and re-enable routes from GUI to have them to work.
Regards,
Pm.

[8191]

@peironem Does a click on "Apply" (without any changes) in the GUI add the routes?
I tested only one or two reboots and routes where applied without manual interaction.

[peironem]

@8191 Yeah, you are right.
After a reboot i have the following routing table

root@opnsense:~ # netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.28.115.105     UGS        igb0
10.10.10.1         link#9             UH         gre0
opnsense           link#9             UHS         lo0
localhost          link#5             UH          lo0
172.28.115.104/29  link#1             U          igb0
opnsense           link#1             UHS         lo0
192.168.0.0/24     link#2             U          igb1
opnsense           link#2             UHS         lo0

If i do nothing but clicking on Apply button on System > Routes > Configuration via GUI, routes actually raise up again.

root@opnsense:~ # netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.28.115.105     UGS        igb0
10.10.10.1         link#9             UH         gre0
opnsense           link#9             UHS         lo0
localhost          link#5             UH          lo0
172.16.10.0/24     10.10.10.1         UGS        gre0
172.28.115.104/29  link#1             U          igb0
opnsense           link#1             UHS         lo0
192.168.0.0/24     link#2             U          igb1
opnsense           link#2             UHS         lo0
192.168.1.0/24     10.10.10.1         UGS        gre0

Any ideas?

[fichtner]

It's the same issue but your boot does not trigger reload in 10-newwanip, maybe because of static WAN config?

[peironem]

It's the same issue but your boot does not trigger reload in 10-newwanip, maybe because of static WAN config?

@fichtner Yeah, sure, my WAN is configured as static IPv4 interface facing on MPLS network with an isp given IP.
Do you have any fix or workaround for that?

[fichtner]

@peironem not yet, but your setup is simple enough to find out why I hope... So the odd thing is we configure interfaces including GRE here:

core/src/etc/rc.bootup

Line 98 in 1a646e0

interfaces_configure(true);

And then later run the code that should put the routes in place:

core/src/etc/rc.bootup

Line 104 in 1a646e0

system_routing_configure(true);

But at the end of the boot sequence the routes are not there, which means:

• The initial configuration does not work (1), or
• Something messes with the routing table after routes were set correctly. (2)

You say applying the routes from System > Routes > Configuration works. Does running /usr/local/etc/rc.routing_configure have the same effect? Trying to rule out (1) here...

Cheers,
Franco

[8191]

@fichtner Wouldn't it be better to focus on a solution with IPsec tunnel establishment than with WAN interface startup?

[fichtner]

#3414 (comment)

[peironem]

@fichtner just tried.
I can confirm that running rc.routing_configure under /usr/local/etc/ has the same effect as using apply button via GUI

[peironem]

Hi @fichtner,
do you need any extra test or the issue is now clear?
Notice that i've just updated the system from OPNsense 20.1.8_1 to OPNsense 20.7.7_1, issue still there.

Thanks in advance.

[peironem]

Just a little update here:
@fichtner i just tried with pfsense and vty routing works fine, that may help as long as opnsese is a pfsense fork.

[AdSchellevis]

highly unlikely, non of that code originates from there....

[McMarius11]

The problem still persists, you can test it by just restarting IPsec VPN then the route is gone and you have to click apply to add it back. a small workaround but hopefully fixed soon

[wobblywob]

Can confirm this bug on Opnsense 21.1.4. I've encountered it while troubleshooting Routed IPsec + FRR OSPF here

[mimugmail]

5 minutes ago I update a live OPNsense from 21.1.2 to 21.1.5 which has a route based IPsec to GCP, after reboot the tunnel is up, traffic is there and static routes are in kernel.

[FootStark]

As some form of mitigation, would it be possible to attach the [routes.configure] action to the [start], [restart] and [reconfigure] actions in src/opnsense/service/conf/actions.d/actions_ipsec.conf?
Maybe for a cleaner solution, a check on devd should be done if any device events are signalled when the routed IPSEC interface comes up...

[fichtner]

I have added an unconditional routing reconfigure after boot to address this via 3e4df24

No need to keep this ticket open any longer. If someone has a way to reproduce the actual issue while configuring a VTI tunnel with a static route let us know.