default route gets lost when "Periodic interface reset" is run

[nscheer]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Description

I stumbled upon this problem recently. I can't really tell when it started, the issue existed in previous versions as well.

I have a cron job running, that executes "Periodic interface reset" for "wan" at 02:00 in the night.
Every morning I notice, that the internet connection is not working. Looking at the web gui, it is obvious that the default route is missing (system / routes / status). Apart of that, everything else seems to be fine - gateway is up and running.

I'm using a rather standard German DSL setup (pppoe for IPv4, DHCPv6 for IPv6)

To Reproduce

Steps to reproduce the behavior:

1. run a cronjob with "Periodic interface reset"
   or
2. run "rc.configure_interface wan" in a shell

Expected behavior

Expected behavior is that internet connection is working again after command completes.

Describe alternatives you considered

As a workaround it is possible to restart the "Gateway monitor watcher" - the default route will then be re-added and everything starts to work again.

Screenshots

There are no applicable screenshots.

Relevant log files

From what I could tell, there is nothing in the logs that reveals this issue. Can post logs, if needed.

Additional context

At the beginning, I thought this was related to me adding a second gateway using the third network port on the APU2. I added a Huawei LTE route as backup. But I disabled everything and the problem still persisted.

Environment

Running on a PCEngines APU2 (apu2e4)

opnsense version info:

Type: opnsense	
Version: 24.7_5	
Architecture: amd64	
Commit: 6578c5a18	
Mirror: https://pkg.opnsense.org/FreeBSD:14:amd64/24.7	
Repositories: OPNsense	
Updated on: Sun Jul 28 23:53:49 CEST 2024

Analysis

From what I can tell the "Periodic interface reset" cron action runs the same code as the script rc.configure_interface.

This script basically only runs a single function configure_interface:

core/src/etc/rc.configure_interface

Line 41 in 4501c59

interface_configure(true, $argument, true);

During the course of this function, there is a call to interface_ppps_configure(), which is executed, because I use pppoe:

core/src/etc/inc/interfaces.inc

Lines 2340 to 2353 in 4501c59

	switch ($wancfg['ipaddr'] ?? 'none') {
	case 'dhcp':
	interface_dhcp_configure($interface);
	break;
	case 'l2tp':
	case 'ppp':
	case 'pppoe':
	case 'pptp':
	interface_ppps_configure($interface);
	break;
	default:
	interface_static_configure($interface, $wancfg);
	break;
	}

Inside interface_ppps_configure() there is a loop, that is supposed to wait, until the interface shows up:

core/src/etc/inc/interfaces.inc

Lines 1251 to 1260 in 4501c59

	/* wait for up to 10 seconds for the interface to appear (ppp(oe)) */
	$i = 0;
	while ($i < 10) {
	exec("/sbin/ifconfig " . escapeshellarg($ppp['if']) . " 2>&1", $out, $ret);
	if ($ret == 0) {
	break;
	}
	sleep(1);
	$i++;
	}

Here seems to be the culprit:
On my system, the shell return code for the first loop iteration is 0 and the output is as follow:

Array
(
    [0] => pppoe0: flags=8890<POINTOPOINT,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1500
    [1] =>      options=0
    [2] =>      nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
)

Thus, the interface is there, but not UP, yet.

Back in configure_interface, there is a call to system_routing_configure that should set the default route:

core/src/etc/inc/interfaces.inc

Line 2414 in 4501c59

system_routing_configure($verbose, $interface);

For setting the default route, the gateways are fetched:

core/src/etc/inc/system.inc

Line 664 in 4501c59

$gateways = new \OPNsense\Routing\Gateways();

But, if there are no gateways, the function system_default_route() won't get called:

core/src/etc/inc/system.inc

Lines 698 to 708 in 4501c59

	if (isset($config['system']['gw_switch_default']) || empty($interface) || $interface == $gateway['interface']) {
	if (empty($ifdetails[$gateway['if']][$type][0])) {
	log_msg("ROUTING: refusing to set {$ipproto} gateway on addressless {$gateway['interface']}({$gateway['if']})", LOG_ERR);
	continue;
	}
	log_msg("ROUTING: configuring {$ipproto} default gateway on {$gateway['interface']}", LOG_INFO);
	system_default_route($gateway, $routes);
	}
	}

Analyzing further, at the time of creation the object OPNsense\Routing\Gateways does not return a gateway for the pppoe interface which results in the default route not to be set, thus leaving my system without a default route.

Now, if I add just some seconds more wait time for the loop that waits for the interface to be visible using ifconfig, the output changes to this:

Array
(
    [0] => pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
    [1] =>      options=0
    [2] =>      inet6 fe80::20d:b9ff:fe4f:b17c%pppoe0 prefixlen 64 tentative scopeid 0x8
    [3] =>      inet6 2001:9e8:4040:ddec:20d:b9ff:fe4f:b17c prefixlen 64 tentative autoconf pltime 172800 vltime 259200
    [4] =>      nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
)

It seems that it just takes some time for the interface to be UP and have a gateway. Before, the call to getDefaultGW() does not return anything (i.e. the gateway entry in the return array is missing).

ifconfig returns a 0 on the command line even if the interface is not up - which is to be expected. What works for me is adding the -u flag to ifconfig, thus only listing UP interfaces and comparing the exec output to make sure that the interface is actually listed.

I'll link a PR in a few minutes.