The IPv6 gateway is reported as offline if the WAN interface has a ULA address.

[subnetspider]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

IPv6 gateway reported as offline when there is a ULA address on the WAN interface because dpinger binds to the wrong address.

To Reproduce

Steps to reproduce the behavior:

1. Do a fresh install of OPNsense and run the setup wizard.
2. Connect the WAN interface to a router that sends a ULA prefix (fd00::/8) in addition to the GUA prefix (2000::/3).
3. Got to "System > Gateways > Single".
4. Edit the gateway WAN_DHCP6 (or WAN_SLAAC when using SLAAC).
5. Remove the tick from the Disable Gateway Monitoring checkbox.
6. Enter a IPv6 GUA address in Monitor IP like 2001:4860:4860::8844.
7. Klick "Save" and "Apply", then refresh the page.

Expected behavior

The IPv6 gateway WAN_DHCP6 should be Online and RTT and RTTd should show some latency.

Describe alternatives you considered

The IPv6 gateway WAN_DHCP6 is reported as Offline with 100% packet loss, even though IPv6 is working.

Screenshots

Relevant log files

Output of ifconfig igb1:

--- snip ---
	inet6 fe80::219:99ff:feec:b4b9%igb1 prefixlen 64 scopeid 0x2
	inet6 fd1f:5dbe:154f:1:219:99ff:feec:b4b9 prefixlen 64 autoconf
	inet6 2003:____:____:____:219:99ff:feec:b4b9 prefixlen 64 autoconf
--- snip ---

Output of netstat -rn6:

Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif Expire
default                           fe80::1%igb1                  UGS        igb1
::1                               link#5                        UHS         lo0
2001:4860:4860::8844              fe80::1%igb1                  UGHS       igb1
--- snip ---

Parameters of dpinger:

/usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP6 -B fd1f:5dbe:154f:1:219:99ff:feec:b4b9 -p /var/run/dpinger_WAN_DHCP6.pid -u /var/run/dpinger_WAN_DHCP6.sock -s 1s -l 4s -t 60s -d 0 2001:4860:4860::8844

Output of tcpdump icmp6 -i igb1 -n -v (WAN interface, no ICMP replies):

15:33:38.536198 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) fd1f:5dbe:154f:1:219:99ff:feec:b4b9 > 2001:4860:4860::8844: [icmp6 sum ok] ICMP6, echo request, seq 312
15:33:39.584874 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) fd1f:5dbe:154f:1:219:99ff:feec:b4b9 > 2001:4860:4860::8844: [icmp6 sum ok] ICMP6, echo request, seq 313
15:33:40.645843 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) fd1f:5dbe:154f:1:219:99ff:feec:b4b9 > 2001:4860:4860::8844: [icmp6 sum ok] ICMP6, echo request, seq 314
15:33:41.647418 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) fd1f:5dbe:154f:1:219:99ff:feec:b4b9 > 2001:4860:4860::8844: [icmp6 sum ok] ICMP6, echo request, seq 315
15:33:42.702948 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) fd1f:5dbe:154f:1:219:99ff:feec:b4b9 > 2001:4860:4860::8844: [icmp6 sum ok] ICMP6, echo request, seq 316

Additional context

When I used the ISP routers addresses fd80::1 (fd80::1%igb1) or fd1f:5dbe:154f:1::1 for Monitor IP, the IPv6 gateway was displayed as Online:

/usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP6 -B fe80::219:99ff:feec:b4b9%igb1 -p /var/run/dpinger_WAN_DHCP6.pid -u /var/run/dpinger_WAN_DHCP6.sock -s 1s -l 4s -t 60s -d 0 fe80::1%igb1

/usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP6 -B fd1f:5dbe:154f:1:219:99ff:feec:b4b9 -p /var/run/dpinger_WAN_DHCP6.pid -u /var/run/dpinger_WAN_DHCP6.sock -s 1s -l 4s -t 60s -d 0 fd1f:5dbe:154f:1::1

When I set the Global Unicast Address I got from DHCPv6/SLAAC as static on the WAN interface and manually added a IPv6 Upstream Gateway with the address fe80::1 (ISP Router), the IPv6 gateway was also displayed as Online.

When I disabled the IPv6 ULA prefix fd1f:5dbe:154f:1::/64 on the ISP Router when using SLAAC or DHCPv6 on the WAN interface, the IPv6 gateway was also displayed as Online.

This leads me to believe that /usr/local/bin/dpinger just grabs the first IPv6 address it finds on the WAN interface to use as a source address / bind to that address.
But since IPv6 ULA addresses are not routable over the Internet (unless you use NAT66), there is no way it will ever get an ICMP reply message, thus the IPv6 gateway will always be reported as Offline.

For the next few days, I will not have access to other routers to use as gateways for OPNsense that advertise a ULA and GUA prefix, so my sample size is rather limited.

Environment

OPNsense 23.7.6-amd64
FreeBSD 13.2-RELEASE-p3
OpenSSL 1.1.1w 11 Sep 2023
AMD GX-222GC SOC with Radeon(TM) R5E Graphics (2 cores, 2 threads)
Network Intel® I350-T4

[fichtner]

@subnetspider thanks for the report... does this make sense? 89ee410

# opnsense-patch 89ee410

[subnetspider]

As far as I can tell, OPNsense will now check whether an IPv6 address is a ULA or not, and also gives ULAs a lower priority (GUA > LL > ULA ?) when determining the primary IPv6 address of an interface.
So I assume this bug is caused by dpinger getting the wrong primary IPv6 address to bind to?
If my guess is correct, this patch should fix the behavior of the WAN interface being reported as offline.

[fichtner]

Yes, the ULA comes first in your assignment causing dpinger to pick it up trying to reach a GUA, which then doesn’t work.

Special ULA handling was never done on this code base as far as I can tell.

Using opnsense-patch command to verify it’s working will help get this into a stable version quickly.

Cheers,
Franco

[subnetspider]

Allright, I have just spun up a OPNsense 23.7.6 VM ("OPNsense 2") and tried to reproduce the problem, as I don't have access to the other OPNsense ("OPNsense 1") right now. Interestingly, even though a ULA prefix is sent by the router I'm using here, dpinger is using the correct IPv6 address.

What I noticed, however, is that while there is a ULA on the WAN interface in both cases, the order of the IPv6 addresses in ifconfig is different:

OPNsense 1 with Problem:

	inet6 fe80::219:99ff:feec:b4b9%igb1 prefixlen 64 scopeid 0x2
	inet6 fd1f:5dbe:154f:1:219:99ff:feec:b4b9 prefixlen 64 autoconf
	inet6 2003:____:____:____:219:99ff:feec:b4b9 prefixlen 64 autoconf

OPNsense 2 without Problem:

	inet6 fe80::e0b6:1eff:fe33:853f%vtnet1 prefixlen 64 scopeid 0x2
	inet6 2a02:____:____:____:e0b6:1eff:fe33:853f prefixlen 64 autoconf
	inet6 fd00::e0b6:1eff:fe33:853f prefixlen 64 autoconf

This is also reflected in the RA send by the router with OPNsense 2, where the GUA prefix is being send before the ULA prefix:

root@OPNsense:~ # sudo tcpdump -vvvv -ttt -i vtnet1 icmp6 and 'ip6[40] = 134'
tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes

 00:00:00.000000 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 136) fe80::2e3a:fdff:fe9f:695 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 136
	hop limit 255, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
	  prefix info option (3), length 32 (4): 2a02:____:____:____::/64, Flags [onlink, auto], valid time 7200s, pref. time 3600s
	    0x0000:  40c0 0000 1c20 0000 0e10 0000 0000 2a02
	    0x0010:  ____ ____ ____ 0000 0000 0000 0000
	  prefix info option (3), length 32 (4): fd00::/64, Flags [onlink, auto], valid time 7200s, pref. time 3600s
	    0x0000:  40c0 0000 1c20 0000 0e10 0000 0000 fd00
	    0x0010:  0000 0000 0000 0000 0000 0000 0000
	  mtu option (5), length 8 (1):  1500
	    0x0000:  0000 0000 05dc
	  route info option (24), length 8 (1):  ::/0, pref=medium, lifetime=1800s
	    0x0000:  0000 0000 0708
	  route info option (24), length 16 (2):  2a02:____:____:____::/64, pref=medium, lifetime=1800s
	    0x0000:  4000 0000 0708 2a02 ____ ____ ____
	  route info option (24), length 16 (2):  fd00::/64, pref=medium, lifetime=1800s
	    0x0000:  4000 0000 0708 fd00 0000 0000 0000
	  source link-address option (1), length 8 (1): 2c:3a:fd:9f:06:95
	    0x0000:  2c3a fd9f 0695

My theory right now is that without the patch, OPNsense will pick the first IPv6 address on an interface it finds, and only if it gets a ULA before a GUA will the bug occur. I will apply the patch to the "OPNsense 1" firewall later this week to see if my theory is correct and this fixes the bug, but unfortunately I can't get there until Friday. Until then :)

[fichtner]

Yes, the ordering is exactly the issue here that will create the edge case that fails. Part of the problem with having ULA is also described quite a bit, for example here: https://blogs.infoblox.com/ipv6-coe/ula-is-broken-in-dual-stack-networks/ -- some vendors even tend to ignore it completely (a bit like we are doing for primary detection now, but primary for us means routable so GUA and not ULA/LLA so I think it fits better in the general code as opposed to restricting Dpinger setup itself).

Cheers,
Franco

[subnetspider]

Last evening I finally got around to applying the patch, and took some screenshots to document it, since I only had my smartphone with me.

OPNsense GUI before applying the patch:

• The WAN_DHCP6 gateway is again marked as offline.

OPNsense CLI before applying the patch

• In htop you can see that dpinger again uses the ULA to check if the gateway "WAN_DHCP6" is up.

OPNsense GUI after applying the patch

• The WAN_DHCP6 gateway is now correctly marked as online.

OPNsense CLI after applying the patch:

• In htop you can see that dpinger now uses the GUA to check if the gateway "WAN_DHCP6" is up.

When I ran opnsense-patch 89ee410 as root, it fixed the problem right away without requiring a reboot, is this to be expected?

Cheers

[fichtner]

Yeah, it needs a gateway event trigger or routing reload. Some GUI options do this or it was a lucky coincidence that it happened right after applying.

I will bring this into 23.7.7. Thanks for the report!