Description
systemd version the issue has been seen with
256
Used distribution
Ubuntu 23
Linux kernel version used
6.5.0-17-generic
CPU architectures issue was seen on
x86_64
Component
systemd-networkd
Expected behaviour you didn't see
RA Prefix Cleanup is NOT cleaning up
This issue shows up in many tests, especially those that test REDIRECT. So I'll use this REDIRECT test (2.3.1c) as the basis for this issue.
Test v6LC.2.3.1: Redirected On-link: Valid (Hosts Only)
Purpose: Verify that a host properly processes valid Redirect messages when redirected on-link: (no Redirect Pkt Option)
Reference: RFC 4861, Section 4.6.1, 4.6.3, and 8.3
Network Configuration
TR1
|
+---+---+
| |
TN1 HUT
Legend: TN1=Test Node, TR1=Test Router, HUT=Host Under Test
TR1 Pfx: 2001:2:0:1000/64
TN1 GA: 2001:2:0:1001:200:10ff:fe10:1180
(offlink address, but is physically onlink)
HUT GA: 2001:2:0:1000:a00:27ff:fe1a:432/64
Procedure
Step | Action | Expected Behavior of HUT(1) |
---|---|---|
1 | TR1 forwards Echo request Src="TN1 Offlink GA" to HUT GA(2). |
HUT sends Echo reply to TR1 as first hop. |
2 | TR1 sends Redirect Dst="TN1 GA" Target="TN1 GA" to HUT. |
|
3 | TR1 forwards Echo request Src="TN1 Offlink GA" to HUT GA. |
HUT sends NS for TN1's GA and Echo reply to TN1 on-link. |
- (1) - Host Under Test
- (2) - Global Address
Unexpected behaviour you saw
There are two steps required to cause this failure:
- The first time this test runs,
networkd
recieves aREDIRECT
message targettingTN1 GA
and it updates the routing table and the test passes. - Subsequent runs fail, because the
REDIRECT
route persists. When we next ping theHUT
fromTN1 GA
, this causesnetworkd
to sendNS
forTN1 GA
, when the test is expecting anEcho Reply
.
Comparing networkd
behaviour with the native kernel
behaviour, you can see that the kernel does NOT add a redirect route to the routing table. So it avoids this issue.
Between test runs, if I manually delete the redirect route, the next test passes. ie. running this command after each test results in the next test passing:
sudo ip -6 r del 2001:2:0:1001:200:10ff:fe10:1180 dev enp0s9
Watch Systemd-Networkd Routing Table
$ old=""; while :; do out=$(ip -6 r show dev enp0s9); new=$(echo "$out" | sed 's/expires.*sec //'); [ "$new" != "$old" ] && date +"%F %T.%N:%n""${out}%n"; old="$new"; sleep 1; done
### 1. Initial state
2024-04-28 09:03:32.687174870:
fe80::/64 proto kernel metric 256 pref medium
### 2. Received RA Prefix=2001:2:0:1000::/64
2024-04-28 09:03:37.782118076:
2001:2:0:1000::/64 proto ra metric 1024 expires 65534sec pref medium
fe80::/64 proto kernel metric 256 pref medium
default via fe80::200:10ff:fe10:1060 proto ra metric 1024 expires 8999sec mtu 1500 pref medium
### 3. Received REDIRECT request targetting 2001:2:0:1001:200:10ff:fe10:1180
### *** Why was a host route added (2001:2:0:1001:200:10ff:fe10:1180)???
2024-04-28 09:03:45.892788195:
2001:2:0:1000::/64 proto ra metric 1024 expires 65526sec pref medium
2001:2:0:1001:200:10ff:fe10:1180 proto redirect metric 1024 pref medium
fe80::/64 proto kernel metric 256 pref medium
default via fe80::200:10ff:fe10:1060 proto ra metric 1024 expires 8991sec mtu 1500 pref medium
### 4. Received RA Prefix=2001:2:0:1000::/64 Lifetime=0, Router Lifetime=0
### Redirect route persists, causing next test to fail.
2024-04-28 09:03:46.909014218:
2001:2:0:1001:200:10ff:fe10:1180 proto redirect metric 1024 pref medium
fe80::/64 proto kernel metric 256 pref medium
Watch Kernel Native Routing Table
NOTE: This test was performed with networkd
stopped and sycctl accept_ra=2
, so this shows how the kernel native solution behaves.
$ old=""; while :; do out=$(ip -6 r show dev enp0s9); new=$(echo "$out" | sed 's/expires.*sec //'); [ "$new" != "$old" ] && date +"%F %T.%N:%n""${out}%n"; old="$new"; sleep 1; done
### 1. Initial State
2024-04-28 09:30:47.279772318:
fe80::/64 proto kernel metric 256 pref medium
### 2. Received RA Prefix=2001:2:0:1000::/64 > new route is added
2024-04-28 09:30:53.372220636:
2001:2:0:1000::/64 proto kernel metric 256 expires 65534sec pref medium
fe80::/64 proto kernel metric 256 pref medium
default via fe80::200:10ff:fe10:1060 proto ra metric 1024 expires 8999sec hoplimit 64 pref medium
### 3. Received REDIRECT request targetting 2001:2:0:1001:200:10ff:fe10:1180
### NOTE: Unlike networkd, there is no "redirect" route added.
### 4. Received RA Prefix=2001:2:0:1000::/64 Lifetime=0, Router Lifetime=0
2024-04-28 09:31:03.515867947:
fe80::/64 proto kernel metric 256 pref medium
Steps to reproduce the problem
For reference only: Previous issue relating to Redirect is #31438.
Run the test twice. IF using tcpreplay, run the "pass" PCAP twice. The 2nd time it will fail.
Additional program output to the terminal or log subsystem illustrating the issue
#########################################
# Debug LOG from first test (PASSES) #
# See below for 2nd test where it fails #
#########################################
enp0s9: NDISC: Received Router Advertisement from fe80::200:10ff:fe10:1060: flags=0x00(null), preference=medium, lifetime=2h 30min
enp0s9: NDISC: Invoking callback for 'router' event.
enp0s9: Requesting NDisc route (n/a): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
Setting '/proc/sys/net/ipv6/neigh/enp0s9/base_reachable_time_ms' to '30000'
No change in value '30000', suppressing write
Setting '/proc/sys/net/ipv6/neigh/enp0s9/retrans_time_ms' to '1000'
No change in value '1000', suppressing write
Setting '/proc/sys/net/ipv6/conf/enp0s9/hop_limit' to '64'
No change in value '64', suppressing write
Setting '/proc/sys/net/ipv6/conf/enp0s9/mtu' to '1500'
No change in value '1500', suppressing write
enp0s9: Requesting NDisc route (n/a): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Requesting NDisc address (configured): 2001:2:0:1000:a00:27ff:fe1a:432/64 (valid for 18h 12min 14s, preferred for 18h 12min 14s), flags: manage-temporary-address,no-prefixroute, scope: global
enp0s9: Setting SLAAC addresses and router.
enp0s9: State changed: configured -> configuring
Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_34 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=34 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
enp0s9: link_check_ready(): dynamic addressing protocols are enabled but none of them finished yet.
enp0s9: Configuring NDisc route (requesting): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Configuring NDisc route (requesting): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Configuring NDisc address (requesting,configured): 2001:2:0:1000:a00:27ff:fe1a:432/64 (valid for 18h 12min 14s, preferred for 18h 12min 14s), flags: manage-temporary-address,no-prefixroute, scope: global
enp0s9: Received new NDisc route (configured): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: ndisc_check_ready(): SLAAC addresses and routes are not set.
enp0s9: Received new NDisc route (configured): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: ndisc_check_ready(): SLAAC addresses and routes are not set.
enp0s9: Received updated NDisc address (configured): 2001:2:0:1000:a00:27ff:fe1a:432/64 (valid for 18h 12min 14s, preferred for 18h 12min 14s), flags: manage-temporary-address,no-prefixroute, scope: global
enp0s9: link_check_ready(): dynamic addressing protocols are enabled but none of them finished yet.
enp0s9: SLAAC addresses and routes set.
enp0s9: link_check_ready(): IPv4LL:no DHCPv4:no DHCPv6:no DHCP-PD:no NDisc:yes
enp0s9: State changed: configuring -> configured
Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_34 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=35 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
enp0s9: NDISC: Received Neighbor Advertisement from fe80::200:10ff:fe10:1060: Router=yes, Solicited=yes, Override=no
enp0s9: NDISC: Invoking callback for 'neighbor' event.
enp0s9: NDISC: Received Redirect message from fe80::200:10ff:fe10:1060: Target=2001:2:0:1001:200:10ff:fe10:1180, Destination=2001:2:0:1001:200:10ff:fe10:1180
enp0s9: NDISC: Invoking callback for 'redirect' event.
enp0s9: Requesting NDisc route (n/a): dst: 2001:2:0:1001:200:10ff:fe10:1180/128, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: redirect, scope: global, type: unicast, flags: n/a
enp0s9: Configuring NDisc route (requesting): dst: 2001:2:0:1001:200:10ff:fe10:1180/128, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: redirect, scope: global, type: unicast, flags: n/a
enp0s9: Received new NDisc route (configured): dst: 2001:2:0:1001:200:10ff:fe10:1180/128, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: redirect, scope: global, type: unicast, flags: n/a
enp0s9: SLAAC addresses and routes set.
ICMPv6: Received a packet from neither link-local nor null address.
enp0s9: NDISC: Failed to receive ICMPv6 packet, ignoring: Cannot assign requested address
enp0s9: NDISC: Received Router Advertisement from fe80::200:10ff:fe10:1060: flags=0x00(null), preference=medium, lifetime=0
enp0s9: NDISC: Invoking callback for 'router' event.
enp0s9: Removing NDisc route (configured): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Removing NDisc route (configured): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Forgetting removed NDisc route (n/a): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Forgetting removed NDisc route (n/a): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: NDISC: Received Neighbor Advertisement from fe80::200:10ff:fe10:1060: Router=yes, Solicited=no, Override=yes
enp0s9: NDISC: Invoking callback for 'neighbor' event.
Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_33 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=36 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_33 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=37 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
######################################
# Debug LOG from second test (FAILS) #
######################################
enp0s9: NDISC: Received Router Advertisement from fe80::200:10ff:fe10:1060: flags=0x00(null), preference=medium, lifetime=2h 30min
enp0s9: NDISC: Invoking callback for 'router' event.
enp0s9: Requesting NDisc route (n/a): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
Setting '/proc/sys/net/ipv6/neigh/enp0s9/base_reachable_time_ms' to '30000'
No change in value '30000', suppressing write
Setting '/proc/sys/net/ipv6/neigh/enp0s9/retrans_time_ms' to '1000'
No change in value '1000', suppressing write
Setting '/proc/sys/net/ipv6/conf/enp0s9/hop_limit' to '64'
No change in value '64', suppressing write
Setting '/proc/sys/net/ipv6/conf/enp0s9/mtu' to '1500'
No change in value '1500', suppressing write
enp0s9: Requesting NDisc route (n/a): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Requesting NDisc address (configured): 2001:2:0:1000:a00:27ff:fe1a:432/64 (valid for 18h 12min 14s, preferred for 18h 12min 14s), flags: manage-temporary-address,no-prefixroute, scope: global
enp0s9: Setting SLAAC addresses and router.
enp0s9: State changed: configured -> configuring
Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_34 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=38 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
enp0s9: link_check_ready(): dynamic addressing protocols are enabled but none of them finished yet.
enp0s9: Configuring NDisc route (requesting): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Configuring NDisc route (requesting): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Configuring NDisc address (requesting,configured): 2001:2:0:1000:a00:27ff:fe1a:432/64 (valid for 18h 12min 14s, preferred for 18h 12min 14s), flags: manage-temporary-address,no-prefixroute, scope: global
enp0s9: Received new NDisc route (configured): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: ndisc_check_ready(): SLAAC addresses and routes are not set.
enp0s9: Received new NDisc route (configured): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: ndisc_check_ready(): SLAAC addresses and routes are not set.
enp0s9: Received updated NDisc address (configured): 2001:2:0:1000:a00:27ff:fe1a:432/64 (valid for 18h 12min 14s, preferred for 18h 12min 14s), flags: manage-temporary-address,no-prefixroute, scope: global
enp0s9: link_check_ready(): dynamic addressing protocols are enabled but none of them finished yet.
enp0s9: SLAAC addresses and routes set.
enp0s9: link_check_ready(): IPv4LL:no DHCPv4:no DHCPv6:no DHCP-PD:no NDisc:yes
enp0s9: State changed: configuring -> configured
Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_34 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=39 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
enp0s9: NDISC: Received Neighbor Advertisement from fe80::200:10ff:fe10:1060: Router=yes, Solicited=yes, Override=no
enp0s9: NDISC: Invoking callback for 'neighbor' event.
ICMPv6: Received a packet from neither link-local nor null address.
enp0s9: NDISC: Failed to receive ICMPv6 packet, ignoring: Cannot assign requested address
enp0s9: NDISC: Received Router Advertisement from fe80::200:10ff:fe10:1060: flags=0x00(null), preference=medium, lifetime=0
enp0s9: NDISC: Invoking callback for 'router' event.
enp0s9: Removing NDisc route (configured): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Removing NDisc route (configured): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: Forgetting removed NDisc route (n/a): dst: n/a, src: n/a, gw: fe80::200:10ff:fe10:1060, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a
enp0s9: NDISC: Received Neighbor Advertisement from fe80::200:10ff:fe10:1060: Router=yes, Solicited=no, Override=yes
enp0s9: NDISC: Invoking callback for 'neighbor' event.
enp0s9: Forgetting removed NDisc route (n/a): dst: 2001:2:0:1000::/64, src: n/a, gw: n/a, prefsrc: n/a, table: main(254), priority: 1024, proto: ra, scope: global, type: unicast, flags: n/a