Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

networkd: IPv6 rapid rotation of temporary addresses #20050

Closed
ghen2 opened this issue Jun 28, 2021 · 14 comments · Fixed by #20108
Closed

networkd: IPv6 rapid rotation of temporary addresses #20050

ghen2 opened this issue Jun 28, 2021 · 14 comments · Fixed by #20108
Labels
bug 🐛 Programming errors, that need preferential fixing dhcp network
Milestone

Comments

@ghen2
Copy link

ghen2 commented Jun 28, 2021

systemd version the issue has been seen with

249rc2

Used distribution

Arch Linux

Linux kernel version used (uname -a)

5.12.13-arch1-2

CPU architecture issue was seen on

x86_64

Expected behaviour you didn't see

stable IPv6 privacy addresses for (long running) outbound connections

Unexpected behaviour you saw

rapid rotation of privacy addresses, breaking current connections

Steps to reproduce the problem

upgrade to systemd 249, restart networkd

Additional program output to the terminal or log subsystem illustrating the issue

Below is a diff of networkctl status 3 with just a few minutes in a between.
A new temporary privacy address has been added and the old one immediatly removed, breaking all existing connections with that old address. This happens every few minutes, despite the much longer valid_lifetime indicated by ip addr list.

Normally, old addresses are first put in "deprecated" state, so they are no longer used for new connections, and removed later. Plus rotation should be much less frequent, respecting valid_lifetime of the addresses (hours for global addresses, days for ULA).

I noticed this behaviour with systemd 249rc2, coming from 248 where it was stable.

 * 3: eno1                       
                      Link File: /usr/lib/systemd/network/99-default.link
                   Network File: /etc/systemd/network/10-ethernet.network
                           Type: ether
                          State: routable (configured)
                   Online state: online
              Alternative Names: enp0s31f6
                           Path: pci-0000:00:1f.6
                         Driver: e1000e
                         Vendor: Intel Corporation
                          Model: Ethernet Connection (6) I219-V
                     HW Address: 1c:69:7a:0b:e1:fe (EliteGroup Computer Systems Co., LTD)
                            MTU: 1500 (min: 68, max: 9000)
                          QDisc: fq_codel
   IPv6 Address Generation Mode: eui64
           Queue Length (Tx/Rx): 1/1
               Auto negotiation: yes
                          Speed: 1Gbps
                         Duplex: full
                           Port: tp
                        Address: 172.16.1.3
                                 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe
-                                2a02:1810:4d12:7000:d342:cf65:aee:5e5e
+                                2a02:1810:4d12:7000:e4d7:c363:6f04:c072
                                 fd1b:229d:8fec::3
-                                fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe
-                                fd1b:229d:8fec:0:d6c5:91d1:211c:5fcb
                                 fe80::1e69:7aff:fe0b:e1fe
                        Gateway: 172.16.1.254
                                 fe80::c43d:c7ff:fea3:20bd
                            DNS: fd1b:229d:8fec::1
              Activation Policy: up
            Required For Online: yes
              DHCP6 Client IAID: 0xb6220feb
              DHCP6 Client DUID: DUID-EN/Vendor:0000ab117803f0a3b327c68e0000
                   Connected To: n/a on port a4:4c:c8:1b:50:63
 

ip -6 addr list

3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2a02:1810:4d12:7000:e4d7:c363:6f04:c072/64 scope global temporary dynamic 
       valid_lft 36712sec preferred_lft 7912sec
    inet6 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 36712sec preferred_lft 7912sec
    inet6 fd1b:229d:8fec::3/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::1e69:7aff:fe0b:e1fe/64 scope link 
       valid_lft forever preferred_lft forever
@ghen2
Copy link
Author

ghen2 commented Jun 28, 2021

Looks like it's starting from an empty state with every cycle:

Jun 28 10:32:10 systemd-networkd[222]: eno1: NDISC: Received Router Advertisement: flags OTHER preference medium lifetime 1800 sec
Jun 28 10:32:10 systemd-networkd[222]: eno1: NDISC: Invoking callback for 'router' event.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Acquiring DHCPv6 lease on NDisc request
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 1s, preferred for 2h 1min 1s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Setting SLAAC addresses.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Setting NDisc routes.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing old NDisc information obtained from fe80::c43d:c7ff:fea3:20bd.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing address: fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe/64 (valid forever, preferred forever), flags: permanent,manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 26s, preferred for 2h 1min 26s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Remembering updated address: fd1b:229d:8fec:0:5487:a11b:9754:7d2a/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe/64 (valid forever, preferred forever), flags: permanent,manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: fd1b:229d:8fec:0:5487:a11b:9754:7d2a/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec:0:5487:a11b:9754:7d2a/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Remembering updated address: 2a02:1810:4d12:7000:981c:e4a7:7023:46ff/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 2s, preferred for 2h 1min 2s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: 2a02:1810:4d12:7000:981c:e4a7:7023:46ff/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000:981c:e4a7:7023:46ff/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 1s, preferred for 2h 1min 1s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Remembering updated address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 1s, preferred for 2h 1min 1s), flags: tentative,manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: NDisc SLAAC addresses set.
Jun 28 10:32:10 systemd-networkd[222]: eno1: No SLAAC address obtained from fe80::c43d:c7ff:fea3:20bd is ready. The old NDisc information will be removed later.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024 
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: NDisc routes set.
Jun 28 10:32:10 systemd-networkd[222]: eno1: No SLAAC address obtained from fe80::c43d:c7ff:fea3:20bd is ready. The old NDisc information will be removed later. 
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering foreign address: 2a02:1810:4d12:7000:594:31f:d12c:59dd/64 (valid for 10h 1min, preferred for 2h 1min), flags: temporary
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering foreign route: dst: 2a02:1810:4d12:7000:594:31f:d12c:59dd/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering updated address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min, preferred for 2h 1min), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:11 systemd-networkd[222]: eno1: Removing old NDisc information obtained from fe80::c43d:c7ff:fea3:20bd.
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering foreign route: dst: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0

@keszybz
Copy link
Member

keszybz commented Jun 30, 2021

Sounds like we need another #19980, but this time for IPv6.

@ghen2
Copy link
Author

ghen2 commented Jun 30, 2021

Please note that this is a regression in 249(rc2), this behaviour was not present in 248.

@keszybz
Copy link
Member

keszybz commented Jun 30, 2021

Yeah, but there were 250 non-merge commits touching src/network in the period. If you could bisect this, that'd probably help.
Please also paste /etc/systemd/network/10-ethernet.network.

FWIW, I don't see this here.

@keszybz
Copy link
Member

keszybz commented Jun 30, 2021

This seems to be the last unadressed issue for v249. I think we should leave it for later, unless @yuwata has some good idea how to fix this.

@yuwata
Copy link
Member

yuwata commented Jun 30, 2021

Hmm, this is caused by 0828a38. But it fixes other issues...

@ghen2
Copy link
Author

ghen2 commented Jun 30, 2021

Please also paste /etc/systemd/network/10-ethernet.network.

[Match]
Name=eno1

[Network]
Address=fd1b:229d:8fec::3/64
IPv6PrivacyExtensions=yes

Address=172.16.1.3/24
Gateway=172.16.1.254
DNS=fd1b:229d:8fec::1

 + SLAAC/privacy addressing based on prefix advertisements (ULA + global, with long lifetimes) from an OpenWRT router, no DHCPv6.

@yuwata yuwata added the bug 🐛 Programming errors, that need preferential fixing label Jun 30, 2021
yuwata added a commit to yuwata/systemd that referenced this issue Jun 30, 2021
Previously, NDisc configurations with `marked` flag are considered as old,
and removed when all netlink messages are processed and at least one
address becomes ready.

However, when request queue was introduced, the logic was not updated.
As all requests are once queued and processed later when each request is
ready, the `marked` flag of each configuration is not removed within the
same event. Thus, all previous configurations are removed before no address
or route request is processed. This causes the issue systemd#20050.

This introduces another request type to clear old NDisc configureations
to make the cleanup is certainly processed after all address and route
requests are processed.

Fixes systemd#20050.
yuwata added a commit to yuwata/systemd that referenced this issue Jul 2, 2021
Previously, NDisc configurations with `marked` flag are considered as old,
and removed when all netlink messages are processed and at least one
address becomes ready.

However, when request queue was introduced, the logic was not updated.
As all requests are once queued and processed later when each request is
ready, the `marked` flag of each configuration is not removed within the
same event. Thus, all previous configurations are removed before no address
or route request is processed. This causes the issue systemd#20050.

This introduces another request type to clear old NDisc configureations
to make the cleanup is certainly processed after all address and route
requests are processed.

Fixes systemd#20050.
yuwata added a commit to yuwata/systemd that referenced this issue Jul 2, 2021
Previously, `ndisc_remove_old_one()` checked `ndisc_{addresses,routes}_configured`
flags, but they are not unset when all addresses or routes are already
assigned.
After the request queue is implemented, the address or route requests
are not processed within the same event of ndisc handler is called, but
will processed later when they are ready. So, calling `ndisc_remove_old()`
in the event of ndisc handler will remove all addresses and routes
previously assigned even they are requested to be updated.

This makes `ndisc_remove_old()` do nothing when there exist some
requests to configure addresses and routes, thus previously assigned
addresses and routes are kept until all requests are processed.

Fixes systemd#20050.
yuwata added a commit to yuwata/systemd that referenced this issue Jul 2, 2021
@yuwata
Copy link
Member

yuwata commented Jul 2, 2021

@ghen2 If possible, could you test #20108?

yuwata added a commit to yuwata/systemd that referenced this issue Jul 2, 2021
Previously, `ndisc_remove_old_one()` checked `ndisc_{addresses,routes}_configured`
flags, but they are not unset when all addresses or routes are already
assigned.
After the request queue is implemented, the address or route requests
are not processed within the same event of ndisc handler is called, but
will processed later when they are ready. So, calling `ndisc_remove_old()`
in the event of ndisc handler will remove all addresses and routes
previously assigned even they are requested to be updated.

This makes `ndisc_remove_old()` do nothing when there exist some
requests to configure addresses and routes, thus previously assigned
addresses and routes are kept until all requests are processed.

Fixes systemd#20050.
yuwata added a commit to yuwata/systemd that referenced this issue Jul 2, 2021
@ghen2
Copy link
Author

ghen2 commented Jul 2, 2021

Yes, #20108 works for me, thanks!

Another difference between 248 and 249 (with or without your patch) is that the SLAAC addresses in the ULA prefix continously appear and disappear (both fixed mac-based in temporary privacy addresses), which is strange. This does not happen for the globally routable prefix though, does networkd distinguish between those?

@ghen2
Copy link
Author

ghen2 commented Jul 2, 2021

In fact, this causes the same issue for outgoing connections within ULA prefix (thus within my LAN).
Your patch fixed it for "outside" connections (sourced from globally routable addresses) but not for local ones.

yuwata added a commit to yuwata/systemd that referenced this issue Jul 2, 2021
@yuwata
Copy link
Member

yuwata commented Jul 2, 2021

Thank you for testing the PR so quickly! I added one more commit in the PR. I hope it fixes the ULA address issue. Please test gain.

@ghen2
Copy link
Author

ghen2 commented Jul 2, 2021

Yes, the ULA case is fixed as well now. Thanks!

@yuwata
Copy link
Member

yuwata commented Jul 2, 2021

Thank you! Your help is much appreciated.

@vp1981
Copy link

vp1981 commented Jul 3, 2021

Same here and PR#20108 fixed the issue (Archlinux, systemd-249rc3).

jamacku pushed a commit to redhat-plumbers/systemd-rhel9 that referenced this issue Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 Programming errors, that need preferential fixing dhcp network
5 participants