IPv6 prefix delegation is not supported

[gdonval]

As far as I can tell, DHCPv6-PD (i.e. prefix delegation of IPv6 through DHCP), does not work in Fedora CoreOS and is simply not supported in the base config.

It is apparently a known problem of NetworkManager:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/375

---

It feels like this is a large use case that is not taken into account.

From a very remote perspective (no offence to the people who decided this), it looks like other tools (like systemd-networkd) would really be a better fit in terms of flexibility, ease of configuration and functionality. If the only "win" of NetworkManager is "but I can configure it with a GUI/CLI and then just copy the configuration", I am afraid that choice was very misguided, for a server distribution... installable through ignition... in an automated manner... At any rate, I find infuriating systemd-networkd was carefully removed from the distribution instead of being left alone.

[bengal]

Hi, IPv6 prefix delegation is supported by NM. To enable it, the downstream interface must be configured with ipv6.method=shared. A PD will be requested on the interface with the IPv6 default route and the prefix will be advertised via RA.

[dustymabe]

Thanks @bengal.

@gdonval can you try the suggestion by @bengal and report success or failure?

[dustymabe]

FYI: We opened an upstream issue to improve the documentation: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/758

[gdonval]

I did follow @bengal's suggestion and even restarted from scratch.

I am getting this journal entry (and no IPv6):

policy: ipv6-pd: no device to obtain a subnet to share on enp0s20 from

Here is a (slightly redacted) output from nmcli connection edit WAN, print ipv6 (naturally after reboot):

ipv6.method:                            shared
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.routing-rules:                     --
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.ra-timeout:                        0 (default)
ipv6.dhcp-duid:                         00:03:XX:XX:XX:XX:XX:XX:XX:XX
ipv6.dhcp-iaid:                         --
ipv6.dhcp-timeout:                      0 (default)
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.dhcp-hostname-flags:               0x0 (none)
ipv6.token:                             --

Please note that I also tried to set ipv6.dhcp-duid to the same value without the starting 00:03, which should just be an indicator that the DUID is of a link-layer type. To no avail. If there is anything I missed, don't hesitate to point it out.

Now to add a magic twist into that twisted hardware, my previous systemd-networkd configuration required that I had to force-activate stateful mode (ForceDHCPv6PDOtherInformation=yes). I don't know if that is possible in NM. Certainly did not see anything in the doc related to this.

[gdonval]

I can provide debug information but it does not tell anything: the RA is sent, lndp events are processed but nothing happens.

---

Just changed a few kernel options (sysctl) like accept_ra = 2 and accept_ra_pinfo = 1 (with proper ipv6 all/default prefix). No improvment. I can go back to pristine FCOS whenever.

[bengal]

In NM the prerequisites for IPv6-PD are:

1. ipv6.method=auto is set on the WAN interface (where the prefix will be requested via DHCPv6-PD)

2. ipv6.method=shared is set on the LAN interface (where the prefix will be distributed)

3. the WAN interface has the IPv6 default route

4. the router on the WAN interface advertises M or O flags via RA, so that DHCPv6 is used (stateful or stateless)

Please check 1. and 2. Message "no device to obtain a subnet to share on enp0s20 from" indicates that there is no device with IPv6 default route.

If the IPv6 methods are configured correctly, please collect and share NM logs at trace level.

[gdonval]

So I created a new dummy0 interface as my LAN connection (since I actually want a prefix and address to be allocated on that same machine).

$ sudo nmcli connection add type dummy ifname dummy0 ipv4.method disabled ipv6.method shared

Also made sure the WAN interface went back to auto.

Anyway, here is the log you requested.

---

BTW, disregard the LAN interface: it's not even activated. My actual lan interface is that dummy-dummy0 one.

---

It does seem to me that I need to somehow force the use of DHCPv6 on my wan interface:

Jul 09 09:43:00 hell NetworkManager[688]: [1625823780.1893] policy: ipv6-pd: none of 0 prefixes of enp0s20 can be shared on dummy0
Jul 09 09:43:00 hell NetworkManager[688]: [1625823780.1894] device (enp0s20): ipv6-pd: device doesn't use DHCPv6, can't request prefixes

I suspect the router (I don't control) is not sending the proper flags, hence the ForceDHCPv6PDOtherInformation=yes systemd option I had to use. But might be something else. I'll wait and see.

---

@bengal If that's any easier for you, I can provide you with direct access to a small FCOS instance in the right environment. Just provide a public SSH key. And again, if there is any consistent documentation online explaining how to do this with NM, I'll gladly try it all.

NM is frustrating because errors I (don't) get aren't returning any result online and configuration seems quite limited in scope so there's really little I can do by myself. I understand that's the point of getting RH support, but this is FCOS, not RH...

[bengal]

What is implemented by NM at the moment is to support PD on WAN interfaces that already use DHCPv6 (not on those using SLAAC). We try to improve PD support when users report problems, as it's not very clear from RFCs what are all the possible ways in which PD can be used. For example some time ago we did some fixes to support PD in DHCPv6 stateless mode or over PPPoE:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/647
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/478

This scenario where the server uses only SLAAC is not supported at the moment (mainly because nobody reported it so far), but we can try to prepare a patch to support it, together with a better documentation.

[gdonval]

Ah I see the misunderstanding! The server does not actually support SLAAC at all... Yet O is set but not M!

But before you say it's a "won't fix"...

By default setting only the 'O' bit in Router Advertisements makes DHCPv6 request network information in a stateless manner using a two-message Information Request and Information Reply message exchange. RFC 7084, requirement WPD-4, updates this behavior for a Customer Edge router so that stateful DHCPv6 Prefix Delegation is also requested when only the 'O' bit is set in Router Advertisements.

For whatever reason this is the behaviour that has been implemented. Both systemd-networkd and dhclient6 provide a specific flag for exactly this because they have no way to actually know. As this is possible behaviour for CE routers in general, it would seem sensible to also have support in NM?

---

Oh oh oh! nearly there... So I found the AFAIK undocumented option ipv6.method=dhcp:

IP configuration method. NMSettingIP4Config and NMSettingIP6Config both support "auto", "manual", and "link-local". See the subclass-specific documentation for other values. In general, for the "auto" method, properties such as "dns" and "routes" specify information that is added on to the information returned from automatic configuration. The "ignore-auto-routes" and "ignore-auto-dns" properties modify this behavior. For methods that imply no upstream network, such as "shared" or "link-local", these properties must be empty. For IPv4 method "shared", the IP subnet can be configured by adding one manual IPv4 address or otherwise 10.42.x.0/24 is chosen.

This seems to be forcing the use of the dhcp6 client (which seems to be systemd's). Now I need to actually get routing to work!

[bengal]

Ah, right. I said SLAAC to mean no-DHCPv6. But the RA here only provides a route without a prefix (and with no M or O flag):

 [1625823780.1838] ndisc[0x56253c66c970,"enp0s20"]:   dhcp-level none
 [1625823780.1838] ndisc[0x56253c66c970,"enp0s20"]:   hop limit      : 64
 [1625823780.1839] ndisc[0x56253c66c970,"enp0s20"]:   gateway fe80::ce46:d6ff:feb2:c9f1 pref medium exp 1800.000

so it's neither SLAAC nor DHCPv6. The client only gets configured with a link-local address and a default route.

With systemd-networkd (or NM with ipv6.method=dhcp), do you actually get an address via DHCPv6?

As you have already noticed, ipv6.method=dhcp does only DHCPv6, and DHCPv6 alone only provides a /128 address or additional information, but not a default route.

Both systemd-networkd and dhclient6 provide a specific flag for exactly this because they have no way to actually know. As this is possible behaviour for CE routers in general, it would seem sensible to also have support in NM?

Yes, but I think the problem here is different. systemd-networkd can be forced to start DHCPv6, and then there is this flag to the request the prefix delegation. The problem with NM is that it doesn't start DHCPv6 in the first place, unless the RA says it's needed.

I guess what is missing in NM is one of:

1. if the WAN interface is not using DHCP and a PD is required, NM should automatically start DHCP on the interface;

or

2. there should be a way to force-start DHCPv6 (together with RA) on the WAN interface even if RA doesn't carry the M,O flags. This is similar to configuring DHCP=ipv6 and IPv6AcceptRA=yes in systemd-networkd.

With these, I suppose there is no need to something similar to systemd's ForceDHCPv6PDOtherInformation because NM knows when a PD is required, and can restart the DHCP client on the WAN with the right flags.

What do you think?

[gdonval]

With systemd-networkd (or NM with ipv6.method=dhcp), do you actually get an address via DHCPv6?

Yep! It's a noprefixroute one but it is the correct prefix.

The logs say that DHCPv6 CLIENT gets started in managed mode. Then the solicit/advert/req dance starts and I get a domain name, DNS and an IP address from it (dhcp6 (iface) option ip6_address => 2001:XXXX:1). I can send the logs if necessary.

As you have already noticed, ipv6.method=dhcp does only DHCPv6, and DHCPv6 alone only provides a /128 address or additional information, but not a default route.

Exactly!

What do you think?

That I am probably too incompetent to notice whether there is a flaw in what you are recommending. It indeed sounds like ForceDHCPv6PDOtherInformation would definitely be superfluous if one can say "just start the DHCP6 client unconditionally". Regarding Options 1 and 2, 1 would be perfect but I am wondering whether there would be situations where both SLAAC and PD are available and one would need/want to force the use of PD.

---

Just noticed I forgot: adding my default route with the gateway provided previously does work! Though I can't set the gateway with nmcli in ipv6.method = dhcp mode (it complains it needs a default address, as I suppose everything then should be manual).

$ sudo ip -6 a add 2001:xx::/64 dev iname mngtmpaddr
$ sudo ip -6 route add default via fe80:xx:: dev iname

[bengal]

Regarding Options 1 and 2, 1 would be perfect but I am wondering whether there would be situations where both SLAAC and PD are available and one would need/want to force the use of PD.

I don't understand what you mean here... SLAAC and PD are used for different purposes: SLAAC to assign an address to the WAN interface, PD to obtain prefixes to share to LAN interfaces.

There was a discussion [1] on the NM mailing list about a case where the ISP server doesn't assign any address to the client but provides prefix delegations; the question was whether the client could use an address from the prefix delegation for the WAN interface. This was explicitly prohibited in a previous version of the RFC but seems allowed now. AFAICT the conclusions on that thread were that:

1. this scenario looked like a misconfiguration on ISP side
2. if really necessary, one could configure a dummy interface with NM in shared mode, and that would be equivalent to having the address on the WAN interface.

[1] https://mail.gnome.org/archives/networkmanager-list/2021-June/msg00026.html

[bengal]

I opened a merge request to support PD in NM when the upstream interface is not configured via DHCP.

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/924

I did some basic testing with the script in the first commit, but some real world testing would be helpful if possible. I can prepare a scratch build of the Fedora NM package if necessary.

[gdonval]

I don't understand what you mean here... SLAAC and PD are used for different purposes: SLAAC to assign an address to the WAN interface, PD to obtain prefixes to share to LAN interfaces.

That's because I misunderstood how NM would determine PD is necessary. If I understand correctly now, as long as a LAN interface requests an address from the WAN interface, NM determines PD is required and will start DHCPv6 on the WAN interface to request a prefix (and an address I suppose)... Or at least that's the point of the MR you submitted.

I did some basic testing with the script in the first commit, but some real world testing would be helpful if possible. I can prepare a scratch build of the Fedora NM package if necessary.

Happy to test this and report back.

[bengal]

This scratch build includes the patch: https://koji.fedoraproject.org/koji/taskinfo?taskID=71766806
Grab all RPMs and then run rpm -Fvh *.rpm to update them.

[dustymabe]

This scratch build includes the patch: https://koji.fedoraproject.org/koji/taskinfo?taskID=71766806
Grab all RPMs and then run rpm -Fvh *.rpm to update them.

In the case of Fedora CoreOS something like this should work:

sudo rpm-ostree override replace https://kojipkgs.fedoraproject.org//work/tasks/6806/71766806/NetworkManager{-libnm,-tui,-cloud-setup,-team,}-1.30.4-1.1.coreos888.fc34.x86_64.rpm

Then reboot.

[gdonval]

In the case of Fedora CoreOS something like this should work:

sudo rpm-ostree override replace https://kojipkgs.fedoraproject.org//work/tasks/6806/71766806/NetworkManager{-libnm,-tui,-cloud-setup,-team,}-1.30.4-1.1.coreos888.fc34.x86_64.rpm

Then reboot.

Ah I suppose that make sense. I tried that very command on the local RPMs but it failed:

error: Importing package 'NetworkManager': Unsupported path: /00-server.conf; See https://github.com/projectatomic/rpm-ostree/issues/233

I suppose koji URLs are treated as some sort of source.

[gdonval]

@bengal Scratch my previous comment (now deleted from github): a reboot was indeed necessary. I get IPv6 connectivity now after reboot using @dustymabe's method to override packages.

[gdonval]

Thank you both, that's amazing!

[gdonval]

Ah, potential "problems" depending on what you think should belong to the present patch:

• the global ip address gets a noprefixroute flag.
• no temporary address is generated when using ipv6.ip6-privacy = 2.
• the ipv6.addr-gen-mode also seems to be ignored (the address I am allocated is 2001::XXX:1).

Turned out those are explicitly defined for stateless, so they definitely don't belong here!

Everything looks fine and I suppose using temporary addresses and such in NM should be the object of another feature request.

[bengal]

* the global ip address gets a `noprefixroute` flag.

NM adds the prefix route by itself so that it can control the route metric, which can be changed through the ipv6.route-metric connection property. The noprefixroute is added by NM to the address to prevent the kernel from adding again the prefix route with a fixed metric.

Kernels >= 4.18 support specifying the prefix route metric when adding the address:

commit 8308f3ff1753d001f7a73f9bb0f02292b5400557
Author: David Ahern <dsahern@gmail.com>
Date:   Sun May 27 08:09:58 2018 -0700

    net/ipv6: Add support for specifying metric of connected routes

    Add support for IFA_RT_PRIORITY to ipv6 addresses.

    If the metric is changed on an existing address then the new route
    is inserted before removing the old one. Since the metric is one
    of the route keys, the prefix route can not be atomically replaced.

however NM needs to also support older kernels and so it still adds the prefix route by itself. It would be a nice improvement to use IFA_RT_PRIORITY when available.

[bengal]

The patch is now merged and backported to the 1.30 branch. It will be in the next 1.30.6 release and it will reach F34 at the next update.

[bengal]

NetworkManager-1.30.6-1.fc34 is now available.

[dustymabe]

Thanks @bengal. This will make it into the next FCOS testing release.

[dustymabe]

The fix for this went into testing stream release 34.20210725.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 34.20210725.3.0.

[gdonval]

I'll reprovision tomorrow and let you know!