CFP: BGP Control Plane automatic Router ID assignment for IPv6 single stack setup

[YutaroHayakawa]

Cilium Feature Proposal

Problem

In IPv4 and dual-stack environments, the Cilium BGP Control Plane derives the Router ID from the IPv4 address assigned to the node. However, in the IPv6 single-stack environment, there's no IPv4 address to use, so users must specify the Router ID manually for each virtual router for each Node.

$ kubectl annotate node <node-name> cilium.io/bgp-virtual-router.64512="router-id=10.0.0.1"

This is a big operational overhead because users must manage the assignment by themselves.

Proposed Solution

User Interface

The basic idea is to provide a pool of Router IDs like 10.0.0.0/16 in advance, and the Cilium Operator manages the assignment as we do for the IPAMs. A design decision we need to make here is how to provide the Router ID.

Option1: Via Helm configuration

Provide Router ID pool through Cilium operator's configuration like bgpControlPlane.routerIDPool=10.0.0.0/16 which will ultimately rendered as a cilium-config ConfigMap entry. This is easy to implement, but changing or extending the range requires operator restart.

Option2: Via CRD

We can either extend CiliumBGPPeeringPolicy or introduce another CRD like CiliumBGPRouterIDPool and provide a Router ID from there. It's more complicated in terms of the implementation, but adding/deleting a range can be done dynamically without operator restart.

In my personal opinion, option 1 should be enough for most cases because the uniqueness of the Router ID needs to be guaranteed only within AS these days RFC6286. Thus, it's easy for operators to allocate a large range. Also, it is possible to start from option 1 and support option 2 later.

Retaining Router ID over restart

The operator and agent must retain the Router ID over restart. This is important for Graceful Restart. In some implementations, when the Router ID of the restarting node changes, the Graceful Restart fails.

The operator can persist the assignment by writing assigned Router ID to either CiliumNode object or CiliumBGPNodeConfig object in BGP Control Plane v2.

[kishiguro]

@YutaroHayakawa How about assign unique 32 bit id to the instance.NewServerWithConfig() or gobgp.NewGoBGPServer()? we may have slice for it and use slice's index + 1 or index + 10.0.0.1 as router-id. Since we only need unique identification, having IPAM feels little bit overkill to me.

[YutaroHayakawa]

@kishiguro Would you mind elaborating more? The problem we want to solve here is assigning router-id for each GoBGP instance on each k8s node in the cluster. Thus, someone needs to manage global assignments. instance.NewServerWithConfig() or gobgp.NewGoBGPServer() are called locally within a single node, so we cannot do global allocation there.

On the other hand, I understand the sentiment of "having IPAM feels little bit overkill". If we have any way to assign router-id statelessly, that would be the preferred solution. However, I couldn't come up with such a way for IPv6 single-stack env.

[iandrewt]

Proxmox implemented something similar for their SDN solution by deriving a router ID from the last 4 bytes of the interface's MAC address. I think having this as an option with a disclaimer that all nodes sharing an ASN need to have unique MAC addresses could solve for the stateless router-id configuration. There would probably also need to be a restriction on only allowing this setup with private range ASNs

https://git.proxmox.com/?p=pve-network.git;a=commit;h=7638f25336ab2c05edc8b0c306176b356da1a98e

[kishiguro]

@iandrewt thanks for the insight. we can do same approach. let me play around with my code.