feat(talos): Add cilium bgpControlPlane support and CNI security settings

[wasurerarenai]

When using VLAN network separation, Cilium's L2 announcements may not traverse the L2 network segments. When this is the case, it can help to have Cilium switch to BGP control plane mode and handle advertising the LoadBalancer IPs via a router instead. This requires a router with BGP support such as pfSense with the FRR package installed (or similar).

In this PR I am re-using the existing CiliumLoadBalancerIPPool from cilium-l2.yaml named "pool". Since this switches from L2 announcements to BGP, their is no conflict and multi-pool IPAM is not required to support both simultaneously. The "LOADBALANCER_CIDR" cluster-wide var is used for tracking this.

The bgp settings maybe movable in the future to outside the talos dict to be used by kube-vip if cilium CNI is not in use.

The cilium-values-init and cilium-values-full partials templates we're also not using some of the Talos required settings from their Cilium CNI install guide. These have been added when the talos distribution is set.

Example config.yaml

distribution: talos
  type: talos
  talos:
    vlan: 1
    bgp:
      enabled: true
      local_asn: 64512
      peer_asn: 64512
      peers:
        - 192.168.1.1
      loadbalancer_network: 10.123.0.0/16

[wasurerarenai]

Previously validation was present to ensure the following config.yaml virtual IPs we're within the node CIDR range. With BGP the LoadBalancers can now be outside of the range, so care should be taken to add exclusions from such future checks if added.

• cloudflare.ingress_vip
• cloudflare.gateway_vip
• cloudflare.tunnel.ingress_vip

[wasurerarenai]

Needed a couple patches to the bgp peering policy to ensure LB IPs were advertised to the router. This worked once the serviceSelector example from cilium was added.

 *=i10.123.0.1/32    x.x.x.10                     100      0 i
 *=i                 x.x.x.11                     100      0 i
 *=i                 x.x.x.12                     100      0 i
 *=i10.123.0.2/32    x.x.x.10                     100      0 i

[onedr0p]

Nice work! I'm certain this config would work with k3s too but I'm not sure where to put the config. I'm still struggling with being happy with the config file in general so this can be iterated on in the future.

[onedr0p]

Maybe having a top level cilium config dict would work to put these values under, or maybe under the cluster or nodes config dict.

I'm open to ideas any way to improve the config file and the options because no matter what I think of the options get messy and intertwined.

Maybe I just go back to a flat config instead of a structured one.

[onedr0p]

I confirmed those cilium changes work on k3s too. I might change the config vars in the future, but this is good for now. Thanks!