Support defining IPAM pools using CiliumPodIPPool CRD

[tklauser]

This PR introduces the CiliumPodIPPool CRD and adds support to use it to define IP pools in Cilium's multi-pool IPAM mode (ref. #22762).

An example CiliumPodIPPool CRD definition of the pool named jupiter with the IPv4 CIDRs 10.10.0.0/16 and 10.20.0.0/16, an IPv4 mask size of 27 bits, the IPv6 CIDR fd00:100::/80 and an IPv6 mask size of 96 bits looks as follows:

apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  name: jupiter
spec:
  ipv4:
    cidrs:
      - 10.10.0.0/16
      - 10.20.0.0/16
    maskSize: 27
  ipv6:
    cidrs:
      - fd00:100::/80
    maskSize: 96

These CRDs are watched by cilium-operator. Deletion and update of existing pools are supported, however updates changing mask sizes on existing pools are rejected.

Please see individual commits for details.

Manually tested on kind as follows:

$ WORKERS=2 make kind
$ make kind-image
$ helm upgrade --install cilium ./install/kubernetes/cilium --namespace kube-system \
    --set debug.enabled=true \
    --set image.override=localhost:5000/cilium/cilium-dev:local \
    --set image.pullPolicy=IfNotPresent \
    --set operator.image.override=localhost:5000/cilium/operator-generic:local \
    --set operator.image.pullPolicy=IfNotPresent \
    --set operator.replicas=1 \
    --set hubble.relay.enabled=true \
    --set ipam.mode=multi-pool \
    --set "ipam.operator.multiPoolMap.default.ipv4CIDRs[0]=10.10.0.0/16" \
    --set "ipam.operator.multiPoolMap.default.ipv4MaskSize=27" \
    --set tunnel=disabled \
    --set autoDirectNodeRoutes=true \
    --set ipv4NativeRoutingCIDR=10.0.0.0/8 \
    --set endpointRoutes.enabled=true \
    --set-string extraConfig.enable-local-node-route=false \
    --set kubeProxyReplacement=strict \
    --set bpf.masquerade=true
$ cat <<EOF | k apply -f -
apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  name: mars
spec:
  ipv4:
    cidrs:
      - 10.20.0.0/16
    maskSize: 24
EOF
$ k get ciliumpodippools
NAME      AGE
mars      10s
$ # note that the default pool was configured using the operator flag and doesn't show up here (yet). This will be addressed in a follow-up PR.
$ k create -f https://raw.githubusercontent.com/cilium/cilium/1.13.3/examples/minikube/http-sw-app.yaml
$ k -n default get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
deathstar-8464cdd4d9-9bc24   1/1     Running   0          12s   10.10.0.42   kind-worker2   <none>           <none>
deathstar-8464cdd4d9-s5nfm   1/1     Running   0          12s   10.10.0.29   kind-worker    <none>           <none>
tiefighter                   1/1     Running   0          12s   10.10.0.20   kind-worker    <none>           <none>
xwing                        1/1     Running   0          12s   10.10.0.3    kind-worker    <none>           <none>
$ # note that all pods have IPs from the default pool's CIDRs
$ k create ns cilium-test
$ k annotate ns cilium-test ipam.cilium.io/ip-pool=mars
$ cilium connectivity test
...
✅ All 42 tests (295 actions) successful, 13 tests skipped, 0 scenarios skipped.
$ k -n cilium-test get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP            NODE                 NOMINATED NODE   READINESS GATES
client-6f6788d7cc-7fw9w               1/1     Running   0          8m56s   10.20.0.238   kind-worker          <none>           <none>
client2-bc59f56d5-hsv2g               1/1     Running   0          8m56s   10.20.0.193   kind-worker          <none>           <none>
echo-external-node-787c859b66-zrblr   0/1     Pending   0          8m55s   <none>        <none>               <none>           <none>
echo-other-node-646976b7dd-5zlr4      2/2     Running   0          8m56s   10.20.1.145   kind-worker2         <none>           <none>
echo-same-node-58f99d79f4-4k5v4       2/2     Running   0          8m56s   10.20.0.202   kind-worker          <none>           <none>
...
$ # note the different IP range (from pool mars) for the cilium-test pods

The following will be added in a separate follow-up PR, see meta issue #25470:

• Creation of CiliumPodIPPools from operator ipam.operator.multiPoolMap helm flags
• CI coverage
• Documentation

Extends #24764

[gandro]

Awesome work! Left a bit of early feedback

[tklauser]

/test

[tklauser]

/test

[joamaki]

LGTM. A nice to have would be to start modularizing operator's IPAM related code as we're touching these parts.

[joamaki]

How hard would it be to implement this as a hive module instead? I suppose we'd need to pull out PooledAllocatorProvider first from the legacy code plus make alloc.Start depend on IPPoolAllocator to make sure it's started first?

[tklauser]

I initially thought about implementing this as a hive cell. Unfortunately, it would require lifting quite a bit of IPAM code around. For example, I think we cannot make alloc.Start depend on IPPoolAllocator unconditionally, since only once specific implementation (namely ipam/allocator/multipool.Allocator) requires the pooled allocator, while all the others don't. Initially, I wanted to include IPPoolAllocator into ipam/allocator.AllocatorProvider, but IIRC that lead to nasty cyclic dependencies. Also, given this PR is already quite large and should make it into the tree for 1.14, I didn't want to include additional refactoring.

Admittedly it's not ideal to introduce new code in a non-modular fashion, but based on the above I'd defer addressing modularization of this code together with the other IPAM bits to a follow-up PR. I'll discuss the specifics with @gandro and @cilium/sig-foundations in a Slack thread or GH issue. Hope that's OK?

[gandro]

Yeah, given the feature freeze coming up in a week, I don't think it's realistic to tackle a larger refactor at the moment unfortunately 😿

The upside is that this is very little and simple code, so it should not make any future refactor more complicated

[gandro]

Awesome work!

[gandro]

Merge button is green! 🚢