Flannel using new CNI API to support Discontiguous Cluster CIDR

[rbrtbnfgl]

Is your feature request related to a problem? Please describe.

From 1.26 K8s supports discontinuous CIDR for the pod https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2593-multiple-cluster-cidrs
On Flannel there is a WIP to support this feature flannel-io/flannel#1658

Describe the solution you'd like

Enable the same feature on K3s

Describe alternatives you've considered

Additional context

[VestigeJ]

I tested some functionality but I'm going to wait until Monday to close this after conferring more with engineering.

healthy cluster three server nodes

$ kg cc -A

NAME                   PERNODEHOSTBITS   IPV4           IPV6     AGE
default-cluster-cidr   8                 10.42.0.0/16   <none>   2m38s

$ get_figs

=========== k3s config =========== 
write-kubeconfig-mode: 644
debug: true
token: watermelongarlicaces
selinux: true
protect-kernel-defaults: true
cluster-init: true
multi-cluster-cidr: true

$ vim newcc.yaml
$ cat newcc.yaml

apiVersion: networking.k8s.io/v1alpha1
kind: ClusterCIDR
metadata:
  name: new-cidr
spec:
  nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          -  "worker2"
  perNodeHostBits: 8
  ipv4: 10.247.0.0/16

$ k apply -f newcc.yaml
$ kg cc -A

NAME                   PERNODEHOSTBITS   IPV4            IPV6     AGE
default-cluster-cidr   8                 10.42.0.0/16    <none>   8m39s
new-cidr               8                 10.247.0.0/16   <none>   5s

$ k cluster-info dump | grep -i podcidr

                "podCIDR": "10.42.2.0/24",
                "podCIDRs": [
                "podCIDR": "10.42.0.0/24",
                "podCIDRs": [
                "podCIDR": "10.42.1.0/24",
                "podCIDRs": [

$ k cluster-info dump | grep -i multi-cluster-cidr

                    "k3s.io/node-args": "[\"server\",\"--write-kubeconfig-mode\",\"644\",\"--debug\",\"true\",\"--token\",\"********\",\"--server\",\"https://3.23.85.54:6443\",\"--selinux\",\"true\",\"--protect-kernel-defaults\",\"true\",\"--cluster-init\",\"true\",\"--multi-cluster-cidr\",\"true\"]",
                    "k3s.io/node-args": "[\"server\",\"--write-kubeconfig-mode\",\"644\",\"--debug\",\"true\",\"--token\",\"********\",\"--selinux\",\"true\",\"--protect-kernel-defaults\",\"true\",\"--cluster-init\",\"true\",\"--multi-cluster-cidr\",\"true\"]",
                    "k3s.io/node-args": "[\"server\",\"--write-kubeconfig-mode\",\"644\",\"--debug\",\"true\",\"--token\",\"********\",\"--server\",\"https://3.23.85.54:6443\",\"--selinux\",\"true\",\"--protect-kernel-defaults\",\"true\",\"--cluster-init\",\"true\",\"--multi-cluster-cidr\",\"true\"]",

$ cat invalidcidrcollision.yaml

apiVersion: networking.k8s.io/v1alpha1
kind: ClusterCIDR
metadata:
  name: invalid-cidr-collision
spec:
  nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          -  "worker2"
  perNodeHostBits: 8
  ipv4: 10.247.0.0/16

$ cat targetanodename.yaml

apiVersion: networking.k8s.io/v1alpha1
kind: ClusterCIDR
metadata:
  name: node-attack-cidr
spec:
  nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          -  "ip-172-31-24-232"
  perNodeHostBits: 8
  ipv4: 10.247.0.0/16

$ kgn -o wide

NAME               STATUS   ROLES                       AGE   VERSION                INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                              KERNEL-VERSION                 CONTAINER-RUNTIME
ip-172-31-16-60    Ready    control-plane,etcd,master   17m   v1.26.2+k3s-99805041   172.31.16.60    <none>        SUSE Linux Enterprise Server 15 SP4   5.14.21-150400.24.11-default   containerd://1.6.19-k3s1
ip-172-31-19-110   Ready    control-plane,etcd,master   25m   v1.26.2+k3s-99805041   172.31.19.110   <none>        SUSE Linux Enterprise Server 15 SP4   5.14.21-150400.24.11-default   containerd://1.6.19-k3s1
ip-172-31-24-232   Ready    control-plane,etcd,master   17m   v1.26.2+k3s-99805041   172.31.24.232   <none>        SUSE Linux Enterprise Server 15 SP4   5.14.21-150400.24.11-default   containerd://1.6.19-k3s1

$ kgp -A -o wide

NAMESPACE     NAME                                      READY   STATUS      RESTARTS      AGE   IP          NODE               NOMINATED NODE   READINESS GATES
kube-system   coredns-7c444649cb-vbj7j                  1/1     Running     0             25m   10.42.0.5   ip-172-31-19-110   <none>           <none>
kube-system   helm-install-traefik-8hgv5                0/1     Completed   1             25m   10.42.0.6   ip-172-31-19-110   <none>           <none>
kube-system   helm-install-traefik-crd-pplhp            0/1     Completed   0             25m   10.42.0.2   ip-172-31-19-110   <none>           <none>
kube-system   local-path-provisioner-5d56847996-xggmc   1/1     Running     0             25m   10.42.0.3   ip-172-31-19-110   <none>           <none>
kube-system   metrics-server-7b67f64457-xktqm           1/1     Running     0             25m   10.42.0.4   ip-172-31-19-110   <none>           <none>
kube-system   svclb-traefik-8e1ffe31-44ljf              2/2     Running     0             17m   10.42.1.2   ip-172-31-24-232   <none>           <none>
kube-system   svclb-traefik-8e1ffe31-psx4r              2/2     Running     0             25m   10.42.0.7   ip-172-31-19-110   <none>           <none>
kube-system   svclb-traefik-8e1ffe31-zxph6              2/2     Running     2 (75s ago)   17m   10.42.2.3   ip-172-31-16-60    <none>           <none>
kube-system   traefik-56b8c5fb5c-65vmg                  1/1     Running     0             25m   10.42.0.8   ip-172-31-19-110   <none>           <none>

$ kg cc -A

NAME                     PERNODEHOSTBITS   IPV4            IPV6     AGE
default-cluster-cidr     8                 10.42.0.0/16    <none>   27m
invalid-cidr-collision   8                 10.247.0.0/16   <none>   12m
new-cidr                 8                 10.247.0.0/16   <none>   19m
node-attack-cidr         8                 10.247.0.0/16   <none>   5m30s

[rbrtbnfgl]

Why are you adding CIDRs with collisions? You should check the logs probably the new CIDRs are not added to the allocator because collide; that's why ip-172-31-24-232 gets an IP from the default and not from the defined CIDR.

[VestigeJ]

I want to understand what happens when we test good behavior as well as bad behavior, that's why I've intentionally added CIDR collisions.

[brandond]

This is enabling upstream functionality; I don't think we're on the hook to fix it if we don't like how it behaves when misconfigured.

[VestigeJ]

$ kg cc -A

NAME                       PERNODEHOSTBITS   IPV4           IPV6     AGE
default-cluster-cidr       8                 10.42.0.0/16   <none>   111m
okay-cidr                  8                 13.97.0.0/16   <none>   91m
server-node-another-cidr   8                 18.18.0.0/16   <none>   61m

$ kd cc/okay-cidr

Name:         okay-cidr
Labels:       <none>
Annotations:  <none>
NodeSelector:
  NodeSelector Terms:
    Term 0:       kubernetes.io/hostname in [ip-172-31-25-81]
PerNodeHostBits:  8
IPv4:             13.97.0.0/16
Events:           <none>

To my contemporaries the steps are as follows

From a server node not the one targeted by the cidr NodeSelector
$ kubectl apply -f okay-cidr.yaml
$ kubectl drain NODE_NAME --ignore-daemonsets --delete-local-data
$ kubectl delete NODE_NAME
$ etcd member remove HEX_ID_FROM_MEMBER_LIST

From the targeted node
$ sudo k3s-killall.sh && sudo k3s-uninstall.sh
rejoin the cluster, wait for the node to become ready and pods to be deployed...
$ kubectl get pods -A -o wide //expect to see IPv4//IPv6 podCIDR in use for pods on that node.
$ expect the node to successfully rejoin the etcd cluster.

Traefik pods for example

$ kgp -n kube-system -o wide | grep -i svclb

svclb-traefik-50a66d94-br6zs              2/2     Running     0          103m   10.42.6.2   ip-172-31-21-173   <none>           <none>
svclb-traefik-50a66d94-gzgvh              2/2     Running     0          104m   10.42.1.2   ip-172-31-30-129   <none>           <none>
svclb-traefik-50a66d94-jgssr              2/2     Running     0          10m    13.97.0.2   ip-172-31-25-81    <none>           <none>
svclb-traefik-50a66d94-rnkzk              2/2     Running     0          118m   10.42.0.7   ip-172-31-23-22    <none>           <none>