[Backport release-1.23] Add whereabouts IPAM to Multus

[thomasferrandiz]

This is a backport for issue #3008.

[mdrahman-suse]

Validated on v1.23.8-rc1+rek2r1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux 5.4.0-1009-aws #9-Ubuntu SMP Sun Apr 12 19:46:01 UTC 2020 x86_64 GNU/Linux
"Ubuntu 20.04 LTS"

Cluster Configuration:

Single node, two network interfaces

• Make sure that the instance has two network interfaces [Azure reference: https://github.com/Update multus charts to v3.8-build2021110403 #3043#issue-1269356351]
• Can be viewed in AWS EC2 instance details > Networking view
• Once logged in, run ip a

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    ...
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    ...

• Get the interface name (needed later for testing) [ens5 in my instance]
• On AWS,
  • select the instance
  • go to actions > Networking > Attach Interface
  • fill out the details, Attach and reboot the instance
  • login to instance, run ip a again
  • results should show two network interfaces

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 ...
   ...
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 ...
    ...
3: ens6: <BROADCAST,MULTICAST> mtu 1500 ...
    ...

Config.yaml:

write-kubeconfig-mode: 644
cni: multus,cilium
token: summerheat

Additional files

• rke2-multus-config.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-multus
  namespace: kube-system
spec:
  valuesContent: |-
    rke2-whereabouts:
      enabled: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

2. Copy rke2-multus-config.yaml

$ sudo mkdir -p /var/lib/rancher/rke2/server/manifests && sudo cp rke2-multus-config.yaml /var/lib/rancher/rke2/server/manifests/rke2-multus-config.yml

3. Install rke2 (Don't start yet)

$ curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=<VERSION> sh -

4. Start rke2 sudo systemctl enable rke2-server.service && sudo systemctl start rke2-server.service
5. Create a NetworkAttachmentDefinition:

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "macvlan-main",
    "type": "macvlan",
    "mode": "bridge",
    "master": "ens5",
      "ipam": {
            "type": "whereabouts",
            "range": "192.168.2.225/28"
      }
    }'
EOF

6. Create a pod that uses it:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF

7. Check the pod for details kubectl describe pod samplepod

Replication Results:

• rke2 version used for replication:

rke2 version v1.23.7+rke2r2 (d0c2bd7f1dbd30f5b7bbc2e3c899d2efde979c25)
go version go1.17.5b7

• The pod samplepod remains in ContainerCreating state
• Upon describe, observed the error below

Warning  FailedCreatePodSandBox  3m19s              kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d5ee4065dd2e98fc23fe34d254b5390b11293d21affccfa0a1fe198a5055eeb5": plugin type="multus" name="multus-cni-network" failed (add): [default/samplepod:macvlan-main]: error adding container to network "macvlan-main": You have not configured the storage engine (looks like you're using an invalid `etcd_host` parameter in your config)

Validation Results:

• rke2 version used for validation:

rke2 version v1.23.8-rc1+rke2r1 (bc865ee5801cf5666b4d98dce1ff82901a0cc9b8)
go version go1.17.5b7

• The pod samplepod was in Running state
• Upon describe, No errors were observed and the pod has a second IP address in the configured range "192.168.2.225/28":

$ kubectl describe pod samplepod
Name:         samplepod
Namespace:    default
Priority:     0
Node:         ip-172-31-40-252/172.31.40.252
Start Time:   Tue, 28 Jun 2022 01:36:22 +0000
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "cilium",
                    "interface": "eth0",
                    "ips": [
                        "10.42.0.39"
                    ],
                    "mac": "62:67:a2:00:39:48",
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/macvlan",
                    "interface": "net1",
                    "ips": [
                        "192.168.2.225"
                    ],
                    "mac": "46:b0:9b:f9:33:68",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: macvlan
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "cilium",
                    "interface": "eth0",
                    "ips": [
                        "10.42.0.39"
                    ],
                    "mac": "62:67:a2:00:39:48",
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/macvlan",
                    "interface": "net1",
                    "ips": [
                        "192.168.2.225"
                    ],
                    "mac": "46:b0:9b:f9:33:68",
                    "dns": {}
                }]
              kubernetes.io/psp: global-unrestricted-psp
Status:       Running
IP:           10.42.0.39
IPs:
  IP:  10.42.0.39
...

• Also observed a new pod got created pod/rke2-multus-rke2-whereabouts-xxxx