Multus fails to create secondary interface for Cilium 1.14.0

[thatsdone]

What happend:

When used with Cilium 1.14.0, multus-cni secondary interface does not appear.
(I confirmed it works for Cilium 1.13.3.)

What you expected to happen:

Secondary interface appears as specified by NetworkAttachmentDefinition custom resource.

How to reproduce it (as minimally and precisely as possible):

Set up a kubernetes cluster using Cilium 1.14.0, install multus-cni and try to create NAD and a pod following quickstart.md

Anything else we need to know?:

Please note that I'm using cri-o 0.1.0/1.27.0
Also, I used '/deployments/multus-daemonset.yml' (crio version didn't work).

Environment:

• Multus version
  image path and image ID (from 'docker images')
  ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:35accc0395c03a8f0706304d79d670602debf4b31228f9da76b8e415c0523387

• Kubernetes version (use kubectl version): v1.27.4

• Primary CNI for Kubernetes cluster: Ciliuim 1.14.0

• OS (e.g. from /etc/os-release): Ubuntu 22.04.1(amd64)

• File of '/etc/cni/net.d/'

• File of '/etc/cni/multus/net.d'

• NetworkAttachment info (use kubectl get net-attach-def -o yaml)
  I followed https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/quickstart.md#storing-a-configuration-as-a-custom-resource

• Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)

• Other log outputs (if you use multus logging)

$ for pod  in `kubectl get pod -A | grep mult | awk '{print $2;}'`; do    echo ${pod};
kubectl logs -n kube-system ${pod} --tail=-1;    echo ===============; done
kube-multus-ds-ghshv
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (in
it)
kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig
kubeconfig file is created.
master capabilities is get from conflist
multus config file is created.
===============
kube-multus-ds-mwhpj
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init)
kubeconfig is created in /host/etc/cni/net.d/multus.d/multus.kubeconfig
kubeconfig file is created.
master capabilities is get from conflist
multus config file is created.
===============

[cyclinder]

Could you describe the pod's events?

[thatsdone]

Hi @cyclinder , please find kubectl describe pod result below (reproduced case).

As you can see, events from multus are missing, I think.

$ kubectl get pod
NAME            READY   STATUS    RESTARTS   AGE
user1-ubuntu1   1/1     Running   0          77s
$ kubectl describe  pod  user1-ubuntu1
Name:             user1-ubuntu1
Namespace:        default
Namespace:        default
Priority:         0
Service Account:  default
Node:             giga-node-1/192.168.203.101
Start Time:       Sun, 30 Jul 2023 17:04:53 +0900
Labels:           <none>
Annotations:      k8s.v1.cni.cncf.io/networks: macvlan-conf
Status:           Running
IP:               10.0.1.190
IPs:
  IP:  10.0.1.190
Containers:
  user1-ubuntu1:
    Container ID:  cri-o://324cc1b57312d2e8b35412a1253958184421ecdbe2d60e1bf783ece56c21ff32
    Image:         ubuntu:22.04
    Image ID:      docker.io/library/ubuntu@sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
    Args:
      86400
    State:          Running
      Started:      Sun, 30 Jul 2023 17:05:00 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wtcjn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-wtcjn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  83s   default-scheduler  Successfully assigned default/user1-ubuntu1 to giga-node-1
  Normal  Pulling    51s   kubelet            Pulling image "ubuntu:22.04"
  Normal  Pulled     44s   kubelet            Successfully pulled image "ubuntu:22.04" in 6.642825096s (6.642854386s including waiting)
  Normal  Created    44s   kubelet            Created container user1-ubuntu1
  Normal  Started    44s   kubelet            Started container user1-ubuntu1
$

[thatsdone]

If I use Cilium 1.13.3, I can see multus events like below, and the secondary interface indeed appears.

$ cilium version
cilium-cli: v0.14.6 compiled with go1.20.4 on linux/amd64
cilium image (default): v1.13.3
cilium image (stable): v1.14.0
cilium image (running): v1.13.3
$ date; kubectl get pod
Thu Aug  3 10:48:00 PM JST 2023
NAME            READY   STATUS    RESTARTS   AGE
user1-alpine1   1/1     Running   0          10s
$ date; kubectl describe pod  user1-alpine1
Thu Aug  3 10:48:09 PM JST 2023
Name:             user1-alpine1
Namespace:        default
Priority:         0
Service Account:  default
Node:             giga-node-3/192.168.203.103
Start Time:       Thu, 03 Aug 2023 22:47:50 +0900
Labels:           <none>
Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "cilium",
                        "interface": "eth0",
                        "ips": [
                            "10.0.2.185"
                        ],
                        "mac": "da:3a:7a:a2:d6:fe",
                        "default": true,
                        "dns": {},
                        "gateway": [
                            "10.0.2.82"
                        ]
                    },{
                        "name": "default/macvlan-conf",
                        "interface": "net1",
                        "ips": [
                            "192.168.1.200"
                        ],
                        "mac": "be:d8:44:41:7f:49",
                        "dns": {},
                        "gateway": [
                            "\u003cnil\u003e"
                        ]
                    }]
                  k8s.v1.cni.cncf.io/networks: macvlan-conf
Status:           Running
IP:               10.0.2.185
IPs:
  IP:  10.0.2.185
Containers:
  user1-alpine1:
    Container ID:  cri-o://24664478be1e93eb43848dd13e6571d70360873881d82a6eaa4faa6ba5ddf93d
    Image:         alpine
    Image ID:      docker.io/library/alpine@sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/ash
      -c
      trap : TERM INT; sleep infinity & wait
    State:          Running
      Started:      Thu, 03 Aug 2023 22:47:56 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lsb79 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-lsb79:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       19s   default-scheduler  Successfully assigned default/user1-alpine1 to giga-node-3
  Normal  AddedInterface  19s   multus             Add eth0 [10.0.2.185/32] from cilium
  Normal  AddedInterface  19s   multus             Add net1 [192.168.1.200/24] from default/macvlan-conf
  Normal  Pulling         18s   kubelet            Pulling image "alpine"
  Normal  Pulled          13s   kubelet            Successfully pulled image "alpine" in 5.603754269s (5.603775594s including waiting)
  Normal  Created         13s   kubelet            Created container user1-alpine1
  Normal  Started         13s   kubelet            Started container user1-alpine1
$

[cyclinder]

Could you show cni files in your node？(/etc/cni/net.d)

[thatsdone]

Could you show cni files in your node？(/etc/cni/net.d)

Ah, I need to rebuild the system, I'll be back in a couple of days.
Also, do you have any ideas to increase debug information? Such, as log level option of multus service.

[thatsdone]

Hi @cyclinder ,

Please find files under /etc/cni/ below.

List of files under /etc/cni after multus installation (on Cilium 1.14 environment)

$ sudo find /etc/cni/ -type f
/etc/cni/net.d/05-cilium.conflist
/etc/cni/net.d/00-multus.conf.cilium_bak
/etc/cni/net.d/100-crio-bridge.conflist.cilium_bak
/etc/cni/net.d/200-loopback.conflist.cilium_bak
/etc/cni/net.d/multus.d/multus.kubeconfig

/etc/cni/net.d/05-cilium.conflist

{
  "cniVersion": "0.3.1",
  "name": "cilium",
  "plugins": [
    {
       "type": "cilium-cni",
       "enable-debug": false,
       "log-file": "/var/run/cilium/cilium-cni.log"
    }
  ]
}

/etc/cni/net.d/00-multus.conf.cilium_bak

{
        "cniVersion": "0.3.1",
        "name": "multus-cni-network",
        "type": "multus",
        "cniConf": "/host/etc/cni/multus/net.d",
        "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",
        "delegates": [
                {"cniVersion":"0.3.1","name":"cilium","plugins":[{"enable-debug":false,"log-file":"/var/run/cilium/cilium-cni.log","type":"cilium-cni"}]}
        ]
}

/etc/cni/net.d/100-crio-bridge.conflist.cilium_bak

{
  "cniVersion": "1.0.0",
  "name": "crio",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [
            { "dst": "0.0.0.0/0" },
            { "dst": "::/0" }
        ],
        "ranges": [
            [{ "subnet": "10.85.0.0/16" }],
            [{ "subnet": "1100:200::/24" }]
        ]
      }
    }
  ]
}

/etc/cni/net.d/200-loopback.conflist.cilium_bak

{
    "cniVersion": "1.0.0",
    "name": "loopback",
    "plugins": [
        {
            "type": "loopback"
        }
    ]
}

/etc/cni/net.d/multus.d/multus.kubeconfig

# Kubeconfig file for Multus CNI plugin.
apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: https://[10.96.0.1]:443
    certificate-authority-data: CERTIFICATE-AUTHORITY-DATA (masked)
users:
- name: multus
  user:
    token: TOKEN (maked)
contexts:
- name: multus-context
  context:
    cluster: local
    user: multus
current-context: multus-context

[cyclinder]

My guess is correct. /etc/cni/net.d/05-cilium.conflist is your first cni flle in /etc/cni/net.d, you should set cni.exclusive to false while cilium installing.

[thatsdone]

Hi @cyclinder

Bingo !!
By setting cni.exclusive as false, finally my multus installation worked.
Thanks a lot!

BTW, maybe it's a good idea to mention this in the installation note of multus, IMHO.

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.4
Kustomize Version: v5.0.1
Server Version: v1.27.4
$ cilium version
cilium-cli: v0.15.4 compiled with go1.20.4 on linux/amd64
cilium image (default): v1.13.4
cilium image (stable): v1.14.0
cilium image (running): 1.14.0
$ kubectl get pod
NAME            READY   STATUS              RESTARTS   AGE
user1-alpine1   0/1     ContainerCreating   0          5s
$ kubectl describe  pod  user1-alpine1
Name:             user1-alpine1
Namespace:        default
Priority:         0
Service Account:  default

(snip)

Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       13s   default-scheduler  Successfully assigned default /user1-alpine1 to franklin-node-1
  Normal  AddedInterface  12s   multus             Add eth0 [10.245.1.21/32] from cilium
  Normal  AddedInterface  12s   multus             Add net1 [192.168.1.200/24] f
rom default/macvlan-conf
  Normal  Pulling         12s   kubelet            Pulling image "alpine"
  Normal  Pulled          8s    kubelet            Successfully pulled image "alpine" in 4.119452958s (4.119473778s including waiting)
  Normal  Created         8s    kubelet            Created container user1-alpine1
  Normal  Started         8s    kubelet            Started container user1-alpine1

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[thatsdone]

Hi, I'm closing this issue because my problem was resolved.