Multus CNI integration.

[RiddlerKnight]

I'm setting up a little lab for POC. I have 4 node, cp = 3, worker = 1. The worker node has 2 physical network interfaces the first one is private and the another is public. I have only 1 public IP available that I want to attach to worker node and expose the nginx ingress controller to public network interfaces.
I don't want to setup external load balancer which increases another complexity.

Currently, I use cilium CNI as the primary and I found that every container has only eth0 attached to --apiserver-advertise-address only.

I not sure can cilium CNI integrate with Multus CNI ?
Have someone experienced this approach?

This is example architect.

[RiddlerKnight]

I do some research and I found this

1. In cilium 1.14.3 helm chart, it's needed to config cni.exclusive=false to prevent cilium change multus config file name to *.cilium_bak in /etc/cni/net.d

2. Following this discussion What is the proper, talos way to install basic (bridge, macvlan etc.) CNI plugins? #7583 , I use bitnami/mutul-cni chart to install multus and add

initContainers:
  - command:
      - /install-cni.sh
    image: ghcr.io/siderolabs/install-cni:v1.5.0
    imagePullPolicy: IfNotPresent
    name: install-cni
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
      - mountPath: /host/opt/cni/bin/
        name: cni-plugin

extraVolumes:
  - name: cni-plugin
    hostPath:
      path: /opt/cni/bin

to install cni-plugin to all node

3. I use ipvlan as NetworkAttachmentDefinition

    {
      "cniVersion": "0.3.1",
      "type": "ipvlan",
      "mode": "l2",
      "master": "ens192",
      "ipam": {
        "type": "host-local",
        "subnet": "103.233.x.x/24",
        "rangeStart": "103.233.x.x",
        "rangeEnd": "103.233.x.x"
      }
    }

4. attach to ingress controller pod.

podAnnotations:
  k8s.v1.cni.cncf.io/networks: |
    [{
      "name": "ipvlan-conf-wk-pub-ip",
      "interface": "net1",
      "namespace": "multus"
    }]

Now everything look good no errors and status annotation in the pod shows the interface attached to the pod but still not access ingress from pub-ip. I'm not sure why.