Conflict with istio-cni-node when cilium is the main CNI plugin

[demikl]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Cilium overwrites the configuration that istio-cni wants to append to the cilium cni config file.

Cilium Version

v1.14.0-rc.0

Kernel Version

Linux ip-100-89-14-191.eu-west-3.compute.internal 5.10.184-175.731.amzn2.x86_64 #1 SMP Tue Jun 27 21:48:55 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.14-eks-c12679a", GitCommit:"05d192f0de17608d98e17761ad3cffa9a6407f2f", GitTreeState:"clean", BuildDate:"2023-05-22T23:41:27Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

No response

Relevant log output

2023-07-11T08:19:29.414585Z    info    install    CNI config file /host/etc/cni/net.d/05-cilium.conflist exists. Proceeding.
2023-07-11T08:19:29.415159Z    info    install    Created CNI config /host/etc/cni/net.d/05-cilium.conflist
2023-07-11T08:19:29.415192Z    info    install    CNI configuration and binaries reinstalled.
2023-07-11T08:19:29.430298Z    info    install    Detect changes to the CNI configuration and binaries, attempt reinstalling...

Anything else?

istio-cni appends its configuration to /etc/cni/net.d/05-cilium.conflist:

{
  "cniVersion": "0.3.1",
  "name": "cilium",
  "plugins": [
    {
      "enable-debug": false,
      "log-file": "/var/run/cilium/cilium-cni.log",
      "type": "cilium-cni"
    },
    {
      "kubernetes": {
        "cni_bin_dir": "/opt/cni/bin",
        "exclude_namespaces": [
          "istio-system",
          "kube-system"
        ],
        "kubeconfig": "/etc/cni/net.d/ZZZ-istio-cni-kubeconfig"
      },
      "log_level": "info",
      "log_uds_address": "/var/run/istio-cni/log.sock",
      "name": "istio-cni",
      "type": "istio-cni"
    }
  ]
}

but cilium immediately reverts it back.

Using cni-exclusive: false does not change this behavior.

Code of Conduct

• I agree to follow this project's Code of Conduct

[demikl]

cilium sysdump attached
cilium-sysdump-20230711-152150.zip

[demikl]

As far as I can understand, the behavior has changed recently through #24389
The startup behavior takes into account the cni-exclusive flag, but the goroutine that constantly watches any changes in the CNI configuration directory is not using this flag.

[julianwiedmann]

@squeed sounds like right down your alley - any pointers? :)

[squeed]

Interesting; it looks like both Cilium and Istio both want to own this configuration file.

My proposal would be to fix this by adding a new Helm value, cni.writeOnce, which disables the file watcher and, instead, writes once and exits.

[squeed]

Aha, that sounds like a use-case the code failed to anticipate.

I think it would be reasonable to disable the CNI file watcher if cni.exclusive is set to false. I'll submit a change for that.

It will get backported, but might not make v1.14.0.

[squeed]

Filed #26773 to disable the file-watcher when cni.exclusive is false.

[demikl]

Thanks for handling this. Do you have a build that I can test ?

[squeed]

@demikl it's not exactly production grade, but you can pull the CI image built in this action from quay.io/cilium/cilium-ci:e1d97737c2836c42aae7b30ca8b152622ae7e0c8@sha256:0d94072b57aa04efd5808239bd63021446045b404dc39c607d04bf5b7dae8f12