Bump cilium to version 1.12

[7oku]

Our story

We need to use cilium on Hetzner Cloud, because traffic flowing from lb (internal ips) => kube-proxy (snat*) => svc/pod on the same node will always originate from the public eth0 ip address of the node, thus breaking a lot of things for us (i.e. services always need to know all public ips in the cluster to exclude them from x-forwarded-for/proxy protocol). We invested a lot of time and tried many tweaks to convince canal/flannel to use the internal ip, but without luck.

With a switch to cilium v1.11.x we essentially fixed this entirely in a snap using the strict kube-proxy replacement - cilium agent will use the private ip out of the box instead of the public one to talk to pods on the same host. We also benefit from the other cilium features (hubble insights, bpf performance ...) and are therefore very happy to find kubeone supporting cilium. However, afair kubeone 1.4.5 installs cilium v1.11.1, which is quite old and showed some bugs in our cluster, which have been resolved in the last months and are gone in cilium v1.12 (stable).

For testing purposes I updated cilium manually with your given helm command and everything seems to be working very well since over a week. I couldn't make out any breaking changes in the release notes. I'm aware cilium v1.12 is really new as it has been released just about 2 weeks ago and understand absolutely it probably needs some testing.

What would be needed to have you bump this version of cilium into kubeone? Alternatively, would it be possible to just be able to configure the desired cilium version in kubeone.yml?

Solution details

• bump cilium to stable 1.12

Alternative approaches

• allow to configure cilium version in kubeone.yml

Use cases

• additional features like native service mesh and ingress support
• mainly bug fixes

[xmudrii]

We tend to avoid upgrading to another major or minor release of any component for existing KubeOne releases. That said, there are currently no plans to upgrade Cilium to 1.12 for KubeOne 1.4 release, but we can consider including it in the upcoming KubeOne 1.5 release.

Until then, as an alternative, you can override the Cilium addon used by KubeOne. The overriding process is briefly described in the Addons document. What you need to do is to:

• enable addons in your KubeOneCluster manifest (.addon.enable: true) and set some addons path (e.g. .addon.path: ./addons)
• in the selected addons directory (e.g. ./addons) create cni-cilium directory (so that you have ./addons/cni-cilium)
• in the cni-cilium directory put all the manifests needed to deploy Cilium 1.12
• when you run kubeone apply, KubeOne will consume manifests that you provided instead of manifests built in KubeOne

Regarding allowing configuring Cilium/CNI version in kubeone.yaml, this is something that we're planning eventually for KubeOne 1.6, but even with that feature, you couldn't put any version. We would have to first test that version and then allow it if it works.

[7oku]

Absolutely, I was not expecting to see that bump happening in a KubeOne 1.4.x release. There was more intention to have it considered at all for KubeOne, so I'm very glad you added it as a milestone for KubeOne 1.5 already!

I'll try the addon approach tomorrow as well, looks promising. Thanks for pointing that out and adding the milestone!

[madalinignisca]

@xmudrii

when you run kubeone apply, KubeOne will consume manifests that you provided instead of manifests built in KubeOne

Does this mean that it will use an addon from local if it has the same name, or that all built in addons are ignored and only the local added will be used?

For example, if I add hetzner ccm and cilium, all other default or manually mentioned will still be used from built in?

And if I add cilium, how do I disable kubeone's default cni?

[xmudrii]

@madalinignisca If you provide an addon with the same name as the built-in addon, that addon will be overridden. Other addons will use manifests built-in in KubeOne.

Regarding Hetzner CCM and Cilium, both are integrated in KubeOne already. For the Hetzner CCM, you need to set .cloudProvider.external: true, and for Cilium you need to set .clusterNetwork.cni.cilium in your KubeOneCluster manifest. For example:

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
name: demo-cluster
cloudProvider:
  hetnzer: {}
  external: true
clusterNetwork:
  kubeProxy:
    # skipInstallation will skip the installation of kube-proxy
    skipInstallation: true # set to false if kubeProxyReplacement is disabled
  cni:
    cilium:
      enableHubble: true # or false if you don't want Hubble
      kubeProxyReplacement: "strict" # or disabled if you don't want kube-proxy replacement

This manifest will provision a cluster with the Hetzner CCM and Cilium (with kube-proxy replacement and Hubble).

[madalinignisca]

I will try with most recent hetzner ccm with networks and cillium. Thx.