System reboot pulls new cilium/cilium:latest image, resulting in crashloopbackoff with wireguard interface

[Combustible]

Bug report

Summary

One of my kubernetes nodes rebooted unexpectedly - which has in the past been fine/recoverable. Today though, it pulled a new cilium image (cilium/cilium:latest) because the ImagePullPolicy is Always, instead of IfNotPresent. This new image resulted in a crash loop for cilium, breaking my cluster until I figured out the image changed and I tagged the prior image and edited the daemonset to force it to be used.

There are three problems here:

• I don't believe a system reboot should silently upgrade the cilium docker image to the latest one, potentially breaking the node as it did in my case (i.e. the ImagePullPolicy should be IfNotPresent, rather than Always)
• Even if the ImagePullPolicy needs to be Always for security updates, installing Cilium using this helm chart should not have picked cilium/cilium:latest, rather than cilium/cilium:v1.7.
• Both latest and v1.7 are broken on my system. I am connecting my nodes together with WireGuard, which does not have a MAC address. The current script seems to die trying to find its mac address, which was apparently not needed in the past.

General Information

• Cilium version

Client: 1.7.90 822a64e46 2020-04-13T14:48:20-07:00 go version go1.14.2 linux/amd64
Daemon: 1.7.90 822a64e46 2020-04-13T14:48:20-07:00 go version go1.14.2 linux/amd64

• Kernel version : 5.3.0-59-generic
• Orchestration system version in use: Kubernetes/Kubeadm

Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

• Original installation process was basically:

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.7.0 \
    --namespace kube-system \
    --set global.kubeProxyReplacement=strict \
    --set global.k8sServiceHost=172.30.200.8 \
    --set global.k8sServicePort=6443 \
        --set global.nodePort.range='3300\,32000'```

• Working docker image:

cilium/cilium (ID: 0d45ffd70126 from 2020-04-24T02:57:19.607562726Z)

• Broken docker images:

cilium/cilium:latest (ID: a396a542ce1d from  2020-06-13T04:25:34.540406702Z)
cilium/cilium:v1.7 (ID: 00ccab104fb9 from 2020-06-12T13:05:47.387468634Z)

Log from cilium pod as it tries to start up and fails:
https://gist.github.com/Combustible/2868225d86b3080d460478745c718b49

Relevant snippet:

level=warning msg="++ ip link show wg0" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="+ MAC=" subsys=datapath-loader
level=error msg="Error while initializing daemon" error="exit status 1" subsys=daemon
level=fatal msg="Error while creating daemon" error="exit status 1" subsys=daemon
level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager

Actual output from ip link show wg0:

5: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none

[rreinurm]

In general latest tag is not recommended practice for production systems with the most of the docker images unless this is what you truly desire. You should pinpoint images to use minor and patch versions to avoid that issue. You can find references for that from several articles, for example https://docs.docker.com/develop/dev-best-practices

When building images, always tag them with useful tags which codify version information, intended destination (prod or test, for instance), stability, or other information that is useful when deploying the application in different environments. Do not rely on the automatically-created latest tag.

Additionally for reboot, the same issue can happen then you add new nodes to Kubernetes cluster meaning that you have different versions of images for same service.

I only wonder if you used helm-chart to install Cilium, you already locked to release --version 1.7.0. Is it possible that you changed image to latest manually?

[brb]

@Combustible you should really avoid using latest as recommended per https://kubernetes.io/docs/concepts/containers/images/#image-names. Anyway, I'm wondering how did you end up using latest considering that you specified --version 1.7.0.

Re wireguard: Do you use the wireguard tunnel for communication among k8s nodes?

[Combustible]

Anyway, I'm wondering how did you end up using latest considering that you specified --version 1.7.0.

This is what I'm wondering too :) - I am almost certain I never specified latest myself. I know the recommendation is not to do this, for exactly the breakage reasons I experienced here. I assumed that the helm chart would set me to a reasonable docker image (i.e. using 1.7.0 would point to v1.7). If the image pull policy was set to IfNotPresent, there would never have been any updating, even within the same major version - it'd have just re-used the locally present image. I'd have noticed the problem with the 1.7 docker image/wireguard when I went to add a new node, but that would have been much less disruptive.

Re wireguard: Do you use the wireguard tunnel for communication among k8s nodes?

Yes, I use wireguard for all of my inter-node communication between k8s nodes. My setup is somewhat simple - I have two nodes - a master and an auxiliary, both baremetal servers hosted on the public internet, with a public link between them. The auxiliary has no externally accessible ports - all of the IPTables rules allow only connections from the wireguard interface or 10.x.x.x that kubernetes tunnels over it. All external traffic goes to the master node. This took some fiddling to get set up but seems to work very nicely, and performance is adequate for my needs. I have a lot of NFS mounts between the two servers, which is why I'm using wireguard to protect everything instead of cilium's built-in encryption scheme. I'm sure this is entirely overkill, but I appreciate having my internal traffic between nodes be integrity protected / encrypted.

I'm running a ton of stuff, the vast majority on the master node, and primarily using the auxiliary node for overflow when the master server doesn't have enough RAM.

I use 172.30.200.x for my wireguard subnet, and 10.x.x.x for the kubernetes CIDR.

[brb]

Closing in favor of #12317.