CNI (installed by RKE) Crashloopback on Ubuntu 22.04 nodes #3114
Comments
|
How is it possible to run k8s v1.24.X with |
|
Did you find a solution to this? I'm experiencing something similar on Kubernetes 1.25.2-00 with containerd 1.5.9 and 1.6.14, but I'm installing manually, rather than with Rancher. |
|
It turned out that for me, flannel was being restarted because kube-proxy was restarting frequently. I finally determined that the root cause was that I needed to set SystemdCgroup in /etc/containerd.config.toml. This fixed it for me: I'm not sure why that's not set by default on Ubuntu 22.04 (at least it's not on Arm64). Maybe it's fixed in more recent versions of containerd, but containerd doesn't seem to regenerate /etc/containerd, let alone the config.toml on reinstall. |
|
Hi, Sorry for this very late response...
I enable CRI on my Docker Daemon so i don't use Dockershim : gitpod-io/gitpod#5410 (comment)
Pffff yeah... It seems to be an issue with Kernel > 5.4 and iptables implementation. I'll try to get more logs next time ;) |
Wow, so you actually use Would this modification work for an existing cluster? Make the changes inside the VMs for containerd, then modify the kubelet params and do an |
|
Note: I suppose containerd was installed by docker, so you actually have docker installed, but I suppose docker is not in use in your scenario. It would be also interesting to only have containerd installed without docker but it's just a note. For an existing cluster, docker could remain installed, I don't care much if the cluster actually uses containerd. |
|
Sorry again for the delay... !
Yes of course ! You will have a down of your production, but it is quite possible ! This is what I did on my homelab 1 year ago and it still working (upgrades included) !
We don't care about Here an example from my homelab new node
quadeare@atlas:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
13576fa11308 rancher/hyperkube:v1.24.6-rancher1 "/opt/rke-tools/entr…" 10 days ago Up 6 days kube-proxy
53f2a41165ba rancher/hyperkube:v1.24.6-rancher1 "/opt/rke-tools/entr…" 10 days ago Up 6 days kubelet
0cecd6700bce rancher/rke-tools:v0.1.87 "/bin/bash" 10 days ago Created service-sidekick
7dbbcaf6c66d rancher/rke-tools:v0.1.87 "nginx-proxy CP_HOST…" 10 days ago Up 6 days nginx-proxyAnd quadeare@atlas:~$ sudo ctr -n k8s.io c ls
CONTAINER IMAGE RUNTIME
062d4a48f99da36d3be5c8dd9bdad13cea80f716867d71cc086807b22ea076a3 registry.k8s.io/pause:3.6 io.containerd.runc.v2
0ef744b6799e99f27eea0e9d14aad7ab9eb1e3a2167e376a609abab4e0f3b64f registry.k8s.io/pause:3.6 io.containerd.runc.v2
104ebeff985ec3c66a7ef2a1b76fb0746a176c7177bd6773243361ecdf6ce336 registry.k8s.io/pause:3.6 io.containerd.runc.v2
136a2261945e2f30bfe6aa55482490f63cc10c351b4eabee09c1eef887dd7931 docker.io/longhornio/longhorn-manager:v1.2.4 io.containerd.runc.v2
1daa32452b74bdb333c1e3e485453da52df712c968c5316d8293de3de9810f39 docker.io/rancher/mirrored-calico-kube-controllers:v3.22.0 io.containerd.runc.v2
1ddc5a1c2a83b432747b75abd36d4c66a1ca5eae21f6b187c36452fad4ef48cc docker.io/rancher/mirrored-coredns-coredns:1.9.3 io.containerd.runc.v2
2054ff552c6a921faf3f66c7b1e2138349c4f029483cd46ffc329174327e78b7 docker.io/longhornio/longhorn-engine:v1.2.4 io.containerd.runc.v2
...
... and many other pods...Well, you can continue using |
@quadeare |
|
Bumping this issue as I have the same problem with ubuntu 22.04 and flannel with rke1, i tried 'sudo update-alternatives --set iptables /usr/sbin/iptables-legacy' which used to be the fix for this but it no longer seems to work. |
|
This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions. |
Sorry for this very very late response... I just upgrade my home lab today and i test your workaround. I'm now able to run RKE1 on Ubuntu 22.04 without any issue. Have a nice day ! |
RKE version: 1.4.1 (same with 1.3.16)
Docker version: (
docker version,docker infopreferred)Operating system and kernel: (
cat /etc/os-release,uname -rpreferred)Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) : Baremetal and Scaleway VM (tested on both
cluster.yml file:
nodes: - address: **Baremetal host on 20.04** internal_address: 10.10.10.xx user: ** role: - controlplane - etcd - worker port: 22 - address: **VM host on 20.04** ==> OK, no issue internal_address: 10.10.10.xx user: ** role: - worker port: 22 - address: **VM host on 22.04** ==> **NOK**, issue with CNI internal_address: 10.10.10.xx user: ** role: - worker port: 22 - address: **Baremetal on 22.04** ==> **NOK**, issue with CNI internal_address: 10.10.10.xx user: ** role: - worker port: 22 services: etcd: backup_config: interval_hours: 24 retention: 12 s3backupconfig: access_key: xxxx secret_key: xxx bucket_name: xxx region: "fr-par" folder: "etcd" endpoint: xxxx kubelet: extra_args: container-runtime: remote container-runtime-endpoint: 'unix:///run/containerd/containerd.sock' max-pods: 250 extra_binds: - '/var/gitpod:/var/gitpod' - '/var/lib/containerd:/var/lib/containerd' # Disable docker-cri enable_cri_dockerd: false # If set to true, RKE will not fail when unsupported Docker version # are found ignore_docker_version: false # Enable use of SSH agent to use SSH private keys with passphrase # This requires the environment `SSH_AUTH_SOCK` configured pointing #to your SSH agent which has the private key added ssh_agent_auth: true # Set the name of the Kubernetes cluster cluster_name: xx-cluster # Currently, only authentication strategy supported is x509. # You can optionally create additional SANs (hostnames or IPs) to # add to the API server PKI certificate. # This is useful if you want to use a load balancer for the # control plane servers. authentication: strategy: x509 sans: - "rancher.xx.xx" - "rancher.xx.yy" kubernetes_version: v1.24.6-rancher1-1 network: plugin: flannel options: flannel_iface: ztrta4f6bpSteps to Reproduce: Install RKE suite with the CNI you want
Results:
CNI Crashloopback ONLY on 22.04 nodes without any error logs.
Kube-system pods list :
CNI pod describe
k describe po kube-flannel-k8w95 Name: kube-flannel-k8w95 Namespace: kube-system Priority: 2000001000 Priority Class Name: system-node-critical Service Account: flannel Node: xx.xx.xx.xx/10.10.10.221 Start Time: Thu, 01 Dec 2022 14:22:46 +0100 Labels: controller-revision-hash=85bb99c5bc k8s-app=flannel pod-template-generation=1 tier=node Annotations: <none> Status: Running IP: 10.10.10.221 IPs: IP: 10.10.10.221 Controlled By: DaemonSet/kube-flannel Containers: install-cni: Container ID: containerd://c30f417aa2ad42df1ecfa132e5b3a0287c6cee18b5c7654ebe2765c8186d5694 Image: rancher/flannel-cni:v0.3.0-rancher6 Image ID: docker.io/rancher/flannel-cni@sha256:23d921611903f6332cef666033924e0d92370548637d497549fe1121eb370feb Port: <none> Host Port: <none> Command: /install-cni.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 137 Started: Thu, 01 Dec 2022 14:25:42 +0100 Finished: Thu, 01 Dec 2022 14:27:08 +0100 Ready: False Restart Count: 3 Environment: CNI_NETWORK_CONFIG: <set to the key 'cni-conf.json' of config map 'kube-flannel-cfg'> Optional: false CNI_CONF_NAME: 10-flannel.conflist Mounts: /host/etc/cni/net.d from cni (rw) /host/opt/cni/bin/ from host-cni-bin (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kkq92 (ro) kube-flannel: Container ID: containerd://012cd8316cfc1d487a41657d2df59f75075b1e70a5a4bad58b3a987d382e61f7 Image: rancher/mirrored-coreos-flannel:v0.15.1 Image ID: docker.io/rancher/mirrored-coreos-flannel@sha256:162f82315dbe939e457697281a8ef04d469edd52bb384a1405f78468ed6fe323 Port: <none> Host Port: <none> Command: /opt/bin/flanneld Args: --ip-masq --kube-subnet-mgr --iface=ztrta4f6bp State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 01 Dec 2022 14:25:16 +0100 Finished: Thu, 01 Dec 2022 14:25:28 +0100 Ready: False Restart Count: 3 Limits: cpu: 100m memory: 50Mi Requests: cpu: 100m memory: 50Mi Environment: POD_NAME: kube-flannel-k8w95 (v1:metadata.name) POD_NAMESPACE: kube-system (v1:metadata.namespace) Mounts: /etc/cni/net.d from cni (rw) /etc/kube-flannel/ from flannel-cfg (rw) /run from run (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kkq92 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: run: Type: HostPath (bare host directory volume) Path: /run HostPathType: cni: Type: HostPath (bare host directory volume) Path: /etc/cni/net.d HostPathType: flannel-cfg: Type: ConfigMap (a volume populated by a ConfigMap) Name: kube-flannel-cfg Optional: false host-cni-bin: Type: HostPath (bare host directory volume) Path: /opt/cni/bin HostPathType: kube-api-access-kkq92: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: :NoSchedule op=Exists :NoExecute op=Exists node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m43s default-scheduler Successfully assigned kube-system/kube-flannel-k8w95 to xx.xx.xx.xx Normal Pulling 4m40s kubelet Pulling image "rancher/flannel-cni:v0.3.0-rancher6" Normal Pulled 4m25s kubelet Successfully pulled image "rancher/flannel-cni:v0.3.0-rancher6" in 14.576588607s Normal Pulling 4m24s kubelet Pulling image "rancher/mirrored-coreos-flannel:v0.15.1" Normal Pulled 4m17s kubelet Successfully pulled image "rancher/mirrored-coreos-flannel:v0.15.1" in 7.540987373s Normal Created 3m45s (x2 over 4m25s) kubelet Created container install-cni Normal Started 3m45s (x2 over 4m25s) kubelet Started container install-cni Normal Pulled 3m45s kubelet Container image "rancher/flannel-cni:v0.3.0-rancher6" already present on machine Normal Killing 3m43s (x2 over 4m16s) kubelet Stopping container install-cni Normal SandboxChanged 3m8s (x2 over 3m45s) kubelet Pod sandbox changed, it will be killed and re-created. Normal Created 3m5s (x3 over 4m17s) kubelet Created container kube-flannel Normal Started 3m5s (x3 over 4m17s) kubelet Started container kube-flannel Normal Killing 3m5s (x3 over 4m16s) kubelet Stopping container kube-flannel Normal Pulled 3m5s (x2 over 3m45s) kubelet Container image "rancher/mirrored-coreos-flannel:v0.15.1" already present on machine Warning BackOff 3m5s kubelet Back-off restarting failed containerFlannel logs :
The text was updated successfully, but these errors were encountered: