[BUG] Cluster fails to start on cgroup v2 #493
Assignees
Labels
help wanted
Extra attention is needed
k3s
This is likely an issue with k3s not k3d itself
priority/high
runtime
Issue with the container runtime (docker)
Milestone
Comments
|
I think I'm seeing the same or similar issue. When I rollback to |
|
Same issue on Fedora 33: Logs: |
|
|
|
Seems like started but keeps restarting for some reason. |
iwilltry42
added
k3s
|
Hi @derricms, thanks for opening this issue and Hi to the others who joined in 👋
@mj41-gdc you also saw an issue I guess with a filesystem like zfs or btrfs, right (/dev/mapper)? |
Yes, btfrs is Fedora 33 default filesystem. My disc is also encrypted (LUKS). Also selinux is enabled. I tried |
|
@mj41-gdc wait...
So k3s v1.20.4 works without issues but in k3d it throws the cgroup v2 incompatibility error? 🤔 Actually, on Fedora, you may as well have issues with |
|
@iwilltry42 My cluster started but container was restarting every few seconds. And I was not able to debug what is the root cause. I'm a newbie here. |
|
That's weird... @mj41-gdc , can you try |
mj41-gdc
added a commit
to mj41-gdc/k3d-debug
that referenced
this issue
Mar 9, 2021
mj41-gdc
added a commit
to mj41-gdc/k3d-debug
that referenced
this issue
Mar 9, 2021
|
Hi @iwilltry42, I did this (in short) Full detail what I did: # setup
mkdir -p ~/devel/k3d
cd ~/devel/k3d
pwd
ls -al
# console 1
# check logs and send them to github if all seems ok
ls -als log*.txt
echo '```' ; head -n 10 log*.txt ; echo '```'
# cleanup previous run
k3d cluster delete default
rm ~/devel/k3d/log-*.txt
ls -al
cat README.md
# console 2
echo "#Start: `date --rfc-3339=ns`" > log-docker-events.txt ; docker events | tee -a log-docker-events.txt
# console 3
echo "#Start: `date --rfc-3339=ns`" > log-start-k3d.txt ; k3d cluster create default --image rancher/k3s:v1.20.4-k3s1 2>&1 | tee -a log-start-k3d.txt ; echo "#End: `date --rfc-3339=ns`" >> log-start-k3d.txt
# console 4
# a few times run these till you see the first restart
echo "#Start_ps: `date --rfc-3339=ns`" | tee -a log-docker-ps.txt ; docker ps | tee -a log-docker-ps.txt
# console 5
echo "#Start_logs: `date --rfc-3339=ns`" | tee -a log-docker-logs.txt ; docker logs --timestamps --details k3d-default-server-0 2>&1 | tee -a log-docker-logs.txt
# console 2
# Press ctrl+c
# check logs, repeat if neededFull logs are a few MBs so I put them here A few lines from each here: Let me know what I should try next. And thank you very much for your time. |
|
Thanks for the input @mj41-gdc ! So the issue is with your filesystem. |
mj41-gdc
added a commit
to mj41-gdc/k3d-debug
that referenced
this issue
Mar 9, 2021
|
One step a time :-). Now I got Full logs here |
mj41-gdc
added a commit
to mj41-gdc/k3d-debug
that referenced
this issue
Mar 9, 2021
|
That's a warning that you can ignore for now 👍 |
|
I tried to switch docker to cgroupfs per kubernetes/kubeadm#1394 (comment) But still the same Per https://unix.stackexchange.com/questions/480747/how-to-find-out-if-systemd-uses-legacy-hybrid-or-unified-mode-cgroupsv1-vs-cgr and great |
mj41-gdc
added a commit
to mj41-gdc/k3d-debug
that referenced
this issue
Mar 9, 2021
|
@iwilltry42 I get some time and motivation yesterday to find out that the error comes from https://github.com/kubernetes/kubernetes/blob/v1.20.4/pkg/kubelet/kubelet.go#L1366-L1369 ( Changes in master:
|
|
I'm really lost here and have no idea at the moment what could be the issue 🤔 |
|
Hi, Regards |
|
Hi @fr33ky , thanks for your input. |
|
Looks like @AkihiroSuda did a good job fixing issues with cgroup v2 in kind (see kubernetes-sigs/kind#2014). This could be a good starting point for us as well, even though our issue seems to be slightly different. |
iwilltry42
changed the title
|
Hello, I have the same issue on Arch Linux. I also have cgroup v2 |
|
I get exactly same errors with cgroup v2. Any hint to fix it? |
Using Debian Sid, in the meantime, I personally switched back to |
Works for me on Arch by executing |
|
For anyone running into this on NixOS, setting |
Thanks. Switching back to cgroup v1 works. |
|
For ArchLinux users that now run systemd v248+ and use systemd-boot here's how I fixed it for my system: ...
-options root=/dev/mapper/root
+options root=/dev/mapper/root systemd.unified_cgroup_hierarchy=0Then verified with |
|
On Arch, the latest rancher-k3d-bin (v4.2.0) would just loop trying to start the servers... I followed what @wdv4758h suggested above. By executing This reverted me back to cgroup 1 verified by |
|
Issue still persist on k3d v4.4.2 with k3s v1.20.6-k3s1 Would be good if docs listed that |
|
I have the same issue on NixOS (unstable channel). Also using cgroups v2. Just figuring out how to switch it to v1 with NixOS and I'll report back if it works. |
For cgroup v2, k3s/k3d needs to have a logic to evacuate the init process from the top-level cgroup to somewhere else, like this: https://github.com/moby/moby/blob/e0170da0dc6e660594f98bc66e7a98ce9c2abb46/hack/dind#L28-L37 |
|
Thanks for the hint @AkihiroSuda , I actually found a way to re-use your linked moby source and the changes you did in |
|
@iwilltry42 Yes, thanks |
#493 (comment) is most likely what you're after |
|
You can give this a try now on cgroupv2: I tested it without issues on Ubuntu 20.10 with cgroupv1 and cgroupv2 (systemd). |
|
@iwilltry42 I confirm that image works correctly with EDIT: I also confirm it works correctly with |
|
I just created a (temporary) fix/workaround using the entrypoint script that we can use until it was fixed upstream (in k3s). See PR #579 . |
|
@iwilltry42 i'm able to confirm that this adds v2 support on my system. thank you! |
|
Fixed by #579 (should not interfere with k3s-io/k3s#3242 later) |
|
works on Fedora 33 with Docker and cgroupv2. Great work. Thank you @iwilltry42 , @AkihiroSuda and others. Detailed logs https://github.com/mj41-gdc/k3d-debug/tree/k3d-issues-493-mj7 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What did you do
Start a minimal cluster on Kali Linux 2020.4
* How was the cluster created?
*
k3d cluster createWhat did you do afterwards?
What did you expect to happen
That a minimal cluster would start
Screenshots or terminal output
{"log":"time=\"2021-02-10T15:54:15.154488575Z\" level=info msg=\"Containerd is now running\"\n","stream":"stderr","time":"2021-02-10T15:54:15.154604054Z"} {"log":"time=\"2021-02-10T15:54:15.276436029Z\" level=info msg=\"Connecting to proxy\" url=\"wss://127.0.0.1:6443/v1-k3s/connect\"\n","stream":"stderr","time":"2021-02-10T15:54:15.276584849Z"} {"log":"time=\"2021-02-10T15:54:15.344809810Z\" level=info msg=\"Handling backend connection request [k3d-minimal-default-server-0]\"\n","stream":"stderr","time":"2021-02-10T15:54:15.344941507Z"} {"log":"time=\"2021-02-10T15:54:15.383483103Z\" level=warning msg=\"**Disabling CPU quotas due to missing cpu.cfs_period_us**\"\n","stream":"stderr","time":"2021-02-10T15:54:15.383600244Z"} {"log":"time=\"2021-02-10T15:54:15.383649950Z\" level=warning msg=\"**Disabling pod PIDs limit feature due to missing cgroup pids support**\"\n","stream":"stderr","time":"2021-02-10T15:54:15.383683752Z"} {"log":"time=\"2021-02-10T15:54:15.383773636Z\" level=info msg=\"Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --cgroups-per-qos=false --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=unix:///run/k3s/containerd/containerd.sock --cpu-cfs-quota=false --enforce-node-allocatable= --eviction-hard=imagefs.available\u003c5%,nodefs.available\u003c5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --feature-gates=SupportPodPidsLimit=false --healthz-bind-address=127.0.0.1 --hostname-override=k3d-minimal-default-server-0 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/tmp/k3s-resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key\"\n","stream":"stderr","time":"2021-02-10T15:54:15.383842163Z"} {"log":"time=\"2021-02-10T15:54:15.384645964Z\" level=info msg=\"Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=k3d-minimal-default-server-0 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables\"\n","stream":"stderr","time":"2021-02-10T15:54:15.38471723Z"} {"log":"Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.\n","stream":"stderr","time":"2021-02-10T15:54:15.387483943Z"} {"log":"Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.\n","stream":"stderr","time":"2021-02-10T15:54:15.387594058Z"} {"log":"F0210 15:54:15.387923 7 server.go:181] cannot set feature gate SupportPodPidsLimit to false, feature is locked to true\n","stream":"stderr","time":"2021-02-10T15:54:15.387966646Z"} {"log":"goroutine 3978 [running]:\n","stream":"stderr","time":"2021-02-10T15:54:15.549704084Z"}Which OS & Architecture
Which version of
k3dWhich version of docker
Server: Engine: Version: 20.10.2+dfsg1 API version: 1.41 (minimum version 1.12) Go version: go1.15.6 Git commit: 8891c58 Built: Fri Jan 8 07:08:51 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.3~ds1 GitCommit: 1.4.3~ds1-1+b1 runc: Version: 1.0.0-rc92+dfsgl GitCommit: 1.0.0-rc92+dfsgl-5+b1 docker-init: Version: 0.19.0 GitCommit:The text was updated successfully, but these errors were encountered: