-
-
Notifications
You must be signed in to change notification settings - Fork 462
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] failing on linux kernel >= 5.12.2 (and >= 5.11.19, ...) #607
Comments
Same error on kernel 5.12.4 Quick workaround : kubernetes-sigs/kind#2240 (comment) |
I'm currently setting up a test system, but I guess that given the workaround mentioned in the |
Just finished my test setup:
This works just fine 👍 @nemonik can you try this please? |
Hi, I just faced this issue and the solution proposed by @iwilltry42 is working perfectly!
|
@iwilltry42 thanks for the workaround, it works well on Arch.
|
@iwilltry42 it works,
but if we use agents server components still restarting frequently.
k3d output
docker ps
docker's error log
OS: Arch Linux |
Closing this, as it will be fixed upstream (in k3s), thanks to @brandond 🙏 |
I've added a new FAQ entry on this issue: https://k3d.io/faq/faq/#nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied Also, thanks to #612 we quickly checked, that (obviously) also other kernel lines are affected, like 5.11 as of 5.11.19. |
So, I finally got to the point where I could try to press forward with this... Creating a cluster straight away with k3d on Arch running 5.12.10-arch1-1 still doesn't work using k3d installed out of AUR.
So, I did as you asked modifying for the latest k3s docker image or by just stripping out the image param like so
or
And I got it working as you advised... @iwilltry42 Thank you for the fix. Sorry it took so long to try it. But is this the advised way now (as per the faq) or has this been replaced over the past 20 some days? |
Thanks for the reply @nemonik 👍
k3d v4.4.4 uses k3s v1.20.6 by default (that's hardcoded at build time), so that won't work.
When setting the Also, v4.4.5 is just being released (there were issues with the release system delaying it), which will be using one of the newer versions by default (so no image or kube-proxy flags required). |
Do you still need the I will watch for k3d v4.4.5 to drop in AUR Thanks for the reply. |
Not with the new versions of k3s (v1.21.1-k3s1 is the new default in k3d v4.4.5 and includes the fix). |
Danke schön @iwilltry42 |
k3d v4.4.5 hit AUR last night... |
Note that I experienced the same issue with lower versions of the Linux Kernel, : My Stack versions$ ./versions.sh
Client: Docker Engine - Community
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:54:22 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:31 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0
k3d version v4.4.1
k3s version v1.20.5-k3s1 (default)
go version go1.17 linux/amd64
hugo v0.83.1+extended linux/amd64 BuildDate=unknown
flux version 0.17.0
The error message that I get for all k3s server nodesk3s server open /proc/sys/net/netfilter/nf_conntrack_max: permission denied And I found :
jbl@fluxcd:~$ uname -r
5.10.0-8-amd64
jbl@fluxcd:~$ cat /proc/version
Linux version 5.10.0-8-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.46-4 (2021-08-03) |
Yes, some Linux distributions backported the fix into earlier kernel versions. It was only present in mainline linux 5.12.2 and 5.11.19. You'd have to check with the Debian folks to see which releases they backported it to. Either way, there's a fix available. Just use newer K3s. |
@brandond Oh, thanks very much for the information :) |
I confirm that upgrading to |
What did you do
How was the cluster created?
k3d cluster create "vald-cluster" -p "8081:80@loadbalancer" --agents 5
What did you do afterwards?
What did you expect to happen
I expect that k3d works with the latest kernels, just as it worked with Linux Kernel 5.11.
When multiple Agents were specified, k3d did not proceed to start the Agents when the cluster was created, and the Docker logs showed that the K3D Server was restarting at a high frequency.
In the container log, it looks like kube-proxy is failing to start due to failure to set nf_conntrack_max.
I looked into this a bit and found a similar problem in kind and minikube, and it seems to be fixed in the following Issue and PR.
for kind
kubernetes-sigs/kind#2240
kubernetes-sigs/kind#2241
for minikube
kubernetes/minikube#11419
Screenshots or terminal output
k3d command cli log
docker ps
docker logs --defail -t f9e815595dcf
Which OS & Architecture
Which version of
k3d
k3d version
Which version of docker
docker version
docker info
The text was updated successfully, but these errors were encountered: