Container networking is broken on hosts with default-deny iptables rules after upgrading to v1.26.3

[mgabeler-lee-6rs]

Environmental Info:
K3s Version:

k3s version v1.26.3+k3s1 (01ea3ff2)
go version go1.19.7

Node(s) CPU architecture, OS, and Version:
Linux CENSORED 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-1 (2023-03-19) x86_64 GNU/Linux

Cluster Configuration:

• Local development environment using k3s

Describe the bug:

After upgrading to k3s 1.26.x (version above) from 1.25.x, nothing would come up, even after wiping all k3s config data and starting a fresh cluster. After digging into logs, the issue traced backwards from "coredns not responding" to "coredns stuck waiting on kubernetes service" to the kubernetes service failing to be initialized on the first cluster start attempt, and never attempting repair thereafter:

E0403 15:02:19.425921 1727565 controller.go:156] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocate IP 10.43.0.1: cannot allocate resources of type serviceipallocations at this time

Steps To Reproduce:

• Installed K3s: Download binary from github and drop in /usr/local/bin
• Start k3s: sudo k3s server --write-kubeconfig-mode 644 --docker --kube-apiserver-arg=service-node-port-range=1024-32767 --tls-san=0.0.0.0

Expected behavior:
It should be able to start a cluster

Actual behavior:
It fails to start the cluster

Additional context / logs:

• Call-out that, due to developer environment, we are using --docker instead of containerd. Unless --docker is no longer supported, please don't just say "you should use containerd / nerdctl". We've evaluated that, and it is not an easy replacement for our workflows & machine setups right now.

[brandond]

E0403 15:02:19.425921 1727565 controller.go:156] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocate IP 10.43.0.1: cannot allocate resources of type serviceipallocations at this time

You will see this in absolutely every K3s server startup log ever. This always happens during initial cluster startup, and is resolved within miliseconds once the rest of the controllers are initialized.

--tls-san=0.0.0.0

This isn't a valid TLS SAN; you will never connect to a node using the IP 0.0.0.0.

Please attach the complete K3s service log, as well as example commands and output showing whatever errors you're encountering. The information you've provided here doesn't provide enough detail to actual discern what's going on with your environment,

[mgabeler-lee-6rs]

--tls-san=0.0.0.0

This isn't a valid TLS SAN; you will never connect to a node using the IP 0.0.0.0.

This is something I got copied from somewhere to allow the fake cert to be accepted for different IP addresses ... I forget where I got it from, and Google is failing me right now. Removing it doesn't help (I wiped the k8s environment to start fresh just to make sure).

You will see this in absolutely every K3s server startup log ever. This always happens during initial cluster startup, and is resolved within miliseconds once the rest of the controllers are initialized.

OK, I thought it wasn't resolved, but I was looking for the service in the kube-system namespace instead of the default namespace.

Please attach the complete K3s service log, as well as example commands and output showing whatever errors you're encountering. The information you've provided here doesn't provide enough detail to actual discern what's going on with your environment,

👍
k3s.log
coredns.log
local-path-provisioner.log

Everything else I see failing seems to boil down to coredns and/or local-path-provisioner failing to start, and those seem to be failing to start because they're timing out trying to contact the main kubernetes api endpoint. At least that is notably different in the logs for those services vs. the logs from a working machine running 1.25.x -- the working machine fails to contact the api endpoint at startup, but only temporarily before that stops and they work normally.

I can hit https://<my-local-ip>:6443 which seems to be where that service points at, so I'm not sure what's going on with the pods.

All my other pod failures at this point are DNS failures trying to talk to the coredns pod that isn't up

[brandond]

Are you running k3s as a user, instead of as a systemd service? There are some odd errors in the logs about cgroups and missing containers:

E0403 15:55:39.202799 1796046 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 66c56e2d3eed4b340690cfd59d48c7a5a61401a554aa92f9c0d95af7dee48357" containerID="66c56e2d3eed4b340690cfd59d48c7a5a61401a554aa92f9c0d95af7dee48357"
I0403 15:55:39.202806 1796046 volume_manager.go:293] "Starting Kubelet Volume Manager"
I0403 15:55:39.202814 1796046 kuberuntime_gc.go:362] "Error getting ContainerStatus for containerID" containerID="66c56e2d3eed4b340690cfd59d48c7a5a61401a554aa92f9c0d95af7dee48357" err="rpc error: code = Unknown desc = Error: No such container: 66c56e2d3eed4b340690cfd59d48c7a5a61401a554aa92f9c0d95af7dee48357"
I0403 15:55:39.202844 1796046 desired_state_of_world_populator.go:151] "Desired state populator starts to run"
E0403 15:55:39.203485 1796046 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: b1ebd4dd0190bd5a5bc3bf9c2a6e42724e2b17d19f109d2c0f5183609b3c068e" containerID="b1ebd4dd0190bd5a5bc3bf9c2a6e42724e2b17d19f109d2c0f5183609b3c068e"
I0403 15:55:39.203504 1796046 kuberuntime_gc.go:362] "Error getting ContainerStatus for containerID" containerID="b1ebd4dd0190bd5a5bc3bf9c2a6e42724e2b17d19f109d2c0f5183609b3c068e" err="rpc error: code = Unknown desc = Error: No such container: b1ebd4dd0190bd5a5bc3bf9c2a6e42724e2b17d19f109d2c0f5183609b3c068e"
E0403 15:55:39.204177 1796046 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 61acb17030e060ee20b9e794fa848dfa68ffe1e6f56eefce3b86cf796da4bbce" containerID="61acb17030e060ee20b9e794fa848dfa68ffe1e6f56eefce3b86cf796da4bbce"
I0403 15:55:39.204195 1796046 kuberuntime_gc.go:362] "Error getting ContainerStatus for containerID" containerID="61acb17030e060ee20b9e794fa848dfa68ffe1e6f56eefce3b86cf796da4bbce" err="rpc error: code = Unknown desc = Error: No such container: 61acb17030e060ee20b9e794fa848dfa68ffe1e6f56eefce3b86cf796da4bbce"
E0403 15:55:39.204395 1796046 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: b998fc83bb0c3d68ecf09aeead4adbacdf0af5bdebd8489fbab0ef2f30dd381f" containerID="b998fc83bb0c3d68ecf09aeead4adbacdf0af5bdebd8489fbab0ef2f30dd381f"
I0403 15:55:39.204406 1796046 kuberuntime_gc.go:362] "Error getting ContainerStatus for containerID" containerID="b998fc83bb0c3d68ecf09aeead4adbacdf0af5bdebd8489fbab0ef2f30dd381f" err="rpc error: code = Unknown desc = Error: No such container: b998fc83bb0c3d68ecf09aeead4adbacdf0af5bdebd8489fbab0ef2f30dd381f"
E0403 15:55:39.205004 1796046 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: 0a643f81c5a469086a00889c11d125529bb0dba91a6a6115d3c92e01b6c0eed7" containerID="0a643f81c5a469086a00889c11d125529bb0dba91a6a6115d3c92e01b6c0eed7"
I0403 15:55:39.205021 1796046 kuberuntime_gc.go:362] "Error getting ContainerStatus for containerID" containerID="0a643f81c5a469086a00889c11d125529bb0dba91a6a6115d3c92e01b6c0eed7" err="rpc error: code = Unknown desc = Error: No such container: 0a643f81c5a469086a00889c11d125529bb0dba91a6a6115d3c92e01b6c0eed7"

K3s as a whole seems to be running in your user slice instead of in a dedicated slice for a systemd service unit?

I0403 15:55:39.312502 1796046 container_manager_linux.go:626] "Failed to ensure state" containerName="/k3s" err="failed to find container of PID 1796046: cpu and memory cgroup hierarchy not unified.  cpu: /user.slice, memory: /user.slice/user-1000.slice/user@1000.service"

[mgabeler-lee-6rs]

Yes, running this "by hand", but still as root (sudo k3s ...). This used to work OK, and fits a little better in our developer environments as we can more directly manage it as part of the rest of the dev stack. Is that no longer a viable option as of 1.26?

[brandond]

I'm not sure that its strictly related, but it probably isn't great either. If you've got docker using the systemd cgroup manager, the kubelet will want to do the same, but it can't because it's all running under your user slice instead of in a dedicated system slice. Kubernetes in general is moving heavily towards using the systemd cgroup manager and cgroupv2, and just running k3s via sudo from a shell does not allow it to do that.

Have you confirmed that you don't have any iptables rules (managed via ufw/firewalld/etc) that might be interfering with things?

[mgabeler-lee-6rs]

We can't run cgroups v2 because our workload requires the ability to run Ubuntu 18.04 os-containers, and that doesn't have a new enough version of systemd to work that way.

I checked iptables rules, yes, and the only ones that stuck out as different between the working and non-working systems were some REJECTs in KUBE-SERVICES ... for the coredns and other services that weren't ready yet due to the kube api server issue, annotated as "has no endpoints", so that part made sense.

I'll try running k3s under a normal systemd unit and see if it makes things better.

[mgabeler-lee-6rs]

That didn't help, here's a fresh log from that:
k3s-systemd.log

[mgabeler-lee-6rs]

I also tried switching away from --docker just to try to isolate issues, it didn't help

[brandond]

the only ones that stuck out as different between the working and non-working systems were some REJECTs in KUBE-SERVICES ... for the coredns and other services that weren't ready yet due to the kube api server issue, annotated as "has no endpoints", so that part made sense.

Can you show the output of kubectl get service,endpoints,networkpolicy -A -o wide ?

If you're dealing with a single-node cluster, the only thing that might be interfering with access to the in-cluster kubernetes endpoint would be something else inserting conflicting iptables rules, or perhaps otherwise mucking with the container network interfaces.

[mgabeler-lee-6rs]

If you're dealing with a single-node cluster, the only thing that might be interfering with access to the in-cluster kubernetes endpoint would be something else inserting conflicting iptables rules, or perhaps otherwise mucking with the container network interfaces.

This was my thought too, but I was struggling with how to see the iptables rules that applied within the network namespaces. Esp. since the coredns image doesn't even have /bin/sh. I guess I'd need to create a pod running in privileged mode / cap_net_admin enabled for the iptables binary to be able to see things? would that be helpful?

Can you show the output of kubectl get service,endpoints,networkpolicy -A -o wide ?

E0403 17:45:08.289309 1924749 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0403 17:45:08.294308 1924749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0403 17:45:08.295147 1924749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0403 17:45:08.298071 1924749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0403 17:45:08.299441 1924749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0403 17:45:08.300804 1924749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0403 17:45:08.302003 1924749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0403 17:45:08.302932 1924749 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
NAMESPACE     NAME                     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
default       service/kubernetes       ClusterIP   10.43.0.1     <none>        443/TCP                  20s   <none>
kube-system   service/kube-dns         ClusterIP   10.43.0.10    <none>        53/UDP,53/TCP,9153/TCP   17s   k8s-app=kube-dns
kube-system   service/metrics-server   ClusterIP   10.43.91.45   <none>        443/TCP                  16s   k8s-app=metrics-server

NAMESPACE     NAME                       ENDPOINTS         AGE
default       endpoints/kubernetes       10.0.0.174:6443   20s
kube-system   endpoints/kube-dns         <none>            5s
kube-system   endpoints/metrics-server                     5s

[brandond]

the CNI and kubelet managed iptables rules are all in the host namespace, I wouldn't expect you to see any in the pods.

What version of Debian and Docker are you running? Have you customized the system configuration in any particularly interesting ways? What do you see if you run ip addr in a pod?

I regularly run k3s on Ubuntu 22.04 (its my primary development OS) both with containerd and with docker and have not seen any issues, nor have I seen anyone else report similar issues, so I suspect there's something else going on with your OS configuration.

[mgabeler-lee-6rs]

The host system is Debian bookworm (almost ready new stable release, but technically still "testing"). I'll try on an Ubuntu 22.04 VM to see if that makes a difference. Docker is 20.10.23+dfsg1 (from the debian docker.io package). Containerd is containerd github.com/containerd/containerd 1.6.18~ds1 1.6.18~ds1-1+b2 in case that's relevant.

I haven't customized this system much, no, installing k3s is the most "interesting" thing I've done to it regarding networking setups. I have these saved iptables rules for a basic "anything outgoing, nothing incoming except LAN subnets" config, and the KUBE rules all get added ahead of this in iptables:

*filter
:INPUT DROP
:FORWARD DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wg+ -j ACCEPT
-A INPUT -s 10.0.0.0/8 -i wl+ -j ACCEPT
-A INPUT -s 10.0.0.0/8 -i en+ -j ACCEPT
-A INPUT -i docker+ -j ACCEPT
-A INPUT -i br-+ -j ACCEPT
COMMIT

Indeed creating a privileged pod saw nothing for iptables. ip addr info from that pod:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0@if106: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ae:c0:bd:8e:dd:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.0.10/24 brd 10.42.0.255 scope global eth0
       valid_lft forever preferred_lft forever

[mgabeler-lee-6rs]

Tried a few more things to isolate the issue:

• Reboot the system, just coz: nope
• Downgrade to current 1.25.x release (1.25.8+k3s1): nope
• Dig through my logs and downgrade to the last 1.25.x version I actually used, which was 1.25.7+k3s1: !! works !!

So, there's a much smaller range to bisect here of where things might have gone wrong

Edit: continuing the bisect, 1.25.8-rc1+k3s1 is also showing the problem, so I guess further bisecting will require building from source, which I think I can do

[brandond]

1.25.7+k3s1: !! works !!
1.25.7-rc1+k3s1 is also showing the problem

There was no second RC of 1.25.7+k3s1, so these tags point to the same commit...

commit f7c20e237d0ad0eae83c1ce60d490da70dbddc0e (tag: v1.25.7-rc1+k3s1, tag: v1.25.7+k3s1)
Author: Matt Trachier <matt.trachier@suse.com>
Date:   Wed Mar 1 15:29:10 2023 -0600

    Update to v1.25.7-k3s1 (#7010)

    * Update to v1.25.7
    * update gh workflows and docker files to proper go version
    ---------
    Signed-off-by: matttrach <matttrach@gmail.com>

[mgabeler-lee-6rs]

building from source is erroring out with some checks in scripts/version.sh
Edit: figured out what was going wrong here and worked around it

There was no second RC of 1.25.7+k3s1, so these tags point to the same commit...

git log shows me:

commit 6c5ac02248834a4d59501f7f31404d1287e358db (tag: v1.25.8-rc2+k3s1, tag: v1.25.8+k3s1)
Author: Roberto Bonafiglia <roberto.bonafiglia@suse.com>
Date:   Wed Mar 22 15:50:03 2023 +0100

    Update flannel to fix NAT issue with old iptables version
    
    Signed-off-by: Roberto Bonafiglia <roberto.bonafiglia@suse.com>

Edit: sorry, had a typo, it was the 1.25.8-rc1 that failed for me, 1.25.7 is passing.