Cannot get k3s to start on a fresh debian 13.4

[flixman]

Hi everybody!

I am installing k3s on a debian stable (13.4) by running on a single node:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -

the service comes up as expected and I check the pods:

# kubectl get pods -n kube-system
NAME                                      READY   STATUS             RESTARTS         AGE
coredns-c4dbffb5f-xzllw                   0/1     Running            3 (69s ago)      5m31s
helm-install-traefik-bkw8p                0/1     CrashLoopBackOff   18 (2m19s ago)   31m
helm-install-traefik-crd-fltlv            0/1     CrashLoopBackOff   17 (2m29s ago)   31m
local-path-provisioner-5c4dc5d66d-hm29v   1/1     Running            19 (2m47s ago)   31m
metrics-server-786d997795-vpj5f           0/1     CrashLoopBackOff   18 (2m42s ago)   31m

then I get describe the coredns pod, and I get the following:

# kubectl describe -n kube-system pod coredns-c4dbffb5f-xzll
Name:                 coredns-c4dbffb5f-xzllw
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      coredns
Node:                 k3s/192.168.123.4
Start Time:           Sat, 02 May 2026 11:41:27 +0200
Labels:               k8s-app=kube-dns
                      pod-template-hash=c4dbffb5f
Annotations:          <none>
Status:               Running
IP:                   10.42.0.15
IPs:
  IP:           10.42.0.15
Controlled By:  ReplicaSet/coredns-c4dbffb5f
Containers:
  coredns:
    Container ID:  containerd://54d216ba665bce79145d8e597cf97f4cba8835daf95b8685d31c369fe71f4d67
    Image:         rancher/mirrored-coredns-coredns:1.14.2
    Image ID:      docker.io/rancher/mirrored-coredns-coredns@sha256:e7e6440cfd1e919280958f5b5a6ab2b184d385bba774c12ad2a9e1e4183f90d9
    Ports:         53/UDP (dns), 53/TCP (dns-tcp), 9153/TCP (metrics)
    Host Ports:    0/UDP (dns), 0/TCP (dns-tcp), 0/TCP (metrics)
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Sat, 02 May 2026 11:47:19 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 02 May 2026 11:45:49 +0200
      Finished:     Sat, 02 May 2026 11:47:19 +0200
    Ready:          False
    Restart Count:  4
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /etc/coredns/custom from custom-config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mpl7r (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  custom-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns-custom
    Optional:  true
  kube-api-access-mpl7r:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    Optional:                 false
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               kubernetes.io/os=linux
Tolerations:                  CriticalAddonsOnly op=Exists
                              node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector k8s-app=kube-dns
                              topology.kubernetes.io/zone:ScheduleAnyway when max skew 1 is exceeded for selector k8s-app=kube-dns
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Normal   Scheduled         6m29s                  default-scheduler  Successfully assigned kube-system/coredns-c4dbffb5f-xzllw to k3s
  Warning  DNSConfigForming  6m27s (x4 over 6m29s)  kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.255.255.1 192.168.123.1 192.168.178.1
  Warning  Unhealthy         6m25s                  kubelet            spec.containers{coredns}: Readiness probe failed: Get "http://10.42.0.15:8181/ready": dial tcp 10.42.0.15:8181: connect: no route to host
  Warning  Unhealthy         88s (x149 over 6m27s)  kubelet            spec.containers{coredns}: Readiness probe failed: Get "http://10.42.0.15:8181/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Pulled            38s (x5 over 6m29s)    kubelet            spec.containers{coredns}: Container image "rancher/mirrored-coredns-coredns:1.14.2" already present on machine and can be accessed by the pod
  Normal   Created           38s (x5 over 6m29s)    kubelet            spec.containers{coredns}: Container created
  Normal   Started           38s (x5 over 6m29s)    kubelet            spec.containers{coredns}: Container started
  Normal   Killing           38s (x4 over 5m8s)     kubelet            spec.containers{coredns}: Container coredns failed liveness probe, will be restarted

After that, I check that both from inside or outside of the cluster, I cannot reach that pod:

# kubectl run -n kube-system --rm -it netcheck --image=curlimages/curl --restart=Never -- sh -c "/usr/bin/curl -v
 http://10.42.0.7:8181/ready"
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
* connect to 10.42.0.7 port 8181 from 10.42.0.16 port 42314 failed: Host is unreachable
* Failed to connect to 10.42.0.7 port 8181 after 3064 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to 10.42.0.7 port 8181 after 3064 ms: Could not connect to server
pod "netcheck" deleted from kube-system namespace
pod kube-system/netcheck terminated (Error)

# curl -v http://10.42.0.7:8181/ready
*   Trying 10.42.0.7:8181...
* connect to 10.42.0.7 port 8181 from 10.42.0.1 port 33090 failed: No route to host
* Failed to connect to 10.42.0.7 port 8181 after 3078 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to 10.42.0.7 port 8181 after 3078 ms: Could not connect to server

To finally observe that cni0 is down:

#ip add
....
6: cni0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 4a:7f:fc:a5:f4:d1 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::487f:fcff:fea5:f4d1/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

Related to cni0, this is what I get after a fresh reboot:

# journalctl -S "11:55" | grep cni0
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Link UP
May 02 11:55:36 k3s kernel: cni0: port 1(veth96039f78) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 1(veth96039f78) entered disabled state
May 02 11:55:36 k3s kernel: cni0: port 1(veth96039f78) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 1(veth96039f78) entered forwarding state
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Gained carrier
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Configuring with /etc/systemd/network/99-dhcp-dynamic.network.
May 02 11:55:36 k3s kernel: cni0: port 1(veth96039f78) entered disabled state
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Lost carrier
May 02 11:55:36 k3s kernel: cni0: port 1(veth1027afa2) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 1(veth1027afa2) entered disabled state
May 02 11:55:36 k3s kernel: cni0: port 1(veth1027afa2) entered disabled state
May 02 11:55:36 k3s kernel: cni0: port 1(veth28e3fbe3) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 1(veth28e3fbe3) entered disabled state
May 02 11:55:36 k3s kernel: cni0: port 1(veth28e3fbe3) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 1(veth28e3fbe3) entered forwarding state
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Gained carrier
May 02 11:55:36 k3s kernel: cni0: port 1(veth28e3fbe3) entered disabled state
May 02 11:55:36 k3s kernel: cni0: port 1(vethffd320c2) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 1(vethffd320c2) entered disabled state
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Lost carrier
May 02 11:55:36 k3s kernel: cni0: port 1(vethffd320c2) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 1(vethffd320c2) entered forwarding state
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Gained carrier
May 02 11:55:36 k3s kernel: cni0: port 2(veth81a9bc2b) entered blocking state
May 02 11:55:36 k3s kernel: cni0: port 2(veth81a9bc2b) entered disabled state
May 02 11:55:36 k3s kernel: cni0: port 1(vethffd320c2) entered disabled state
May 02 11:55:36 k3s kernel: cni0: port 2(veth81a9bc2b) entered disabled state
May 02 11:55:36 k3s systemd-networkd[534]: cni0: Lost carrier

I am completely lost as per what can possibly be the reason for this not starting properly. Might anybody here have a hint on what to look for?

[flixman]

the problem was with systemd-networkd rule that had a filter too broad, and was interfering with k3s' management of the interfaces.