Random K3s node gets marked as "NotReady" at random intervals for a random amount of time.

[CephalonZeash]

Environmental Info:
K3s Version:

k3s version v1.24.4+k3s1 (c3f830e)
go version go1.18.1

Node(s) CPU architecture, OS, and Version:
Debian 11 Proxmox VM created using the Generic cloud image with 2GB RAM and 2 CPU cores, running on an x86 CPU.

Linux kcp-1 5.10.0-18-cloud-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

Cluster Configuration:

3 servers, 3 agents, made with these commands:

• First server:

curl -sfL https://get.k3s.io | \
    K3S_TOKEN=$TOKEN \
    sh -s - server \
    --node-taint CriticalAddonsOnly=true:NoExecute \
    --tls-san $IP --tls-san $ENDPOINT \
    --kubelet-arg "node-status-update-frequency=4s" \
    --kube-controller-manager-arg "node-monitor-period=4s" \
    --kube-controller-manager-arg "node-monitor-grace-period=4s" \
    --kube-controller-manager-arg "pod-eviction-timeout=10s" \
    --cluster-init

• Other servers:

curl -sfL https://get.k3s.io | \
      K3S_TOKEN=$TOKEN \
      sh -s - server \
      --node-taint CriticalAddonsOnly=true:NoExecute \
      --tls-san $IP --tls-san $ENDPOINT \
      --server https://$ENDPOINT:6443

• Agents:

sudo curl -sfL https://get.k3s.io | \
      K3S_URL=https://$ENDPOINT:6443 \
      K3S_TOKEN=$TOKEN \
      sh -

Where $ENDPOINT is the address of my HAProxy instance, which sits outside the cluster to load balance all the nodes, and is automatically updated once a new node is added/removed.

Describe the bug:

Upon creation, random nodes get marked as "NotReady" at random intervals for a random amount of time (but never too long, mostly between 1-7 seconds). This persists until all nodes are shut down and started up again, which I figured out by rebooting the Proxmox node.

Steps To Reproduce:

• Installed K3s:
• Run kubectl get nodes a few times

Expected behavior:

Nodes are marked as "Ready" since nothing external is running on them and the VMs are all healthy.

Actual behavior:

Nodes are marked as "NotReady" at random intervals for a random amount of time, despite nothing external running on them and the VMs being all healthy.

Additional context / logs:

Output of journalctl -u k3s on kcp-1 (the second k3s server) is in file.

issue.txt

kubectl get node kcp-1 -o yaml says that "Kubelet stopped posting node status."

Any help is much appreciated.

[brandond]

   --kubelet-arg "node-status-update-frequency=4s" \
   --kube-controller-manager-arg "node-monitor-period=4s" \
   --kube-controller-manager-arg "node-monitor-grace-period=4s" \
   --kube-controller-manager-arg "pod-eviction-timeout=10s" \

You have reduced the values of these settings way below their defaults. I'm guessing you've been reading the conversation in #1264 (comment)? I don't believe that Kubernetes itself is tested with the intervals and timeouts turned down that low. You're probably going to need to tune those to something that is stable for your environment.

Also, if you are customizing the controller-manager args on one server, you should customize them on all of them. Same with kubelet args - if you want the correct values on all the nods in your cluster, you must do so manually. Customized component args are not automatically shared across cluster nodes.

[CephalonZeash]

Thanks, that fixed this issue, but now nodes never get marked as Unknown after being shutdown? I changed those parameters to this:

--kubelet-arg "node-status-update-frequency=4s"
--kube-controller-manager-arg "node-monitor-period=4s"
--kube-controller-manager-arg "node-monitor-grace-period=16s"
--kube-controller-manager-arg "pod-eviction-timeout=10s"

The shutdown is from within Proxmox, so I'm guessing it's not really graceful on the K8s side of things. Before they would get marked Unknown after a bit of time, but now they're just stuck on NotReady. Pods thus don't get evicted and are just stuck on a node that doesn't exist.

Example, kwn-0 (one of the agents) is shutdown, here is the output of kubectl get node kwn-0 -o yaml:

apiVersion: v1
kind: Node
metadata:
  annotations:
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"8a:27:52:6b:e2:85"}'
    flannel.alpha.coreos.com/backend-type: vxlan
    flannel.alpha.coreos.com/kube-subnet-manager: "true"
    flannel.alpha.coreos.com/public-ip: 192.168.100.230
    k3s.io/hostname: kwn-0
    k3s.io/internal-ip: 192.168.100.230
    k3s.io/node-args: '["agent"]'
    k3s.io/node-config-hash: T4PRBIRVSOQRA3QDILHPHLCHWYOOBHYREUR5NLCYGP3FJKOWH3SQ====
    k3s.io/node-env: '{"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/ec00304416df58a8da2a883b1b87ab882b199ef11c4e01b28f07d643c8067d91","K3S_TOKEN":"********","K3S_URL":"https://192.168.100.210:6443"}'
    node.alpha.kubernetes.io/ttl: "0"
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2022-09-30T21:56:30Z"
  finalizers:
  - wrangler.cattle.io/node
  - wrangler.cattle.io/managed-etcd-controller
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: k3s
    beta.kubernetes.io/os: linux
    egress.k3s.io/cluster: "true"
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: kwn-0
    kubernetes.io/os: linux
    node.kubernetes.io/instance-type: k3s
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"wrangler.cattle.io/managed-etcd-controller": {}
          v:"wrangler.cattle.io/node": {}
    manager: deploy@kcp-1
    operation: Update
    time: "2022-09-30T21:56:30Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:k3s.io/hostname: {}
          f:k3s.io/internal-ip: {}
          f:k3s.io/node-args: {}
          f:k3s.io/node-config-hash: {}
          f:k3s.io/node-env: {}
          f:node.alpha.kubernetes.io/ttl: {}
          f:volumes.kubernetes.io/controller-managed-attach-detach: {}
        f:labels:
          .: {}
          f:beta.kubernetes.io/arch: {}
          f:beta.kubernetes.io/instance-type: {}
          f:beta.kubernetes.io/os: {}
          f:egress.k3s.io/cluster: {}
          f:kubernetes.io/arch: {}
          f:kubernetes.io/hostname: {}
          f:kubernetes.io/os: {}
          f:node.kubernetes.io/instance-type: {}
      f:spec:
        f:podCIDR: {}
        f:podCIDRs:
          .: {}
          v:"10.42.4.0/24": {}
        f:providerID: {}
        f:taints: {}
    manager: k3s
    operation: Update
    time: "2022-09-30T22:12:18Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:flannel.alpha.coreos.com/backend-data: {}
          f:flannel.alpha.coreos.com/backend-type: {}
          f:flannel.alpha.coreos.com/kube-subnet-manager: {}
          f:flannel.alpha.coreos.com/public-ip: {}
      f:status:
        f:conditions:
          k:{"type":"DiskPressure"}:
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
          k:{"type":"MemoryPressure"}:
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
          k:{"type":"PIDPressure"}:
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
          k:{"type":"Ready"}:
            f:lastHeartbeatTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
        f:images: {}
        f:nodeInfo:
          f:bootID: {}
    manager: k3s
    operation: Update
    subresource: status
    time: "2022-09-30T22:12:18Z"
  name: kwn-0
  resourceVersion: "4503"
  uid: bb12fef3-8812-4cd6-ab23-439ecac589df
spec:
  podCIDR: 10.42.4.0/24
  podCIDRs:
  - 10.42.4.0/24
  providerID: k3s://kwn-0
  taints:
  - effect: NoSchedule
    key: node.kubernetes.io/unreachable
    timeAdded: "2022-09-30T22:12:18Z"
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    timeAdded: "2022-09-30T22:12:22Z"
status:
  addresses:
  - address: 192.168.100.230
    type: InternalIP
  - address: kwn-0
    type: Hostname
  allocatable:
    cpu: "2"
    ephemeral-storage: "15897147380"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 2032932Ki
    pods: "110"
  capacity:
    cpu: "2"
    ephemeral-storage: 16341640Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 2032932Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2022-09-30T22:11:39Z"
    lastTransitionTime: "2022-09-30T22:12:18Z"
    message: Kubelet stopped posting node status.
    reason: NodeStatusUnknown
    status: Unknown
    type: MemoryPressure
  - lastHeartbeatTime: "2022-09-30T22:11:39Z"
    lastTransitionTime: "2022-09-30T22:12:18Z"
    message: Kubelet stopped posting node status.
    reason: NodeStatusUnknown
    status: Unknown
    type: DiskPressure
  - lastHeartbeatTime: "2022-09-30T22:11:39Z"
    lastTransitionTime: "2022-09-30T22:12:18Z"
    message: Kubelet stopped posting node status.
    reason: NodeStatusUnknown
    status: Unknown
    type: PIDPressure
  - lastHeartbeatTime: "2022-09-30T22:11:39Z"
    lastTransitionTime: "2022-09-30T22:12:18Z"
    message: Kubelet stopped posting node status.
    reason: NodeStatusUnknown
    status: Unknown
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - docker.io/library/nginx@sha256:0b970013351304af46f322da1263516b188318682b2ab1091862497591189ff1
    - docker.io/library/nginx:latest
    sizeBytes: 56768208
  - names:
    - docker.io/rancher/klipper-lb@sha256:02f8cb41d53fe08b5726a563ce36c3675ad7f2694b65a8477f6a66afac89fef7
    - docker.io/rancher/klipper-lb:v0.3.5
    sizeBytes: 3334605
  - names:
    - docker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893
    - docker.io/rancher/mirrored-pause:3.6
    sizeBytes: 301463
  nodeInfo:
    architecture: amd64
    bootID: 1fc284be-c40f-4571-9a29-6fc9e89b9ca4
    containerRuntimeVersion: containerd://1.6.8-k3s1
    kernelVersion: 5.10.0-18-cloud-amd64
    kubeProxyVersion: v1.24.6+k3s1
    kubeletVersion: v1.24.6+k3s1
    machineID: 8c6006364dcf4d6bacf97c2ae7db8a70
    operatingSystem: linux
    osImage: Debian GNU/Linux 11 (bullseye)
    systemUUID: 8c600636-4dcf-4d6b-acf9-7c2ae7db8a70

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.