[Bug]: Stuck at "waiting for the condition on deployments/system-upgrade-controller" (cilium pod stuck)

[byRoadrunner]

Description

When creating a completely new cluster with version 2.11.3 the cluster installation keeps failing at:

module.kube-hetzner.null_resource.kustomization (remote-exec): + echo 'Waiting for the system-upgrade-controller deployment to become available...'
module.kube-hetzner.null_resource.kustomization (remote-exec): Waiting for the system-upgrade-controller deployment to become available...
module.kube-hetzner.null_resource.kustomization (remote-exec): + kubectl -n system-upgrade wait --for=condition=available --timeout=360s deployment/system-upgrade-controller
module.kube-hetzner.null_resource.kustomization: Still creating... [10s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [20s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [30s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [40s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [50s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [1m0s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [1m10s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [1m20s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [1m30s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [1m40s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [1m50s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [2m0s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [2m10s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [2m20s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [2m30s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [2m41s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [2m51s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [3m1s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [3m11s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [3m21s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [3m31s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [3m41s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [3m51s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [4m1s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [4m11s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [4m21s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [4m31s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [4m41s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [4m51s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [5m1s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [5m11s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [5m21s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [5m31s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [5m41s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [5m51s elapsed]
module.kube-hetzner.null_resource.kustomization: Still creating... [6m1s elapsed]
module.kube-hetzner.null_resource.kustomization (remote-exec): error: timed out waiting for the condition on deployments/system-upgrade-controller

I sshed to cp1 and checked the status of the pods in kube-system namespace

kubectl get pods -n kube-system
NAME                                              READY   STATUS              RESTARTS        AGE
cilium-25fr6                                      0/1     CrashLoopBackOff    6 (4m14s ago)   13m
cilium-2cfgs                                      0/1     CrashLoopBackOff    6 (2m25s ago)   13m
cilium-65rtq                                      0/1     Running             4 (2m53s ago)   13m
cilium-cnbfd                                      1/1     Running             0               13m
cilium-operator-f5dcdcc8d-976tq                   1/1     Running             0               13m
cilium-operator-f5dcdcc8d-x8xcn                   1/1     Running             0               13m
cilium-rfwpb                                      1/1     Running             0               13m
coredns-6799fbcd5-dlcjq                           1/1     Running             0               14m
csi-smb-controller-597b6984b9-7bhg2               3/3     Running             0               12m
csi-smb-node-95dfd                                3/3     Running             0               12m
csi-smb-node-hlbp2                                0/3     Pending             0               12m
csi-smb-node-hljv6                                0/3     Pending             0               12m
csi-smb-node-rwnps                                0/3     Pending             0               12m
csi-smb-node-s87c7                                3/3     Running             0               12m
hcloud-cloud-controller-manager-b9b8b75c7-ckklr   1/1     Running             0               13m
hcloud-csi-controller-7dd59b5d75-2tvth            5/5     Running             0               13m
hcloud-csi-node-88ddr                             0/3     ContainerCreating   0               13m
hcloud-csi-node-bz8p4                             3/3     Running             0               13m
hcloud-csi-node-tmkdq                             0/3     ContainerCreating   0               13m
hcloud-csi-node-wd2t8                             0/3     ContainerCreating   0               13m
hcloud-csi-node-xk2h4                             3/3     Running             0               13m
helm-install-cert-manager-vjkpm                   0/1     Completed           0               13m
helm-install-cilium-9gf4f                         0/1     Completed           0               13m
helm-install-csi-driver-smb-g76h7                 0/1     Completed           0               13m
helm-install-traefik-5lm7x                        0/1     Completed           0               13m
kured-v25tc                                       1/1     Running             0               13m
kured-whgkc                                       1/1     Running             0               13m
metrics-server-67c658944b-682tq                   1/1     Running             0               14m

So it seems like cilium is not starting correctly.
These are the only error/warning things i find in the logs.

level=error msg="Error while inserting service in LB map" error="Unable to upsert service [xxxx:xxx:xxxx:xxx::1]:80 as IPv6 is disabled" k8sNamespace=traefik k8sSvcName=traefik subsys=k8s-watcher
level=error msg="Error while inserting service in LB map" error="Unable to upsert service [xxxx:xxx:xxx:xxx::1]:443 as IPv6 is disabled" k8sNamespace=traefik k8sSvcName=traefik subsys=k8s-watcher
level=warning msg="Unable to update CiliumNode resource, will retry" error="Operation cannot be fulfilled on ciliumnodes.cilium.io \"redacted-cl01-control-plane-hel1-ply\": the object has been modified; please apply your changes to the latest version and try again" subsys=nodediscovery

System upgrade controller had the following events in kubectl describe:

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  15m                  default-scheduler  0/5 nodes are available: 2 node(s) had untolerated taint {node.cilium.io/agent-not-ready: }, 3 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  10m                  default-scheduler  0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node.cilium.io/agent-not-ready: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  14s (x2 over 5m14s)  default-scheduler  0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node.cilium.io/agent-not-ready: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling..

If needed I can provide more logs of specific pods. I just wanted to ask first whether and if so which additional logs are required, because I first need to go through and remove private information.
Thanks in advance. If you need any more information feel free to ask.

Kube.tf file

locals {
  hcloud_token = "xxxxxx"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token

  source = "kube-hetzner/kube-hetzner/hcloud"
  
  #temporary fix for permission issue (https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/1145#issuecomment-1875459438)
  postinstall_exec = ["restorecon -v /usr/local/bin/k3s"]

  version = "2.11.3"

  ssh_hcloud_key_label = "role=admin"

  ssh_max_auth_tries = 10

  network_region = "eu-central" # change to `us-east` if location is ash

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-hel1",
      server_type = "cpx11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small-x86-fsn1",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "agent-small-x86-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "agent-large-x86-nbg1",
      server_type = "cpx21",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 0
    },
    {
      name        = "storage",
      server_type = "cpx21",
      location    = "fsn1",
      # Fully optional, just a demo.
      labels      = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count       = 0
    },
    {
      name        = "egress",
      server_type = "cpx11",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/role=egress"
      ],
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule"
      ],
      floating_ip = true
      count = 0
    },
    # Arm based nodes
    {
      name        = "agent-arm-small",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 0
    }
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "nbg1"

  enable_csi_driver_smb = true

  automatically_upgrade_k3s = true

  automatically_upgrade_os = true

  initial_k3s_channel = "stable"

  use_cluster_name_in_node_name = true

  cni_plugin = "cilium"

  dns_servers = [
    "1.1.1.1",
    "8.8.8.8",
    "2606:4700:4700::1111",
  ]

  use_control_plane_lb = true

  control_plane_lb_type = "lb11"

  export_values = true
}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

No response

Platform

macOS 14.2.1

[bverhagen]

I am able to successfully run terraform apply, but afterwards I have the same issue as @byRoadrunner , causing all pods to remain in Pending state.

Yesterday this still worked for me. Last night's auto update resulted in this issue. (re-)creating nodes did not fix this.

[bverhagen]

I managed to fix my cilium issues by tricking kube-hetnzer into reinstalling the Cilium CRDs (that is where the cilium pods were stuck at). In my case, I toggled the cilium egress gateway. Not sure this is feasible solution for you, as your terraform apply is failing (but with some luck, Terraform reinstalls Cilium CRDs before it is hitting the issue you encounter).

[mysticaltech]

Ok, it's selinux. Working on a fix now.

sudo ausearch -m AVC
----
time->Fri Jan  5 23:03:02 2024
type=AVC msg=audit(1704495782.399:1158): avc:  denied  { map_create } for  pid=3952 comm="cilium-operator" scontext=system_u:system_r:container_t:s0:c629,c815 tcontext=system_u:system_r:container_t:s0:c629,c815 tclass=bpf permissive=0
----
time->Fri Jan  5 23:03:06 2024
type=AVC msg=audit(1704495786.659:1167): avc:  denied  { map_create } for  pid=4047 comm="cilium" scontext=system_u:system_r:container_t:s0:c709,c825 tcontext=system_u:system_r:container_t:s0:c709,c825 tclass=bpf permissive=0

[mysticaltech]

@byRoadrunner @bverhagen This was fixed in v2.11.5, but also using the kube.tf setup above, the control plane need to be at least cx21 as cpx11 with 2GB was not enough to allocate bpf memory for cilium, only +500MB was free and it was not enough, when you upgrade the node it works well, and this was only needed for the control-plane. Maybe with less options (like no smb etc). This was on top of the selinux issue.

[bverhagen]

@mysticaltech : Thx! I use cax11 nodes for the control plane, so I can certainly live with that limitation!