Cilium installation failing in K3D cluster

[Archisman-Mridha]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

I am trying to install Cilium in my K3D cluster but everytime I am getting an error.

Here is my K3D cluster config file -

apiVersion: k3d.io/v1alpha3
kind: Simple

name: cloudnative-tools-testing

servers: 1
agents: 2

ports:
    - port: 8080:80
      nodeFilters:
        - loadbalancer
    - port: 443:443
      nodeFilters:
        - loadbalancer

options:
    k3s:
        extraArgs:
            - arg: --no-deploy=traefik
              nodeFilters:
                - server:*
            - arg: --disable=traefik
              nodeFilters:
                - server:*
            - arg: --disable-network-policy
              nodeFilters:
                - server:*
            - arg: --flannel-backend=none
              nodeFilters:
                - server:*

I am starting the cluster using this command - k3d cluster create --config ./cluster.config.yaml --api-port=6443

Next I am executing these commands to install Cilium -

docker exec -it k3d-cloudnative-tools-testing-server-0 mount bpffs /sys/fs/bpf -t bpf && \
    docker exec -it k3d-cloudnative-tools-testing-server-0 mount --make-shared /sys/fs/bpf
docker exec -it k3d-cloudnative-tools-testing-agent-0 mount bpffs /sys/fs/bpf -t bpf && \
    docker exec -it k3d-cloudnative-tools-testing-agent-0 mount --make-shared /sys/fs/bpf
docker exec -it k3d-cloudnative-tools-testing-agent-1 mount bpffs /sys/fs/bpf -t bpf && \
    docker exec -it k3d-cloudnative-tools-testing-agent-1 mount --make-shared /sys/fs/bpf

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium --version 1.9.1 \
	--namespace kube-system \
	--set kubeProxyReplacement=strict \
	--set hostServices.enabled=false \
	--set externalIPs.enabled=true \
	--set nodePort.enabled=true \
	--set hostPort.enabled=true \
	--set bpf.masquerade=false \
	--set image.pullPolicy=IfNotPresent \
	--set ipam.mode=kubernetes

Cilium Version

cilium-cli: v0.12.12 compiled with go1.19.4 on linux/amd64
cilium image (default): v1.12.5

Kernel Version

Linux 5.15.79.1-microsoft-standard-WSL2 #1 SMP Wed Nov 23 01:01:46 UTC 2022 x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4+k3s1", GitCommit:"c3f830e9b9ed8a4d9d0e2aa663b4591b923a296e", GitTreeState:"clean", BuildDate:"2022-08-25T03:45:26Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

🔍 Collecting sysdump with cilium-cli version: v0.12.12, args: [sysdump]
🔮 Detected Cilium installation in namespace "kube-system"
Detected Cilium operator in namespace "kube-system"
🔍 Collecting Kubernetes nodes
🔍 Collect Kubernetes nodes
🔍 Collecting Kubernetes events
🔍 Collect Kubernetes version
🔍 Collecting Kubernetes pods
🔍 Collecting Kubernetes services
🔍 Collecting Kubernetes namespaces
🔍 Collecting Kubernetes pods summary
🔍 Collecting Kubernetes endpoints
🔍 Collecting Kubernetes network policies
🔍 Collecting Cilium network policies
🔍 Collecting Kubernetes leases
🔍 Collecting Cilium egress NAT policies
🔍 Collecting Cilium cluster-wide network policies
🔍 Collecting Cilium local redirect policies
🔍 Collecting Cilium Egress Gateway policies
🔍 Collecting Cilium endpoints
🔍 Collecting Cilium identities
🔍 Collecting Cilium nodes
🔍 Collecting Ingresses
🔍 Collecting CiliumClusterwideEnvoyConfigs
🔍 Collecting CiliumEnvoyConfigs
🔍 Collecting Cilium etcd secret
🔍 Collecting the Cilium configuration
couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
🔍 Collecting the Cilium daemonset(s)
🔍 Collecting the Hubble daemonset
🔍 Collecting the Hubble Relay configuration
🔍 Collecting the Hubble UI deployment
🔍 Collecting the Hubble Relay deployment
🔍 Collecting the 'clustermesh-apiserver' deployment
🔍 Collecting the Cilium operator deployment
🔍 Collecting the CNI configuration files from Cilium pods
🔍 Collecting the CNI configmap
🔍 Collecting gops stats from Cilium pods
🔍 Collecting gops stats from Hubble pods
🔍 Collecting bugtool output from Cilium pods
🔍 Collecting gops stats from Hubble Relay pods
🔍 Collecting logs from Cilium pods
⚠️ Deployment "hubble-ui" not found in namespace "kube-system" - this is expected if Hubble UI is not enabled
🔍 Collecting logs from Cilium operator pods
⚠️ Deployment "clustermesh-apiserver" not found in namespace "kube-system" - this is expected if 'clustermesh-apiserver' isn't enabled
🔍 Collecting logs from 'clustermesh-apiserver' pods
⚠️ Deployment "hubble-relay" not found in namespace "kube-system" - this is expected if Hubble is not enabled
🔍 Collecting logs from Hubble pods
couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
🔍 Collecting logs from Hubble Relay pods
couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
🔍 Collecting logs from Hubble UI pods
🔍 Collecting bugtool output from Tetragon pods
🔍 Collecting platform-specific data
🔍 Collecting kvstore data
🔍 Collecting Hubble flows from Cilium pods
⚠️ Container "cilium-agent" for pod "cilium-dz9fz" in namespace "kube-system" is not running. Trying EphemeralContainer or separate Pod instead...
⚠️ Container "cilium-agent" for pod "cilium-nnd96" in namespace "kube-system" is not running. Trying EphemeralContainer or separate Pod instead...
⚠️ Container "cilium-agent" for pod "cilium-kfzs7" in namespace "kube-system" is not running. Trying EphemeralContainer or separate Pod instead...

Relevant log output

/¯¯\
 /¯¯\__/¯¯\    Cilium:         4 errors
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet         cilium             Desired: 3, Unavailable: 3/3
Containers:       cilium             Running: 3
                  cilium-operator    Running: 2
Cluster Pods:     0/3 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.9.6: 3
                  cilium-operator    quay.io/cilium/operator-generic:v1.9.6: 2
Errors:           cilium             cilium          3 pods of DaemonSet cilium are not ready
                  cilium             cilium-x694f    unable to retrieve cilium status: container cilium-agent is in CrashLoopBackOff, exited with code 1: 2023-01-23T09:35:22.345063585Z level=fatal msg="Error while creating daemon" error="cannot add static proxy rules: exit status 2" subsys=daemon
                  cilium             cilium-5kgkz    unable to retrieve cilium status: container cilium-agent is in CrashLoopBackOff, exited with code 1: try 'kubectl -n kube-system logs -c cilium-agent cilium-5kgkz'
                  cilium             cilium-c8sj7    unable to retrieve cilium status: container cilium-agent is not running, exited with code 1: try 'kubectl -n kube-system logs -c cilium-agent cilium-c8sj7'

Anything else?

Logs I got after running kubectl -n kube-system logs -c cilium-agent cilium-c8sj7 | grep error -

level=warning msg="iptables modules could not be initialized. It probably means that iptables is not available on this system" error="could not load module ip_tables: exit status 1" subsys=iptables
level=warning msg="Failed to sysctl -w" error="could not open the sysctl file /proc/sys/net/core/bpf_jit_enable: open /proc/sys/net/core/bpf_jit_enable: no such file or directory" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
level=error msg="Command execution failed" cmd="[iptables -w 5 -t raw -A CILIUM_PRE_raw -m mark --mark 0x00000200/0x00000f00 -m comment --comment cilium: NOTRACK for proxy traffic -j NOTRACK]" error="exit status 2" subsys=iptables
level=error msg="Error while initializing daemon" error="cannot add static proxy rules: exit status 2" subsys=daemon
level=fatal msg="Error while creating daemon" error="cannot add static proxy rules: exit status 2" subsys=daemon

Code of Conduct

• I agree to follow this project's Code of Conduct

[aanm]

@Archisman-Mridha can you try with a more recent version. 1.9 is EOL

[Archisman-Mridha]

@Archisman-Mridha can you try with a more recent version. 1.9 is EOL

I tried. Still getting same error

[borkmann]

Do you have the requirements enabled in the underlying kernel? Looks like iptables is unable to install proxy rules.
https://docs.cilium.io/en/stable/operations/system_requirements/#linux-kernel

[Archisman-Mridha]

I found out it was happening because of WSL2. To my knowledge, WL2 doesn't support eBPF and there are also other hazards like systemd and dbus are not present (we can install them though). If I want to get WSL2 and eBPF working together, I need to recompile the WSL2 kernel.