cilium-agent crashes with 'protocol not supported'

[elessyl]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Trying to add a new ARM64 (Raspberry Pi 4B) node to an existing cluster.
I was expecting the cilium POD will start normally, however it was in CrashLoopBackOff status.
It looks like there is a 'protocol not supported' error raised.

Cilium Version

cilium-cli: 0.12.0 compiled with go1.18.4 on darwin/arm64
cilium image (default): v1.12.0
cilium image (stable): v1.12.0
cilium image (running): v1.12.0

Kernel Version

Linux worker-arm64-001 5.15.0-1012-raspi #14-Ubuntu SMP PREEMPT Fri Jun 24 13:10:28 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Kubernetes Version

All nodes are on 1.24.3-00

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:23:26Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

cilium-sysdump-20220801-214249.zip

Relevant log output

level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.234.0.0/16
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://x.x.x.x:6443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=debug msg="Starting new controller" name=k8s-heartbeat subsys=controller uuid=a728903d-778d-493e-af14-7436652e2ee5
level=debug msg="Controller func execution time: 4.112µs" name=k8s-heartbeat subsys=controller uuid=a728903d-778d-493e-af14-7436652e2ee5
level=debug msg="Skipping Leases support fallback discovery" subsys=k8s
level=debug msg="Starting new controller" name=endpoint-gc subsys=controller uuid=91f98097-0846-4831-aae6-fcae82e7a9b0
level=debug msg="Controller func execution time: 6.076µs" name=endpoint-gc subsys=controller uuid=91f98097-0846-4831-aae6-fcae82e7a9b0
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"bpf-lb-sock\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=192.168.123.234 mtu=1500 subsys=mtu
level=debug msg="Performing regular background work" subsys=nodemanager syncInterval=1m0s
level=fatal msg="Error while creating daemon" error="protocol not supported" subsys=daemon

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[andy-v-h]

Are you installing with helm?

[elessyl]

I think my problem might be RTFM.
Kernel modules were missing, which are in a different package on Ubuntu (kernel-modules-extra): xfrm_user/xfrm_interface

I might still have other issues with datapath-loader but I will rebuild the node from scratch first.

[samip5]

How does one get the required modules on arm64 (Raspberry Pi running Ubuntu 22.04)?
I'm using Helm and that error should say which protocol it's talking about.

level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.10.0.0/16
level=info msg="Using autogenerated IPv6 allocation range" subsys=node v6Prefix="f00d::a0a:0:0:0/96"
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://10.0.105.10:6443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"bpf-lb-sock\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=10.0.105.10 mtu=1500 subsys=mtu
level=fatal msg="Error while creating daemon" error="protocol not supported" subsys=daemon

[elessyl]

How does one get the required modules on arm64 (Raspberry Pi running Ubuntu 22.04)? I'm using Helm and that error should say which protocol it's talking about.

level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.10.0.0/16
level=info msg="Using autogenerated IPv6 allocation range" subsys=node v6Prefix="f00d::a0a:0:0:0/96"
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://10.0.105.10:6443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"bpf-lb-sock\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=10.0.105.10 mtu=1500 subsys=mtu
level=fatal msg="Error while creating daemon" error="protocol not supported" subsys=daemon

Please make sure you have linux-modules-extra-raspi package installed. After you installed it, I recommend to reboot your node and it should come up normally.