The node-agent is repeatedly crashing with a CrashLoopBackOff error : cannot create bpf perf link

[antonissmal]

Description

I'm encountering an issue with the node-agent in my Kubernetes cluster running Kubescape. The node-agent is repeatedly crashing with a CrashLoopBackOff error, and I'm seeing the following error messages in the logs:

{"level":"error","ts":"2024-01-29T12:13:19Z","msg":"error starting exec tracing","error":"creating tracer: attaching exit tracepoint: cannot create bpf perf link: permission denied"}
{"level":"fatal","ts":"2024-01-29T12:13:19Z","msg":"error starting the container watcher","error":"starting app behavior tracing: creating tracer: attaching exit tracepoint: cannot create bpf perf link: permission denied"}

Environment

OS: Linux (kernel version: 5.14.0-362.13.1.el9_3.x86_64)
Kubescape : v3.0.3
Helm chart : v1.8.1
Kubernetes Server : v1.27.6
Kubernetes Client : v1.26.3

Expected behavior

The node-agent pods should start successfully and not crash with the CrashLoopBackOff error.

Actual Behavior

The node-agent pods are repeatedly crashing with a CrashLoopBackOff error, and the logs show errors related to creating a BPF perf link due to permission denied.

Additional context

The node-agent container is using the quay.io/kubescape/node-agent:v0.1.114 image, and the pod has resource limits and requests defined.

I've checked the AppArmor profile for the node-agent container, and it's set to "unconfined," indicating that it's not confined by AppArmor security policies.

I also reviewed the RBAC permissions and ClusterRoleBinding for the node-agent service account, and it appears to have the necessary permissions and already checked kernel capabilities and seems to be right.

I use cilium CNI on the cluster.

[dwertent]

@antonissmal thank you for reporting.
We are looking into it.
Can you please share some more information about the cluster? is it a managed cluster? ks3? etc.

Edit:
Does this help you?

[antonissmal]

Hello,
Thank you for your response.

Kubeadm self-managed cluster: v1.27.6
ContainerD: containerd.io v1.6.22

Another extra info is that i already use cilium as CNI that uses ebpf.

I saw also today that another same issue has another guy and open an issue in kubescape repo:
kubescape/kubescape#1596

[dwertent]

Hi, @antonissmal
Please follow the workaround mentioned in this link: here.
After you try it, let me know if it works for you. If so, we'll make sure to update the troubleshooting documentation accordingly.