Kernel with CONFIG_IKHEADERS for BPF tools on Kubernetes

[alban]

I would like the Minikube kernel to be compiled with CONFIG_IKHEADERS.

This is missing from linux_defconfig.

BCC tools need to have access to kernel headers. It can be done either by installing linux-headers packages or by having a kernel compiled with CONFIG_IKHEADERS so that enough information can be retrieved via /sys/kernel/kheaders.tar.xz.

BCC tools are used in Inspektor Gadget, a collection of tools for developers of Kubernetes applications. I would like Inspektor Gadget to support Minikube.

/cc @mauriciovasquezbernal

[tstromberg]

/cc @priyawadhwa

[tstromberg]

@alban - building the headers into the kernel is likely the nicer of the two options, but I could go either way.

I'd be happy to approve a PR that does this: https://minikube.sigs.k8s.io/docs/contrib/building/iso/#adding-kernel-modules

[priyawadhwa]

@alban we have a doc on running BCC tools as a container in minikube, maybe this would work for you until the headers get added to the kernel

https://minikube.sigs.k8s.io/docs/tutorials/ebpf_tools_in_minikube/

[alban]

This issue should be reopened:

Minikube v1.12.0-beta.1 had Linux 5.4.40 (with CONFIG_IKHEADERS) but Minikube v1.12.0 has Linux 4.19.107 (without CONFIG_IKHEADERS since this build option only appeared in Linux 5.2).

See the revert commit: #8649

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[alban]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mauriciovasquezbernal]

/remove-lifecycle stale

[alban]

Additionally, I would like to have CONFIG_DEBUG_INFO_BTF (available from Linux 5.2) to benefit from BPF programs compiled with CO-RE.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[mauriciovasquezbernal]

/remove-lifecycle stale

[sharifelgamal]

So this won't be able to be supported until we update the kernel version of the ISO to v5, which we currently cannot for several reasons. I'm freezing this so that it won't get closed while we wait for that.

[pnasrat]

I came across this issue while experimenting with eBPF and minikube and it looks like current Minikube has the required support as now on the 5.10 kernel per above mentioned #12707 and this issue may be closable (bar the doc updated mentioned below)

eBPF tools can be shown to work in a number of ways by eBPF tools that support the options requested

Note the blog post Running eBPF Tools in Minikube would need to be updated.

• Remove the custom iso
• Use a docker command with volumes for /sys/kernel/kheaders.tar.xz, /sys/fs/bpf
• Use an updated container as the bcc shipped with 'zlim/bcc:latest uses libbcc Version: 0.8.0-78.git.eba6beb4 from Feb 2019 so doesn't include the necessary to load from loader: Add support for reading kernel headers from /proc iovisor/bcc#2312 - below I've just used a fedora container and installed bcc into it to test

Version info

minikube version
minikube version: v1.31.2
commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396e
minikube ssh -- uname -r
5.10.57

Updated example of minikube and execsnoop

This means that if a container has mounted /sys/kernel/kheaders.tar.xz and is privileged with the right tools installed (xz, bcc) you can run execsnoop. Note needs xz here to read headers

minikube ssh
docker run --rm   --privileged -v /sys/kernel/kheaders.tar.xz:/sys/kernel/kheaders.tar.xz -it   -v /lib/modules:/lib/modules:ro   -v /sys/fs/bpf:/sys/fs/bpf fedora:latest /bin/bash
dnf install -y xz bcc
cd /usr/share/bcc/tools
./execsnoop
[root@15ccc8e21c91 tools]# ./execsnoop
PCOMM            PID     PPID    RET ARGS
bridge           103806  1041      0 /opt/cni/bin/bridge
portmap          103811  1041      0 /opt/cni/bin/portmap
iptables         103815  2295      0 /usr/sbin/iptables -w 5 -W 100000 -S KUBE-KUBELET-CANARY -t mangle
ip6tables        103816  2295      0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-KUBELET-CANARY -t mangle
ip6tables        103817  2295      0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-KUBELET-CANARY -t nat
runc             103820  8279      0 /usr/bin/runc --root /var/run/docker/runtime-runc/moby --log /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/651ba58ea0979c86fd16e779b30e6246c85dada4d84bb5dc0ec5547db06 --log-format json exec --process /tmp/runc-process1214458527 --detach --pid-file /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/651ba58ea0979c86fd16e779b30e6246c85dada4d84bb5dc0ec5547db06 651ba58ea0979c86fd16e779b30e6246c85dada4d84bb5dc0ec5547db06cb964

Kernel config of minikube iso

See also

minikube/deploy/iso/minikube-iso/board/minikube/x86_64/linux_x86_64_defconfig

Line 25 in 6bdc0f1

CONFIG_IKCONFIG=y

zgrep -E '(CONFIG_DEBUG_INFO_BTF|CONFIG_IKHEADERS|BPF)' /proc/config.gz

CONFIG_IKHEADERS=y
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_LSM=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_JIT_DEFAULT_ON=y
# CONFIG_BPF_PRELOAD is not set
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_BPFILTER is not set
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
# CONFIG_BPF_STREAM_PARSER is not set
CONFIG_HAVE_EBPF_JIT=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_BPF_EVENTS=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
# CONFIG_TEST_BPF is not set