Distro versions and BTF support

[grantseltzer]

I couldn't find a central source of truth for which LTS distro releases have had BTF backported to them but went through some of the big ones manually using vagrant:

Ubuntu 14.04 - linux 3.13 - no ebpf support
Ubuntu 16.04 - linux 4.4.0 - ebpf, no BTF
Ubuntu 18.04 - linux 4.15. - ebpf, no BTF
Ubuntu 20.04 - linux 5.4 - BTF is supported in kernel, but CONFIG_DEBUG_INFO_BTF is not set
Ubuntu 20.10 - linux 5.8 - BTF enabled

Centos 7 - linux 3.10 - ebpf yes, no BTF support
Centos 8 - linux 4.18 - BTF supported and enabled

Debian 10 - linux 4.19 - No BTF support

Based on this we can't only support tracee CO:RE, and compiling against various kernel headers for releases would still be very helpful.

[rafaeldtinoco]

Hello Grant,

Ubuntu kernels are providing BTF debug info support (through sysfs) from Ubuntu Groovy and on. For the HWE kernels (Groovy kernel running on Focal or Bionic), they do support CONFIG_DEBUG_INFO_BTF but don't have config option enabled because link-vmlinux.sh - from kernel tree - uses pahole (from dwarves package) and it needs to be recent to support BTF (not available before Groovy in Ubuntu userland set of packages). Canonical kernel kernel team has disabled BTF support for HWE kernels for Bionic and Focal because builders don't have recent pahole binary.

That explains why I'm packaging ELF files containing BTF section only for Focal and Bionic in my project (conntracker) at:
https://github.com/rafaeldtinoco/conntracker#ubuntu-bionic-and-focal-special-need-for-ebpf-feature

It is not perfect but, by having always the latest kernel vmlinux BTF section available, you can almost all the times run same code even in previous minor versions. For Bionic I have a BTF file for 4.15.x and another for 5.4.x... for Focal I have a BTF file for 5.4.x and 5.8.x. By stripping the ELF files with the BTF sections you will usually end up with a ~3MB ELF file containing the BTF info. I usually use the procedure at: https://github.com/rafaeldtinoco/portablebpf/#portable-libbpf-based-ebpf-code-older-kernels.

This is how I made conntracker CO.RE in Ubuntu =) ... + the following patches:

https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/libbpf-introduce-legacy-kprobe-events-support.patch
which allows me to kprobe ANY kernel function even after GCC optimizations (with symbol name changes)

and the upstream commit:

commit b09a499
Author: Rafael David Tinoco <rafaeldtinoco@ubuntu.com>
Date:   Tue Mar 23 01:09:52 2021

    libbpf: Add bpf object kern_version attribute setter

    Unfortunately some distros don't have their kernel version defined
    accurately in <linux/version.h> due to different long term support
    reasons.

    It is important to have a way to override the bpf kern_version
    attribute during runtime: some old kernels might still check for
    kern_version attribute during bpf_prog_load().

    Signed-off-by: Rafael David Tinoco <rafaeldtinoco@ubuntu.com>
    Signed-off-by: Alexei Starovoitov <ast@kernel.org>
    Acked-by: Andrii Nakryiko <andrii@kernel.org>
    Acked-by: John Fastabend <john.fastabend@gmail.com>
    Link: https://lore.kernel.org/bpf/20210323040952.2118241-1-rafaeldtinoco@ubuntu.com

that allows eBPF code to bypass eBPF ELF attribute version checks in older kernels (like v4.15), as showed in:
https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L288

LINKS:
https://github.com/rafaeldtinoco/conntracker/tree/main/ebpf
https://github.com/rafaeldtinoco/conntracker/tree/main/debian.bionic
https://github.com/rafaeldtinoco/conntracker/tree/main/debian.focal
https://github.com/rafaeldtinoco/conntracker/tree/main/debian.groovy

[rafaeldtinoco]

The TL;DR version is you have 2 options:

Recompile the kernels using their source package - patching old kernels link-vmlinux.sh script (https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/link-vmlinux.sh.patch) AND enabling CONFIG_DEBUG_INFO_BTF for the kernels that support - and this will make the generated vmlinux ELF file to contain a .BTF section with the types info (AND enable /sys/kernel/btf/vmlinux in the new kernels).

Old kernels would not have CONFIG_DEBUG_INFO_BTF option so they have to use BTF info from the -dbgsym package (from /usr/lib/debug/boot/vmlinux file) OR from a stand alone BTF ELF file.

Old kernels also need my 'legacy kprobe' patch to libbpf if they want to use kprobes in any kernel function (instead of probing syscall entry/return only).

What you probably really want is simply to install linux-image-generic-dbgsym kernel and (linux-image-hwe-generic-dbgsym package for the HWE kernel) - for ANY kernel version - and use pahole -J to create a .BTF ELF section (from the DWARF info) in the vmlinux file itself (from the -dbgsym package), extract the BTF section to a new ELF file and use it to serve libbpf for your CO.RE package.

Bionic and Focal would need those BTF ELF files for their vanilla kernels and their HWE (groovy) ones.

Groovy kernel already supports CONFIG_DEBUG_INFO_BTF and has it enabled (/sys/kernel/btf/vmlinux) so CO.RE would already be feasible there without a BTF extra file (but the patch for 'legacy kprobe' also adds the functionality of allowing optimized functions to be kprobed).

Details are in the README file in the portablebpf repo (and a real example is in the conntracker ebpf/ directory).
The good thing about this approach is that you can have CO.RE code running in older kernels, older environments (not updated to latest v4.x kernels that might (or not) support BTF.

I hope this helps you!

[grantseltzer]

Ah awesome, thank you so much @rafaeldtinoco. I'm not sure we're going to go through the trouble of setting that up but regardless this is great to know. I published these findings about distro/kernel btf support because we were exploring whether or not we can remove the dependency steps of compiling against various kernel headers and instead rely solely on BTF support. I don't think that's the case and there isn't much of a need for it besides cleaning up the code base a bit. I'm working on making support CO:RE in tracee-ebpf at the moment.

[rafaeldtinoco]

Yep, it definitely makes sense! Specially considering that my eBPF code had to be very simple to be supported in v4.15 kernels (no global data, no bpf hash map support, etc) if using kprobes only approach. I just wanted to clarify your initial comment on BTF not being available and the workarounds that were needed to override that. I'm glad it clarified it. Best!

[rafaeldtinoco]

To end discussion I have opened the following LP bug:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1926330

Checking if we can convince kernel team to have 'pahole' binary backported to Bionic and Focal, enabling CONFIG_DEBUG_INFO_BTF option on both. This way tracee-bpf would be supported, at least, in Bionic and Focal.. as long as both are using HWE kernels.

[rafaeldtinoco]

BTW this discussion is getting close to:

#574 (reply in thread)

now.

[itaysk]

Here's a list of BTF enabled distros from libbpf readme on GitHub: https://github.com/libbpf/libbpf#bpf-co-re-compile-once--run-everywhere

Fedora 31+
RHEL 8.2+
OpenSUSE Tumbleweed (in the next release, as of 2020-06-04)
Arch Linux (from kernel 5.7.1.arch1-1)
Ubuntu 20.10
Debian 11 (amd64/arm64)

[yanivagman]

In addition to this list, manjaro maintainers also added BTF support which should be part of the next kernel release after I asked them to reconsider this:
https://forum.manjaro.org/t/co-re-support-in-kernel/46134/19

[rafaeldtinoco]

• Ubuntu Focal (latest LTS) info:

Ubuntu has the concept of HWE kernels (running newer kernels in older Linux versions). In this case, very common in cloud production environments, those kernels, despite being new, don't support BTF.

I have opened this BUG to address the issue and hopefully it will get fixed. If not, we might need to generate BTF for Ubuntu Focal kernels AND Ubuntu Focal HWE kernels.

[rafaeldtinoco]

BTW this discussion is getting close to:

#574 (reply in thread)

now.