kernel lockdown impact on bpf

[yonghong-song]

Starting from kernel 5.4, lockdown is actually merged in the kernel.
This issue is created to track its impact on bcc and tracing. We can add to proper documentation once 5.4 is released.

The bcc/tracing will be impacted when lockdown confidentiality mode is turned on.
The following is kernel source code for different lockdown subcategories for confidentiality mode:
https://github.com/torvalds/linux/blob/master/include/linux/security.h#L104-L126

The following three modes will have impact on bcc:

LOCKDOWN_DEBUGFS
LOCKDOWN_KPROBES
LOCKDOWN_BPF_READ
LOCKDOWN_PERF

kprobe will be disabled, debugfs most part will be disabled unless it is unprivileged. bpf_probe_read() and bpf_probe_read_str() will be disabled. perf_event_open() cannot open an event to get kernel data (others are fine).

Things could change as the 5.4 still has some way to go.

[thiagoftsm]

Hi @yonghong-song ,

thank you very much for the information, this will help us to fix possible problems and to do plans for the next version.

[dailyherold]

Posted same thing in bpftrace/bpftrace#925, but I think we may need to escalate this given my fresh 19.10 install with 5.3.0-29-generic kernel had lockdown mode enabled which prevented bcc from working until I found bpftrace/bpftrace#853 and disabled it.

[thiagoftsm]

Hi @dailyherold ,

In all my tests with the future eBPF collectors, I did not have any problem. Lockdown is present since 5.2, but I did not have any missing chart or performance in the collectors I am working right now.
We will have the proper logs, case something wrong happens.

Best regards!

[dailyherold]

@thiagoftsm any logs I can pull for you if I re-enable lockdown and run? I definitely want to help validate if it will be a problem for anyone trying to compile on a fresh 19.10 system. I do realize desktop ubuntu installs aren't the primary use case for bcc, but do imagine that a lot of people picking up the BPF book (like me!) may be on newer installs and wanting to mess around on their own system when not at work.

[thiagoftsm]

Hi @dailyherold ,

As I said before, I am not having problems with lockdown, all the eBPF collectors are running perfect in my environment, and it will be available for everybody in few weeks.
I remember I had problems in the beginning with lockdown, but I overcome them using as basis codes that came from the Linux kernel source to build our libraries.
I would like to ask you something to confirm that I am not missing nothing, when I had the first problems with lockdown, I found this link https://gehrcke.de/2019/09/running-an-ebpf-program-may-require-lifting-the-kernel-lockdown/ that gave me a short explanation about the problem. Since my first problems, I am always keeping the kernel lockdown enabling when I am developing to be sure it won't block me, but it is possible the information that I got from the links is not enough. Are you doing something different of the link?
A last information, I used bcc in the beginning, but we decided do not move in front with it, because we did not want to add dependencies to Netdata, right now we are compiling everything with C directly.

Best regards!

[brendangregg]

Many distros are enabling lockdown, breaking BPF. This is the worst OS change I've ever seen.

• Ubuntu: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/disco/commit/arch/x86/kernel/setup.c?id=5b5c31ad1f31e127ec102bb6dd0e89e97fed8b34
• Fedora: https://git.kernel.org/pub/scm/linux/kernel/git/jwboyer/fedora.git/commit/?h=f31&id=9b2af6f393445a64a777bcc86c7478319290c093
• Debian: https://salsa.debian.org/kernel-team/linux/blob/sid/debian/patches/features/all/lockdown/efi-lock-down-the-kernel-if-booted-in-secure-boot-mo.patch
• SLES: openSUSE/kernel@d3416b9

The disable method using sysrq keys is being removed. On Ubuntu, the only way to disable it going forward will be to run sudo mokutil --disable-validation, which prompts for the password and then the system must be rebooted. This is ridiculous.

It's only a small consolation that it doesn't currently affect virtual machine users, such as on AWS EC2. How long until those support secure boot, and therefore lockdown, and now BPF is disabled on them as well?

See https://lwn.net/Articles/784674/ for background on this controversial patch, which includes quotes like:

most users will really strongly object to turning off BPF when they actually just want to protect kernel integrity.

And yes, we need to object really strongly. Everyone needs to contact their distro vendors and ask them to disable lockdown by default, or find another way to allow BPF by default. I'm 100% serious.

A gist I get from that article is that they were trying to create a mode ("tristate") to not break BPF, but the description doesn't seem to make sense. If the "integrity" state was the compromise state to allow BPF, then it should allow BPF programs to run, not the opposite.

Whoever has the cycles: please study what's going on in detail to list what our options are, other than avoiding distros that do this.

PS. The second-worst OS change I've seen is compiling everything (including libc) without frame pointers by default. That still hurts us at Netflix on a daily basis.

[vadorovsky]

@brendangregg The patch in SLES/openSUSE should not break BPF. We enable integrity lockdown mode when booting in EFI Secure Mode. Without Secure Boot, the lockdown mode is none. We never enable confidentionality. And anyway, we ship Cilium and BCC as parts of our distro and products, so if anything breaks BPF, we are going to fix that.

[brendangregg]

@mrostecki Yay, thank you! :)

[thomasdullien]

What is the best way to reach out to distro providers (for example, Fedora) to voice concern?

The thing I understand least is: If their goal is to protect cryptographic keys, why blacklist bpf_probe_read and bpf_probe_read_str entirely instead of just modifying those functions to not permit access to a dedicated memory region to keep the keys in?

[viega]

Agreed w/ Thomas. The current "solution" will significantly (negatively) impact other security applications (host intrusion detection). Thomas' solution seems far better!

[Maryse47]

AFAIK distros provided lockdown out-of-tree patch for years and in those versions bpf was disabled completely not only bpf_probe_read() and bpf_probe_read_str() like in upstream version.

[thomasdullien]

FWIW, after talking with Matthew Garrett (one of the lockdown patch authors) he was kind enough to talk to the Redhat folks that had auto-booted into confidentiality mode in recent Fedora versions and convince them that confidentiality mode has a specific use-case that is not satisfied there.

Fedora 30 and 31 will now boot into integrity mode instead of confidentiality mode. This still makes "naked" kprobes impossible, but at least still allows "regular" reading of kernel memory from eBPF.

Relevant Redhat bugzilla entry https://bugzilla.redhat.com/show_bug.cgi?id=1815571.

[qmonnet]

For the record, Ubuntu also switched to integrity mode by default (launchpad bug / fix for Focal).

[qmonnet]

There are now a PR and a ticket for switching to integrity mode in Debian.

Edit: Now merged.

[jclyons]

posted a question here bpftrace/bpftrace#853 (comment) also relevant to this repository - we shouldn't have to ask users to disable boot protection in order to use a userspace program. if there was a way to get authority trust for our userspsace program that uses BPF, that would allow us to distribute userspace programs that are able to take advantage of BPF but are also trusted without additional configuration