Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kernel lockdown impact on bpf #2565

Open
yonghong-song opened this issue Oct 21, 2019 · 12 comments
Open

kernel lockdown impact on bpf #2565

yonghong-song opened this issue Oct 21, 2019 · 12 comments

Comments

@yonghong-song
Copy link
Collaborator

@yonghong-song yonghong-song commented Oct 21, 2019

Starting from kernel 5.4, lockdown is actually merged in the kernel.
This issue is created to track its impact on bcc and tracing. We can add to proper documentation once 5.4 is released.

The bcc/tracing will be impacted when lockdown confidentiality mode is turned on.
The following is kernel source code for different lockdown subcategories for confidentiality mode:
https://github.com/torvalds/linux/blob/master/include/linux/security.h#L104-L126

The following three modes will have impact on bcc:

LOCKDOWN_DEBUGFS
LOCKDOWN_KPROBES
LOCKDOWN_BPF_READ
LOCKDOWN_PERF

kprobe will be disabled, debugfs most part will be disabled unless it is unprivileged. bpf_probe_read() and bpf_probe_read_str() will be disabled. perf_event_open() cannot open an event to get kernel data (others are fine).

Things could change as the 5.4 still has some way to go.

@thiagoftsm

This comment has been minimized.

Copy link

@thiagoftsm thiagoftsm commented Oct 23, 2019

Hi @yonghong-song ,

thank you very much for the information, this will help us to fix possible problems and to do plans for the next version.

@dailyherold

This comment has been minimized.

Copy link

@dailyherold dailyherold commented Feb 9, 2020

Posted same thing in iovisor/bpftrace#925, but I think we may need to escalate this given my fresh 19.10 install with 5.3.0-29-generic kernel had lockdown mode enabled which prevented bcc from working until I found iovisor/bpftrace#853 and disabled it.

@thiagoftsm

This comment has been minimized.

Copy link

@thiagoftsm thiagoftsm commented Feb 9, 2020

Hi @dailyherold ,

In all my tests with the future eBPF collectors, I did not have any problem. Lockdown is present since 5.2, but I did not have any missing chart or performance in the collectors I am working right now.
We will have the proper logs, case something wrong happens.

Best regards!

@dailyherold

This comment has been minimized.

Copy link

@dailyherold dailyherold commented Feb 10, 2020

@thiagoftsm any logs I can pull for you if I re-enable lockdown and run? I definitely want to help validate if it will be a problem for anyone trying to compile on a fresh 19.10 system. I do realize desktop ubuntu installs aren't the primary use case for bcc, but do imagine that a lot of people picking up the BPF book (like me!) may be on newer installs and wanting to mess around on their own system when not at work.

@thiagoftsm

This comment has been minimized.

Copy link

@thiagoftsm thiagoftsm commented Feb 10, 2020

Hi @dailyherold ,

As I said before, I am not having problems with lockdown, all the eBPF collectors are running perfect in my environment, and it will be available for everybody in few weeks.
I remember I had problems in the beginning with lockdown, but I overcome them using as basis codes that came from the Linux kernel source to build our libraries.
I would like to ask you something to confirm that I am not missing nothing, when I had the first problems with lockdown, I found this link https://gehrcke.de/2019/09/running-an-ebpf-program-may-require-lifting-the-kernel-lockdown/ that gave me a short explanation about the problem. Since my first problems, I am always keeping the kernel lockdown enabling when I am developing to be sure it won't block me, but it is possible the information that I got from the links is not enough. Are you doing something different of the link?
A last information, I used bcc in the beginning, but we decided do not move in front with it, because we did not want to add dependencies to Netdata, right now we are compiling everything with C directly.

Best regards!

@brendangregg

This comment has been minimized.

Copy link
Member

@brendangregg brendangregg commented Feb 11, 2020

Many distros are enabling lockdown, breaking BPF. This is the worst OS change I've ever seen.

The disable method using sysrq keys is being removed. On Ubuntu, the only way to disable it going forward will be to run sudo mokutil --disable-validation, which prompts for the password and then the system must be rebooted. This is ridiculous.

It's only a small consolation that it doesn't currently affect virtual machine users, such as on AWS EC2. How long until those support secure boot, and therefore lockdown, and now BPF is disabled on them as well?

See https://lwn.net/Articles/784674/ for background on this controversial patch, which includes quotes like:

most users will really strongly object to turning off BPF when they actually just want to protect kernel integrity.

And yes, we need to object really strongly. Everyone needs to contact their distro vendors and ask them to disable lockdown by default, or find another way to allow BPF by default. I'm 100% serious.

A gist I get from that article is that they were trying to create a mode ("tristate") to not break BPF, but the description doesn't seem to make sense. If the "integrity" state was the compromise state to allow BPF, then it should allow BPF programs to run, not the opposite.

Whoever has the cycles: please study what's going on in detail to list what our options are, other than avoiding distros that do this.

PS. The second-worst OS change I've seen is compiling everything (including libc) without frame pointers by default. That still hurts us at Netflix on a daily basis.

@mrostecki

This comment has been minimized.

Copy link

@mrostecki mrostecki commented Feb 27, 2020

@brendangregg The patch in SLES/openSUSE should not break BPF. We enable integrity lockdown mode when booting in EFI Secure Mode. Without Secure Boot, the lockdown mode is none. We never enable confidentionality. And anyway, we ship Cilium and BCC as parts of our distro and products, so if anything breaks BPF, we are going to fix that.

@brendangregg

This comment has been minimized.

Copy link
Member

@brendangregg brendangregg commented Mar 10, 2020

@mrostecki Yay, thank you! :)

@thomasdullien

This comment has been minimized.

Copy link

@thomasdullien thomasdullien commented Mar 20, 2020

What is the best way to reach out to distro providers (for example, Fedora) to voice concern?

The thing I understand least is: If their goal is to protect cryptographic keys, why blacklist bpf_probe_read and bpf_probe_read_str entirely instead of just modifying those functions to not permit access to a dedicated memory region to keep the keys in?

@viega

This comment has been minimized.

Copy link

@viega viega commented Mar 20, 2020

Agreed w/ Thomas. The current "solution" will significantly (negatively) impact other security applications (host intrusion detection). Thomas' solution seems far better!

@Maryse47

This comment has been minimized.

Copy link

@Maryse47 Maryse47 commented Mar 20, 2020

AFAIK distros provided lockdown out-of-tree patch for years and in those versions bpf was disabled completely not only bpf_probe_read() and bpf_probe_read_str() like in upstream version.

@thomasdullien

This comment has been minimized.

Copy link

@thomasdullien thomasdullien commented Mar 31, 2020

FWIW, after talking with Matthew Garrett (one of the lockdown patch authors) he was kind enough to talk to the Redhat folks that had auto-booted into confidentiality mode in recent Fedora versions and convince them that confidentiality mode has a specific use-case that is not satisfied there.

Fedora 30 and 31 will now boot into integrity mode instead of confidentiality mode. This still makes "naked" kprobes impossible, but at least still allows "regular" reading of kernel memory from eBPF.

Relevant Redhat bugzilla entry https://bugzilla.redhat.com/show_bug.cgi?id=1815571.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.