[Modern eBPF] libpman: tracing program type is not supported #2792

kennylindberg · 2023-09-08T12:54:19Z

When trying to start falco using modern eBPF, we get the following error:

# /usr/bin/falco --pidfile=/var/run/falco.pid --modern-bpf
[...]
Fri Sep  8 14:38:29 2023: One ring buffer every '2' CPUs.
libpman: tracing program type is not supported (errno: 22 | message: Invalid argument)
Fri Sep  8 14:38:29 2023: An error occurred in an event source, forcing termination...
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
Error:

Falco version: 0.35.1
OS: Oracle Linux 7.9
Kernel: 5.4.17-2136.322.6.2.el7uek.x86_64

Despite the kernel version is version 5.4, BPF ring buffer and BTF should be available:

# bpftool feature probe kernel | grep -q "map_type ringbuf is available" && echo "true" || echo "false"
true
# file /sys/kernel/btf/vmlinux
/sys/kernel/btf/vmlinux: data

Running a stracktrace seems like the kernel version isn't detected correctly (KERNEL_VERSION(0, 0, 0)):

bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=2, insns=0x7fff51c782f0, license="GPL", log_level=1, log_size=4096, log_buf=" s\307Q\377\177", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=0x18 /* BPF_??? */, ...}, 128) = -1 EINVAL (Invalid argument)
write(2, "libpman: tracing program type is"..., 87libpman: tracing program type is not supported (errno: 22 | message: Invalid argument)

The text was updated successfully, but these errors were encountered:

Andreagit97 · 2023-09-08T13:55:52Z

It seems like BTF programs are not supported on your machine, try this:

sudo bpftool feature probe kernel | grep -q "program_type tracing is available" && echo "true" || echo "false"

Unfortunately these https://falco.org/docs/event-sources/kernel/#requirements are essential requirements to run the modern bpf probe :/

kennylindberg · 2023-09-11T06:51:15Z

As far as I can see, BTF should be supported and compiled into the kernel:

# cat /boot/config-5.4.17-2136.322.6.2.el7uek.x86_64 | grep BTF
# CONFIG_VIDEO_SONY_BTF_MPX is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_PAHOLE_HAS_SPLIT_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y

The command you've sent is unreliable, as it returns false on my machine, but also returns false on our Oracle Linux 8 machines where the modern BPF actually works:

# bpftool feature probe kernel | grep "program_type tracing"
eBPF program_type tracing is NOT available
[root@localhost ~]# systemctl status falco-modern-bpf
● falco-modern-bpf.service - Falco: Container Native Runtime Security with modern ebpf
   Loaded: loaded (/usr/lib/systemd/system/falco-modern-bpf.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-09-11 08:46:08 CEST; 6min ago
     Docs: https://falco.org/docs/
 Main PID: 789 (falco)
    Tasks: 10 (limit: 22887)
   Memory: 55.5M
   CGroup: /system.slice/falco-modern-bpf.service
           └─789 /usr/bin/falco --pidfile=/var/run/falco.pid --modern-bpf

Sep 11 08:46:08 localhost.localdomain falco[789]: Falco version: 0.35.1 (x86_64)
Sep 11 08:46:08 localhost.localdomain falco[789]: Falco initialized with configuration file: /etc/falco/falco.yaml
Sep 11 08:46:08 localhost.localdomain falco[789]: Loading rules from file /etc/falco/falco_rules.yaml
Sep 11 08:46:08 localhost.localdomain falco[789]: Loading rules from file /etc/falco/falco_rules.local.yaml
Sep 11 08:46:08 localhost.localdomain falco[789]: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Sep 11 08:46:08 localhost.localdomain falco[789]: Starting health webserver with threadiness 4, listening on port 8765
Sep 11 08:46:08 localhost.localdomain falco[789]: Loaded event sources: syscall
Sep 11 08:46:08 localhost.localdomain falco[789]: Enabled event sources: syscall
Sep 11 08:46:08 localhost.localdomain falco[789]: Opening 'syscall' source with modern BPF probe.
Sep 11 08:46:08 localhost.localdomain falco[789]: One ring buffer every '2' CPUs.

Andreagit97 · 2023-09-11T08:29:23Z

Having the BTF feature enabled on the system doesn't mean that all BTF programs are supported... let me explain better, there are several program types in BPF (https://docs.kernel.org/bpf/libbpf/program_types.html). In the modern probe, we need the prog type BPF_PROG_TYPE_TRACING but as you can notice the BPF_PROG_TYPE_TRACING can have different attach types:

+-------------------------------------------+----------------------------------------+----------------------------------+-----------+
| ``BPF_PROG_TYPE_TRACING``                 | ``BPF_MODIFY_RETURN``                  | ``fmod_ret+`` [#fentry]_         |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``fmod_ret.s+`` [#fentry]_       | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_FENTRY``                   | ``fentry+`` [#fentry]_           |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``fentry.s+`` [#fentry]_         | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_FEXIT``                    | ``fexit+`` [#fentry]_            |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``fexit.s+`` [#fentry]_          | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_ITER``                     | ``iter+`` [#iter]_               |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``iter.s+`` [#iter]_             | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_RAW_TP``                   | ``tp_btf+`` [#fentry]_           |           |
+-------------------------------------------+----------------------------------------+----------------------------------+-----------+

More in detail in the modern probe we need the BPF_TRACE_RAW_TP attach type but right now there are no available functions to probe this specific attach type, so what we do is an approximation, as you said. We probe the BPF_TRACE_FENTRY attach type since it is the only attach type we can probe in the kernel... So yes the check is not 100% reliable but we cannot do more than that :/

kennylindberg · 2023-09-11T08:54:32Z

Thank you for the detailed explanation.

Wouldn't it be possible just to probe this directly through the BTF sys fs export /sys/kernel/btf/vmlinux?

As you can see, the following tells me that the BPF type BPF_TRACE_RAW_TP is available:

# strings /sys/kernel/btf/vmlinux | grep -i BPF_TRACE_RAW_TP
BPF_TRACE_RAW_TP

Also, it tells me that the BPF type BPF_TRACE_FENTRY isn't available, why the modern BPF fails to start:

# strings /sys/kernel/btf/vmlinux | grep -i BPF_TRACE_FENTRY
(no output)

Andreagit97 · 2023-09-11T13:09:50Z

Yeah, this seems a great idea! The best solution would probably be to implement in libbpf the possibility of probing our exact attach type BPF_TRACE_RAW_TP but at the moment the one you proposed could be a possible workaround!
Unfortunately, we are really near the release so I don't think we can do that for Falco 0.36 :/

kennylindberg · 2023-09-11T13:44:14Z

That sounds great! We will look forward to that :)

FedeDP · 2023-10-25T12:43:40Z

Hi! I just released Falco 0.36.2-rc1 that should solve this issue; care to try it?
You can use docker images or packages just like you would do for a normal Falco release!

kennylindberg · 2023-10-25T12:59:16Z

Sure! However, I don't see the RPM package to be available anywhere?

FedeDP · 2023-10-25T13:00:09Z

Oh yep because it got published under the rpm-dev repo: https://download.falco.org/packages/rpm-dev/falco-0.36.2-rc1-x86_64.rpm

kennylindberg · 2023-10-25T13:05:31Z

Ah thanks. It's still failing now with the following output:

Wed Oct 25 15:04:21 2023: One ring buffer every '2' CPUs.
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
; int BPF_PROG(sys_enter,
0: (bf) r6 = r1
; int BPF_PROG(sys_enter,
1: (79) r7 = *(u64 *)(r6 +8)
; return g_64bit_interesting_syscalls_table[syscall_id & (SYSCALL_TABLE_SIZE - 1)];
2: (bf) r1 = r7
3: (57) r1 &= 511
; return g_64bit_interesting_syscalls_table[syscall_id & (SYSCALL_TABLE_SIZE - 1)];
4: (18) r2 = 0xffffa92ec28ce000
6: (0f) r2 += r1
last_idx 6 first_idx 0
regs=2 stack=0 before 4: (18) r2 = 0xffffa92ec28ce000
regs=2 stack=0 before 3: (57) r1 &= 511
regs=2 stack=0 before 2: (bf) r1 = r7
regs=80 stack=0 before 1: (79) r7 = *(u64 *)(r6 +8)
7: (71) r2 = *(u8 *)(r2 +0)
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=map_value(id=0,off=0,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; if(!syscalls_dispatcher__64bit_interesting_syscall(syscall_id))
8: (15) if r2 == 0x0 goto pc+76
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; return g_settings.dropping_mode;
9: (18) r2 = 0xffffa92ec28ce5a8
11: (71) r2 = *(u8 *)(r2 +12)
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=map_value(id=0,off=1448,ks=4,vs=242537,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; if(!maps__get_dropping_mode())
12: (15) if r2 == 0x0 goto pc+37
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; return g_64bit_sampling_syscall_table[syscall_id & (SYSCALL_TABLE_SIZE - 1)];
13: (18) r2 = 0xffffa92ec28ce200
15: (0f) r2 += r1
16: (71) r1 = *(u8 *)(r2 +0)
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=map_value(id=0,off=512,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; if(sampling_flag == UF_NEVER_DROP)
17: (15) if r1 == 0x2 goto pc+32
 R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2=map_value(id=0,off=512,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0) R10=fp0
18: (15) if r1 == 0x4 goto pc+66
 R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2=map_value(id=0,off=512,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0) R10=fp0
; if((bpf_ktime_get_boot_ns() % SECOND_TO_NS) >= (SECOND_TO_NS / maps__get_sampling_ratio()))
19: (85) call bpf_uuused15#125
unknown func bpf_uuused15#125
processed 17 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
-- END PROG LOAD LOG --
libbpf: prog 'sys_enter': failed to load: -22
libbpf: failed to load object 'bpf_probe'
libbpf: failed to load BPF skeleton 'bpf_probe': -22
libpman: failed to load BPF object (errno: 22 | message: Invalid argument)
Wed Oct 25 15:04:21 2023: An error occurred in an event source, forcing termination...
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
Error: Initialization issues during scap_init

FedeDP · 2023-10-25T13:41:34Z

This seems a different problem, right?

19: (85) call bpf_uuused15#125
unknown func bpf_uuused15#125

I think the issue is because on 5.4 BPF_FUNC_ktime_get_boot_ns did not exist; but that is what we are using in the modern bpf probe since it was made to work against kenrels >= 5.8: https://github.com/falcosecurity/libs/blob/master/driver/modern_bpf/helpers/interfaces/attached_programs.h#L59.
Perhaps we might need to make the same workaround we do for old bpf probe (https://github.com/falcosecurity/libs/blob/master/driver/bpf/bpf_helpers.h#L24)?
I am not sure this is the reason of the failure anyway, let me ping @Andreagit97!

Andreagit97 · 2023-10-25T15:37:10Z

Uhm yes, the issue seems the one described by @FedeDP, BTW I've never seen this bpf_uuused15... To be honest, this issue is due to some missing backporting on kernel 5.4.17-2136.322.6.2.el7uek.x86_64 so I think that the best solution here is not to use the modern bpf at all. We don't know which helpers are backported or not... the suggestion here is to switch to a recent kernel version ASAP or to use one of the other 2 drivers (legacy_bpf,kernel module) if possible

kennylindberg · 2023-10-26T07:23:55Z

All right, I see. Thank you for your help and the explanation.

kennylindberg added the kind/bug label Sep 8, 2023

Andreagit97 modified the milestones: TBD, 0.36.0 Sep 8, 2023

Andreagit97 self-assigned this Sep 11, 2023

Andreagit97 modified the milestones: 0.36.0, TBD, 0.37.0 Sep 11, 2023

Andreagit97 mentioned this issue Oct 11, 2023

fix(modern): perform an exact check on BPF_TRACE_RAW_TP attach type falcosecurity/libs#1404

Merged

incertum mentioned this issue Oct 16, 2023

[UMBRELLA] Errors at Falco start-up related to Falco's kernel driver {kmod, bpf, modern_bpf} #2873

Open

Andreagit97 modified the milestones: 0.37.0, 0.36.2 Oct 24, 2023

kennylindberg closed this as completed Oct 26, 2023

Andreagit97 mentioned this issue Nov 7, 2023

Falco won't start on 0.36 - /sys/devices/system/cpu/cpu8/online: No such file or directory #2843

Closed

Andreagit97 changed the title ~~libpman: tracing program type is not supported~~ [Modern eBPF] libpman: tracing program type is not supported Dec 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Modern eBPF] libpman: tracing program type is not supported #2792

[Modern eBPF] libpman: tracing program type is not supported #2792

kennylindberg commented Sep 8, 2023 •

edited

Andreagit97 commented Sep 8, 2023

kennylindberg commented Sep 11, 2023 •

edited

Andreagit97 commented Sep 11, 2023 •

edited

kennylindberg commented Sep 11, 2023

Andreagit97 commented Sep 11, 2023 •

edited

kennylindberg commented Sep 11, 2023

FedeDP commented Oct 25, 2023

kennylindberg commented Oct 25, 2023

FedeDP commented Oct 25, 2023

kennylindberg commented Oct 25, 2023

FedeDP commented Oct 25, 2023

Andreagit97 commented Oct 25, 2023

kennylindberg commented Oct 26, 2023

[Modern eBPF] libpman: tracing program type is not supported #2792

[Modern eBPF] libpman: tracing program type is not supported #2792

Comments

kennylindberg commented Sep 8, 2023 • edited

Andreagit97 commented Sep 8, 2023

kennylindberg commented Sep 11, 2023 • edited

Andreagit97 commented Sep 11, 2023 • edited

kennylindberg commented Sep 11, 2023

Andreagit97 commented Sep 11, 2023 • edited

kennylindberg commented Sep 11, 2023

FedeDP commented Oct 25, 2023

kennylindberg commented Oct 25, 2023

FedeDP commented Oct 25, 2023

kennylindberg commented Oct 25, 2023

FedeDP commented Oct 25, 2023

Andreagit97 commented Oct 25, 2023

kennylindberg commented Oct 26, 2023

kennylindberg commented Sep 8, 2023 •

edited

kennylindberg commented Sep 11, 2023 •

edited

Andreagit97 commented Sep 11, 2023 •

edited

Andreagit97 commented Sep 11, 2023 •

edited