Improve PID monitoring (step 2) #13530

thiagoftsm · 2022-08-18T02:09:43Z

Summary

This PR is reducing CPU and memory usage for eBPF.plugin.
It is also disabling integration by default integration for legacy kernels, and some kernels that are LTS, but they do not have trampoline.
This PR is also bringing co-re codes for more threads.

Test Plan

Compile this branch.
Start netdata using the default configuration for ebpf.plugin and verify that is your kernel was compiled with CONFIG_DEBUG_INFO_BTF (zgrep BTF /proc/config.gz) the integration with apps and cgroup will be enabled by default, on the other hand, you should not have them enabled.
Verify that you do not have libbpf errors (grep libbpf /var/log/netdata/error.log) for threads different of socket. For socket this is expected after kernel 5.19 to be released. This will be fixed in next PR.
Now stop netdata and enabling all threads, you should have charts normally.
Veriify that memory used by ebpf.programs were reduced, they should have 10992 instead the traditional 32768 by default when you run bpftool map show.

Additional Information

This PR was tested on:

Linux Distribution	kernel version	Threads	bpftool
Slackware Current	5.19.2	slackware_5_19_threads.txt	slackware_5_19_bpftool.txt
Arch Linux	5.19.2	arch_5_19_threads.txt	arch_5_19_bpftool.txt
Ubuntu 22.04	5.15.0-33-generic	ubuntu_5_15_threads.txt	ubuntu_5_15_bpftool.txt
Alma 9	5.14.0-70.17.1.el9_0.x86_64	alma_5_14_threads.txt	--
Debian 11	5.10.0-16-amd64	debian_5_10_threads.txt	debian_5_10_bpftool.txt
Alma 8.6	4.18.0-372.19.1.el8_6	alma_4_18_threads.txt	alma8_4_18_bpftool.txt
Ubuntu 18.04	4.15.0-180	ubuntu_4_15_threads.txt	ubuntu_4_15_bpftool.txt

For users: How does this change affect me?

Describe the PR affects users: - Which area of Netdata is affected by the change? ebpf.plugin - Can they see the change or is it an under the hood? If they can see it, where? The plugin will use less resources from host. - How is the user impacted by the change? A better performance from plugin; - What are there any benefits of the change? plugin improvement.

…o_string

collectors/ebpf.plugin/README.md

collectors/ebpf.plugin/ebpf.c

collectors/ebpf.plugin/README.md

vlvkobal · 2022-08-22T11:28:04Z

==30926== Thread 3 EBPF SOCKET:
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x4847D18: strlen (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x17C14E: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x17BC9A: find_prev_line (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C188: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x484846C: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x17C1B2: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x484847F: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x17C1B2: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x4848484: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x17C1B2: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x484846C: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x17C229: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x484847F: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x17C229: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x4848484: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x17C229: fixup_verifier_log (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BB65: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==    by 0x12F2C7: ebpf_socket_thread (ebpf_socket.c:3944)
==30926==    by 0x15860F: thread_start (threads.c:185)
==30926==    by 0x4E0D78C: start_thread (pthread_create.c:442)
==30926==
--30926-- WARNING: unhandled eBPF command 28
--30926-- WARNING: unhandled eBPF command 28
==30926== Conditional jump or move depends on uninitialised value(s)
==30926==    at 0x4847D18: strlen (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==30926==    by 0x4DE25C7: __vfprintf_internal (vfprintf-process-arg.c:397)
==30926==    by 0x4DE2C74: buffered_vfprintf (vfprintf-internal.c:1748)
==30926==    by 0x16B030: __base_pr (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x16B11C: libbpf_print (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BC17: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x12B019: socket_bpf__load (socket.skel.h:127)
==30926==    by 0x12B019: ebpf_socket_load_and_attach (ebpf_socket.c:404)
==30926==    by 0x12B13B: ebpf_socket_load_bpf (ebpf_socket.c:3890)
==30926==
==30926== Syscall param write(buf) points to uninitialised byte(s)
==30926==    at 0x4E7DECF: __libc_write (write.c:26)
==30926==    by 0x4E7DECF: write (write.c:24)
==30926==    by 0x4E052CC: _IO_file_write@@GLIBC_2.2.5 (fileops.c:1180)
==30926==    by 0x4E0466F: new_do_write (fileops.c:448)
==30926==    by 0x4E05A00: _IO_new_file_xsputn (fileops.c:1254)
==30926==    by 0x4E05A00: _IO_file_xsputn@@GLIBC_2.2.5 (fileops.c:1196)
==30926==    by 0x4DE2CE2: buffered_vfprintf (vfprintf-internal.c:1766)
==30926==    by 0x16B030: __base_pr (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x16B11C: libbpf_print (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17BC17: bpf_object_load_prog (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17C5C1: bpf_object__load_progs (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17DEF1: bpf_object_load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x17E1CB: bpf_object__load (in /home/vlad/netdata/ebpf.plugin)
==30926==    by 0x189AE3: bpf_object__load_skeleton (in /home/vlad/netdata/ebpf.plugin)
==30926==  Address 0x8cb1855 is on thread 3's stack
==30926==  in frame #4, created by buffered_vfprintf (vfprintf-internal.c:1716)

thiagoftsm · 2022-08-22T19:28:17Z

@vlvkobal about these errors, they have relationship with internal libbpf that is linked statically with the threads and the co-re code generated by the compiler. For we properly fix this we will either need to change the compiler output in ebpf-co-re or create a patch in libbpf.

A new libbpf was released today, a library that is bringing huge changes, I will investigate this better since libbpf code until we use use it here. I prefer to address this issue in another PR.

thiagoftsm added 6 commits August 17, 2022 22:34

ebpf_pid_step2: Update documentation

1309f3e

ebpf_pid_step2: Change default returned by ebpf_convert_collect_pid_t…

09346e6

…o_string

ebpf_pid_step2: Update configuration file

97507c8

ebpf_pid_step2: Change the way load flag is manipulated

45ddcd4

ebpf_pid_step2: Add FD CO-RE

a25ce2a

ebpf_pid_step2: Add VFS CO-RE

3e5063f

thiagoftsm requested review from DShreve2, vkalintiris and vlvkobal as code owners August 18, 2022 02:09

thiagoftsm marked this pull request as draft August 18, 2022 02:09

github-actions bot added area/collectors Everything related to data collection area/docs collectors/ebpf labels Aug 18, 2022

thiagoftsm added 5 commits August 18, 2022 03:29

ebpf_pid_step2: Fix compilation for old kernels

e45c906

ebpf_pid_step2: Fix different issues

d4d7dc9

update_4_threads: Update co-re version and update cachestat

7eff891

update_4_threads: Update dc code adding tracers and function calls

7f79fb8

update_4_threads: Update fd code adding tracers and function calls

af9504d

github-actions bot added the area/packaging Packaging and operating systems support label Aug 18, 2022

thiagoftsm added 11 commits August 19, 2022 01:33

update_4_threads: Fix for 2 functions

b202bd4

update_4_threads: Update shm code adding tracers and function calls

09f413f

update_4_threads: Add missing function to threads

f7da242

update_4_threads: Disable trampoline for FD to avoid error

1075032

update_4_threads: Update swap code adding tracers and function calls

2c36a29

update_4_threads: Add adjust size for SHM

4687141

update_4_threads: Update vfs code adding tracers and function calls

3a08afb

update_4_threads: Update ebpf version

753d735

update_4_threads: Add missing call to adjust_apps_cgroup

debf70f

update_4_threads: Fix codacy warnings

3be6f16

ebpf_pid_step2: Change ebpf.d.conf enabling fd and vfs

2bd228c

thiagoftsm marked this pull request as ready for review August 21, 2022 18:57

thiagoftsm requested a review from Ferroin as a code owner August 21, 2022 18:57

thiagoftsm requested review from underhood and removed request for vkalintiris August 21, 2022 18:57

vlvkobal reviewed Aug 22, 2022

View reviewed changes

collectors/ebpf.plugin/README.md Outdated Show resolved Hide resolved

collectors/ebpf.plugin/README.md Outdated Show resolved Hide resolved

collectors/ebpf.plugin/ebpf.c Outdated Show resolved Hide resolved

collectors/ebpf.plugin/README.md Outdated Show resolved Hide resolved

thiagoftsm added 4 commits August 22, 2022 14:34

ebpf_pid_step2: Update README.md

7b3c44f

ebpf_pid_step2: Fix typo

084d24b

ebpf_pid_step2: Update ebpf_set_load_mode algorithm

3507b25

ebpf_pid_step2: Add suggestions received from documentation team

ab88ca4

thiagoftsm requested a review from vlvkobal August 22, 2022 19:28

vlvkobal approved these changes Aug 23, 2022

View reviewed changes

underhood approved these changes Aug 23, 2022

View reviewed changes

thiagoftsm merged commit 575986a into netdata:master Aug 23, 2022

thiagoftsm deleted the ebpf_pid_step2 branch August 23, 2022 12:45

thiagoftsm mentioned this pull request Sep 5, 2022

[Bug]: High %sys usage of eBPF plugin when all programs are turned off #13626

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve PID monitoring (step 2) #13530

Improve PID monitoring (step 2) #13530

thiagoftsm commented Aug 18, 2022 •

edited

vlvkobal commented Aug 22, 2022

thiagoftsm commented Aug 22, 2022

Improve PID monitoring (step 2) #13530

Improve PID monitoring (step 2) #13530

Conversation

thiagoftsm commented Aug 18, 2022 • edited

Summary

Test Plan

Additional Information

vlvkobal commented Aug 22, 2022

thiagoftsm commented Aug 22, 2022

thiagoftsm commented Aug 18, 2022 •

edited