high kernel memory usage when using ebpf plugin

[ilyam8]

Bug report summary

Used RAM drops on my server from 1040 to 923 right after i kill ebpf.plugin process (117 MB).

Topic on our forum: https://community.netdata.cloud/t/netdata-consuming-high-ram-amount/1005.

That is @haydenseitz server: 3072 => 2582 (490 MB).

On @vlvkobal server it was 200 MB.

---

The tricky part that it is not memory used by the ebpf.plugin process. The process itself uses several megabytes of RAM.

OS / Environment

Linux pc 5.10.23-1-MANJARO #1 SMP PREEMPT Thu Mar 11 18:47:18 UTC 2021 x86_64 GNU/Linux

/etc/os-release [Click]

/etc/arch-release:Manjaro Linux
/etc/lsb-release:DISTRIB_ID=ManjaroLinux
/etc/lsb-release:DISTRIB_RELEASE=21.0.1
/etc/lsb-release:DISTRIB_CODENAME=Ornara
/etc/lsb-release:DISTRIB_DESCRIPTION="Manjaro Linux"
/etc/manjaro-release:Manjaro Linux
/etc/os-release:NAME="Manjaro Linux"
/etc/os-release:ID=manjaro
/etc/os-release:ID_LIKE=arch
/etc/os-release:BUILD_ID=rolling
/etc/os-release:PRETTY_NAME="Manjaro Linux"
/etc/os-release:ANSI_COLOR="32;1;24;144;200"
/etc/os-release:HOME_URL="https://manjaro.org/"
/etc/os-release:DOCUMENTATION_URL="https://wiki.manjaro.org/"
/etc/os-release:SUPPORT_URL="https://manjaro.org/"
/etc/os-release:BUG_REPORT_URL="https://bugs.manjaro.org/"
/etc/os-release:LOGO=manjarolinux

Netdata version

Version: netdata v1.30.0-18-g8bc32ad6

buildinfo [Click]

Configure options:  '--prefix=/opt/netdata/usr' '--sysconfdir=/opt/netdata/etc' '--localstatedir=/opt/netdata/var' '--libexecdir=/opt/netdata/usr/libexec' '--libdir=/opt/netdata/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--disable-cloud' 'CFLAGS=-O2' 'LDFLAGS='
Features:
    dbengine:                YES
    Native HTTPS:            YES
    Netdata Cloud:           NO (by user request)
    TLS Host Verification:   YES
Libraries:
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  YES
    libcrypto:               YES
    libm:                    YES
    LWS:                     YES shared-lib
    mosquitto:               YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    YES
    IPMI:                    NO
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES

Installation method

From source.

Component Name

ebpf

Steps To Reproduce

1. have ebpf.plugin running, check used ram.
2. kill ebpf.plugin (sudo kill $(pidof ebpf.plugin)), check used ram.

Expected behavior

having ebpf.plugin running doesn't contribute a lot into the kernel memory usage. If that is expected, this should be documented.

[netdata-community-bot]

This issue has been mentioned on the Netdata Community. There might be relevant details there:

https://community.netdata.cloud/t/netdata-consuming-high-ram-amount/1005/24

[thiagoftsm]

This has direct relationship with the issue #10861.
I will make a PR to close both.

[ilyam8]

Ok, ~significant increase of kernel memory usage when using ebpf.plugin is expected.

From what i've seens it can be +100MB - +500MB. @thiagoftsm if you can explain why it is +100 on some systems and +500 on others, please do.

This PR reduces memory usage by 3 times.

In additions we need to mention it in the documentation @thiagoftsm.

[odyslam]

My proposal is to mention this on the docs, perhaps create a whole section about it and then create an FAQ for it. cc @joelhans

[joelhans]

There's already a Performance section in the eBPF doc, which we can appropriate for this information, especially given that the text that's already there is probably not correct, given this whole issue. We can replace the entire section with something like this:

Performance

eBPF monitoring is complex and produces a large volume of metrics. We've discovered scenarios where the eBPF plugin significantly increases kernel memory usage by several hundred MB.

If your node is experiencing high memory usage and there is no obvious culprit to be found in the apps.mem chart, consider testing for high kernel memory usage by disabling eBPF monitoring. Next, restart Netdata with sudo systemctl restart netdata to see if system memory usage (see the system.ram chart) has dropped significantly.

Beginning with v1.30.1-xxxxx-nightly, kernel memory usage is configurable via the pid table size setting in ebpf.conf.

---

Happy to hear feedback to make it better—I'm just trying to condense everything I'm reading in issues/forum posts without much fundamental understanding.

[ilyam8]

@joelhans We added Percpu dimension to the mem.kernel chart (Memory->kernel->Memory Used by Kernel).

(kill $(pidof ebpf.plugin))

[thiagoftsm]

There's already a Performance section in the eBPF doc, which we can appropriate for this information, especially given that the text that's already there is probably not correct, given this whole issue. We can replace the entire section with something like this:

Performance

eBPF monitoring is complex and produces a large volume of metrics. We've discovered scenarios where the eBPF plugin significantly increases kernel memory usage by several hundred MB.

If your node is experiencing high memory usage and there is no obvious culprit to be found in the apps.mem chart, consider testing for high kernel memory usage by disabling eBPF monitoring. Next, restart Netdata with sudo systemctl restart netdata to see if system memory usage (see the system.ram chart) has dropped significantly.

Beginning with v1.30.1-xxxxx-nightly, kernel memory usage is configurable via the pid table size setting in ebpf.conf.

Happy to hear feedback to make it better—I'm just trying to condense everything I'm reading in issues/forum posts without much fundamental understanding.

I liked your suggestion, I will update the documentation and you can review them in #10962