README.md

BCC eBPF PMDA

This PMDA extracts live performance data from extended BPF (Berkeley Packet Filter) in-kernel programs by using the BCC (BPF Compiler Collection) Python frontend.

eBPF was described by Ingo Molnár as:

One of the more interesting features in this cycle is the ability to attach eBPF programs (user-defined, sandboxed bytecode executed by the kernel) to kprobes. This allows user-defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively.

BCC has made creating new eBPF programs easier, and the BCC project offers a wide variety of such tools (https://github.com/iovisor/bcc/tree/master/tools). However, basically all these programs are individual, disjoint utilities that are mostly meant for interactive use. This is not a suitable approach to collect, monitor and analyze performance data in larger environments where there are hundreds, if not thousands, installations and where human intervention is unfeasible at best.

This PMDA loads and acts as a bridge for any number of configured, separate BCC PMDA Python modules running eBPF programs. Each module consists of an eBPF program (e.g. sysfork.bpf) and a corresponding Python file (e.g. sysfork.python).

Requirements

• A Linux kernel with eBPF support (RHEL 7.6+ or kernel 4.7+)
• BCC (https://github.com/iovisor/bcc) and BCC Python module (0.5+)

Configuration

There are several modules available to trace system and application behavior from different angles, including number of (failed) system calls (per pid), any available kernel tracepoints, USDT/dtrace/stap probes, uprobes, kprobes, and application internals (like new threads, objects, method calls).

Modules can be enabled and configured in the bcc.conf file. The modules setting of the [pmda] section lists the enabled modules:

[pmda]
modules = runqlat,tcpperpid

Each module has their own section with individual module-related configuration, for example the tcpperpid module:

[tcpperpid]
module = tcpperpid
cluster = 11
dport = 80,443

In this case the tcpperpid module traces only TCP sessions with a remote port of 80 or 443.

For many modules process(es) to monitor can be defined as a list of names, PIDs, or regular expressions. PMDA-wide parameter process_refresh can be used to monitor newly created processes matching the list of processes.

Installation

yum install pcp pcp-pmda-bcc
systemctl enable --now pmcd
cd $PCP_PMDAS_DIR/bcc
vi bcc.conf
./Install

On most systems $PCP_PMDAS_DIR is /var/lib/pcp/pmdas.

Uninstalling the PMDA

cd $PCP_PMDAS_DIR/bcc
./Remove

Frontend

CLI

$ pminfo -f bcc.disk.all.latency                                

bcc.disk.all.latency
    inst [0 or "0-1"] value 0
    inst [1 or "2-3"] value 0
    inst [2 or "4-7"] value 0
    inst [3 or "8-15"] value 0
    inst [4 or "16-31"] value 0
    inst [5 or "32-63"] value 1956
    inst [6 or "64-127"] value 5715
    inst [7 or "128-255"] value 6321
    inst [8 or "256-511"] value 3416

Web Frontend

All BCC PMDA modules can be visualized in Grafana, using the grafana-pcp plugin.

Troubleshooting

• Check logfile for errors: $PCP_LOG_DIR/pmcd/bcc.log
• Check if the bcc Python module is installed: /usr/bin/pcp python -c 'import bcc'
• Check if BCC PMDA metrics are registered: pminfo | grep bcc
• Check if BCC PMDA metrics contain data: pminfo -f bcc.proc.sysfork (if the sysfork module is enabled)
• Check if BCC is working on the system: yum install bcc-tools && /usr/share/bcc/tools/runqlat

License

• PMDA: GPLv2
• eBPF/BCC programs: Apache License 2.0