This PMDA extracts live performance data from extended BPF (Berkeley Packet Filter) in-kernel programs by using the BCC (BPF Compiler Collection) Python frontend.
eBPF was described by Ingo Molnár as:
One of the more interesting features in this cycle is the ability to attach eBPF programs (user-defined, sandboxed bytecode executed by the kernel) to kprobes. This allows user-defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively.
BCC has made creating new eBPF programs easier, and the BCC project offers a wide variety of such tools (https://github.com/iovisor/bcc/tree/master/tools). However, basically all these programs are individual, disjoint utilities that are mostly meant for interactive use. This is not a suitable approach to collect, monitor and analyze performance data in larger environments where there are hundreds, if not thousands, installations and where human intervention is unfeasible at best.
This PMDA loads and acts as a bridge for any number of configured, separate BCC PMDA Python modules running eBPF programs. Each module consists of an eBPF program (e.g. sysfork.bpf) and a corresponding Python file (e.g. sysfork.python).
- A Linux kernel with eBPF support (RHEL 7.6+ or kernel 4.7+)
- BCC (https://github.com/iovisor/bcc) and BCC Python module (0.5+)
There are several modules available to trace system and application behavior from different angles, including number of (failed) system calls (per pid), any available kernel tracepoints, USDT/dtrace/stap probes, uprobes, kprobes, and application internals (like new threads, objects, method calls).
Modules can be enabled and configured in the bcc.conf file. The
modules
setting of the [pmda]
section lists the enabled modules:
[pmda]
modules = runqlat,tcpperpid
Each module has their own section with individual module-related
configuration, for example the tcpperpid
module:
[tcpperpid]
module = tcpperpid
cluster = 11
dport = 80,443
In this case the tcpperpid
module traces only TCP sessions with a remote
port of 80 or 443.
For many modules process(es) to monitor can be defined as a list of names,
PIDs, or regular expressions. PMDA-wide parameter process_refresh
can be
used to monitor newly created processes matching the list of processes.
yum install pcp pcp-pmda-bcc
systemctl enable --now pmcd
cd $PCP_PMDAS_DIR/bcc
vi bcc.conf
./Install
On most systems $PCP_PMDAS_DIR
is /var/lib/pcp/pmdas
.
cd $PCP_PMDAS_DIR/bcc
./Remove
$ pminfo -f bcc.disk.all.latency
bcc.disk.all.latency
inst [0 or "0-1"] value 0
inst [1 or "2-3"] value 0
inst [2 or "4-7"] value 0
inst [3 or "8-15"] value 0
inst [4 or "16-31"] value 0
inst [5 or "32-63"] value 1956
inst [6 or "64-127"] value 5715
inst [7 or "128-255"] value 6321
inst [8 or "256-511"] value 3416
All BCC PMDA modules can be visualized in Grafana, using the grafana-pcp plugin.
- Check logfile for errors:
$PCP_LOG_DIR/pmcd/bcc.log
- Check if the bcc Python module is installed:
/usr/bin/pcp python -c 'import bcc'
- Check if BCC PMDA metrics are registered:
pminfo | grep bcc
- Check if BCC PMDA metrics contain data:
pminfo -f bcc.proc.sysfork
(if thesysfork
module is enabled) - Check if BCC is working on the system:
yum install bcc-tools && /usr/share/bcc/tools/runqlat
- PMDA: GPLv2
- eBPF/BCC programs: Apache License 2.0