tracing: tests for USDT tracepoints

[0xB10C]

#22006 added the first three USDT based tracepoints to Bitcoin Core. To provide a semi-stable tracepoint API the tracepoints need test coverage. The tracepoints can be tested in the functional tests using the Python wrapper of BCC.

Before adding more tracepoints, the existing three tracepoints from #22006 should be tested.

Notes:

1. We currently only support the tracepoints on Linux. The tests should be skipped on other operating systems.
2. Hooking into the tracepoints via the Linux kernel requires special privileges. Since kernel version 5.8. (Aug. 2020) the CAP_BPF can be used. On older kernel version the overloaded catch-all capability CAP_SYS_ADMIN is required. Functional tests shouldn't require CAP_SYS_ADMIN as that essentially means running the test suite with root privileges.
3. The tests require the BCC Python library. This should be an optional dependency. Tests should be skipped if the dependency isn't present.

The connect_block tracepoint can be tested by mining blocks with transactions and checking that the tracepoint passes the correct data. The net inbound_message and outbound_message tracepoints can be tested by checking the traffic between two nodes.

[0xB10C]

Note so I don't forget about this:

By chance I came across:

bitcoin/ci/test/04_install.sh

Lines 28 to 30 in c006ab2

	if [[ $BITCOIN_CONFIG = *--with-sanitizers=*address* ]]; then # If ran with (ASan + LSan), Docker needs access to ptrace (https://github.com/google/sanitizers/issues/764)
	DOCKER_ADMIN="--cap-add SYS_PTRACE"
	fi

Which adds the CAP_SYS_PTRACE capability to the docker container being started. If we'd run the functional tests in docker we use --cap-add BPF (assuming a 5.8+ kernel).

docker docs: Runtime privilege and Linux capabilities

[0xB10C]

Next to CAP_BPF, CAP_PERFMON is likely also required.

---

I've started to work on how to run tracepoint tests in the functional test suite:

1. on developer machines without requiring to run the functional test framework as root.
2. in the CI (particularly our current cirrus CI setup)

[0xB10C]

I've focused on getting a proof-of-concept (PoC) running on CirrussCI for now. I solved a few of the bigger issues, but I'm at a point where I think the CI containers don't have enough permissions to attach to the USDT tracepoints. I've learned a bit about the CI and want to document my process here for future re-attempts.

My PoC branch is https://github.com/0xB10C/bitcoin/commits/2022-01-tracepoint-ci-poc

tl;dr: I think we are not able to set the required Linux capabilities to test USDT tracepoints on CirrusCi: Unable to set capabilities [--caps=CAP_SYS_ADMIN+ep].

first commit: My approach for the PoC was to drop all tasks besides the [lint] task for less CI load. No need to re-test the same Bitcoin Core code over and over. I've then added a CI task specifically for the USDT tracing tests, disabling e.g. the unit tests and existing functional tests for faster CI feedback. I checked that the newly added CI task builds the (or uses the cached) depends including the systemtap depends added in #23724 and that use usdt is true the configure stage.

second commit: The next step was to set the prerequisites for the USDT functional tests up. This test should only be run when a) the tracepoints are compiled and b) the Python bcc (iovisor/bcc) is installed. I've added the relevant checks to test_framework.py and a ENABLE_USDT_TRACEPOINTS flag in the config.ini.in. Additionally, I've added bpfcc-tools to the installed packages for the bcc Python package.

third commit: Then, I've added an interface_usdt.py PoC functional test based on the log_raw_p2p_msgs.py example. It checks that at least one inbound and one outbound P2P message is received over the USDT interface.

fourth commit: The BPF bytecode compilation from the provided C code failed. The kernel headers are missing. It took me a while to remember that containers use the host's kernel. This means the relevant kernel headers can't be found/installed in the Ubuntu jammy container. According to the Cirrus CI documentation, the container instances run on a GKE cluster of compute-optimized instances running in Google Cloud. These run Google's Container-Optimized OS COS with their own kernel. Specifically, (I think) the cos-89-16108-534-8 release with the COS-5.4.144 Linux kernel. It's possible to retrieve the kernel source and run the make oldconfig and make prepare steps using the kernel configuration in /proc/config.gz. This is, for example, done in iovisor/kubectl-trace (Kubernetes tracing) for COS too. See fetch-linux-headers.sh. Fetching the kernel headers and the make steps take a few minutes. However I think this could be cached and only needs to be done one time when Cirrus CI upgrades their host OS to a newer version. We need to set the BCC_KERNEL_SOURCE env var to the generated kernel headers. The BPF bytecode compilation succeeds now.

fifth commit: We lack the required permissions to open the BPF maps. I assumed this could be solved by adding the required capabilities (to docker). It wasn't clear to me that DOCKER_EXEC does, in fact, not run docker exec <args>. The CI is run in the CirrusCI container itself at the moment. See DANGER_RUN_CI_ON_HOST. (Maybe just EXEC would be a better name). So adding --cap-add=CAP_X to DOCKER_ADMIN (see #23296 (comment)) did not work. I've opted to use capsh to set the required capabilities. With the host's 5.4 Linux kernel, we can't use CAP_BPF and CAP_PERFMON as they were introduced in 5.8. We have to fall back to CAP_SYS_ADMIN. However, this fails with Unable to set capabilities [--caps=CAP_SYS_ADMIN+ep]. Later, I discovered cirruslabs/cirrus-ci-docs#654, where MarcoFalke already asked about this for wine.

Here's the failed CI task: https://cirrus-ci.com/task/6305351615119360

---

Maybe @MarcoFalke has further ideas? Or @laanwj? I think having automatically run tests for the tracepoints is important. I don't think running these on a different CI is a good solution though.

I'll be focusing on a PoC for local development machines for now. It should be possible to run these with sudo capsh ..., dropping privilege while only keeping the required capabilities. While these functional tests would be skipped on most development machines, they could at least be run to e.g. test release candidates and on PRs adding new tracepoints (with tests).

[maflcko]

Linux containers in Cirrus CI already run in docker (I don't know which permissions they have turned on, though), so it is not possible to start docker again.

Maybe just EXEC would be a better name

Yeah, or CI_EXEC

While it may be possible to specify the flags for the one task running outside the Cirrus infrastructure (https://cirrus-ci.com/task/5786179323822080?logs=ci#L21), I think it may be preferable to not rely on that to make the tasks easier to run locally inside docker. I use this often with DANGER_RUN_CI_ON_HOST.

It is possible that arm, freebsd or macos run on actual vms. If any of them support usdt, you could try writing or adjusting the task for them.

[0xB10C]

Linux containers in Cirrus CI already run in docker (I don't know which permissions they have turned on, though), so it is not possible to start docker again.

Makes sense. These capabilities are enabled (via capsh --print):

cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=eip

Maybe just EXEC would be a better name

Yeah, or CI_EXEC

Agree, that's even better.

It is possible that arm, freebsd or macos run on actual vms. If any of them support usdt, you could try writing or adjusting the task for them.

Good point. We tested the tracepoints in #23724 on arm64. CirrusCi runs their ARM tasks on AWS. I'll try to adjust the arm task, though I think they'll run the Kubernetes containers with the same permissions/capabilites.

[0xB10C]

Adding tests for all existing tracepoints in #24358. These are currently skipped in the CI (and probably most other test suite runs too).

[0xB10C]

GitHub seems to have decided that "partly fixes #23296" means it should close this issue. I still want to see if we can get the tests running in the CI. I'll extract the "running the tests in the CI" part into a new issue.

[0xB10C]

For reference, @laanwj noted in #24358 (comment) how he was able to run the tracepoint tests with a non-root user on his system. This is important developer documentation. I'll try to reproduce this on my system and will add it to either the docs, write a blog post about it, or both.