On Ubuntu 22.04 (dev build), bpftrace installs without errors but doesn't work

[QuantumDonDoerner]

What reproduces the bug?

My system is a dev build of Ubuntu 22.04

#cat /etc/os-release
PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

#uname -a
Linux dd-ubuntu 5.13.0-1010-kvm #11+22.04.1-Ubuntu SMP Fri Jan 14 14:04:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

bpftrace installs OK per README.md:

• Change into tools directory: incant "cd .../bpftrace/tools # wherever you put it"
• Invoke vfscount: incant "bpftrace vfscount.bt". I expect output similar to .../bpftrace/tools/vfscount_example.txt. Instead, I get:

#bpftrace vfscount.bt
ERROR: Failed to compile: Could not read symbols from /sys/kernel/debug/tracing/available_filter_functions: No such file or directory

There are errors that match this, having to do with not mounting the debug filesystem, or with mounting it in an unexpected location. I checked this; the file system is present and mounted at the expected place:

#cat /proc/mounts | grep debugfs
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0

Then I checked the structure of the filesystem. The requisite directory "tracing" is there, but the structure below that is not as expected.

#cd /sys/kernel/debug
#ls
acpi block clear_warn_once device_component dma_buf energy_model hid sched swiotlb virtio-ports
bdi btt clk devices_deferred dynamic_debug fault_around_bytes ras sleep_time tracing x86
#ls tracing
README buffer_size_kb error_log options saved_cmdlines_size set_event_pid trace_clock trace_pipe uprobe_events
available_events buffer_total_size_kb events per_cpu saved_tgids synthetic_events trace_marker tracing_cpumask uprobe_profile
available_tracers current_tracer free_buffer printk_formats set_event timestamp_mode trace_marker_raw tracing_on
buffer_percent dynamic_events instances saved_cmdlines set_event_notrace_pid trace trace_options tracing_thresh

From which I conclude I am either missing some required package, or that the structure of debugfs has changed.

bpftrace --info

#bpftrace --info
System
OS: Linux 5.13.0-1010-kvm #11+22.04.1-Ubuntu SMP Fri Jan 14 14:04:27 UTC 2022
Arch: x86_64

Build
version: v0.14.0
LLVM: 11.1.0
ORC: v2
foreach_sym: yes
unsafe uprobe: no
bfd: no
bpf_attach_kfunc: yes
bcc_usdt_addsem: yes
bcc bpf_attach_uprobe refcount: yes
bcc library path resolution: yes
libbpf: yes
libbpf btf dump: yes
libbpf btf dump type decl: yes
libdw (DWARF support): no

Kernel helpers
probe_read: yes
probe_read_str: yes
probe_read_user: yes
probe_read_user_str: yes
probe_read_kernel: yes
probe_read_kernel_str: yes
get_current_cgroup_id: yes
send_signal: yes
override_return: no
get_boot_ns: yes
dpath: no

Kernel features
Instruction limit: 1000000
Loop support: yes
btf (depends on Build:libbpf): yes
map batch (depends on Build:libbpf): yes
uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes

Map types
hash: yes
percpu hash: yes
array: yes
percpu array: yes
stack_trace: yes
perf_event_array: yes

Probe types
kprobe: yes
tracepoint: yes
perf_event: yes
kfunc: no
iter:task: yes
iter:task_file: yes

Next steps

For my purposes, I will back down to 20.04 LTS, but if there are pending changes to the structure of debugfs, I wanted to provide a heads-up.

[QuantumDonDoerner]

Update: on Ubuntu 20.04 LTS, all works as documented.

[viktormalik]

Hi, /sys/kernel/debug/tracing/available_filter_functions is indeed necessary for bpftrace. It should be mounted as a part of tracefs (not debugfs). Could you try running cat /proc/mounts | grep tracefs?

I tried Ubuntu 22.04 from the official Vagrant box and the file's there. My tracefs mounts:

# cat /proc/mounts | grep tracefs
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/debug/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0

[zhihaop]

I had the same problem in Ubuntu 22.04, but it was solved by installing bpftrace-dbgsym.

[lxyscls]

ajor/bpftrace#50, I have meet such problem on Ubuntu 22.04@_@

[ajor]

@lxyscls I'm not sure if these issues are related, but for the ajor/bpftrace#50 one I've proposed a fix here: #2292

[kobi-ca]

I had the same problem in Ubuntu 22.04, but it was solved by installing bpftrace-dbgsym.

@zhihaop - how did you install bpftrace-dbgsym on ubuntu 22.04? I'm unable to find it in the repos.

[zhihaop]

I had the same problem in Ubuntu 22.04, but it was solved by installing bpftrace-dbgsym.

@zhihaop - how did you install bpftrace-dbgsym on ubuntu 22.04? I'm unable to find it in the repos.

@kobi-ca You can install bpftrace-dbgsym after setting up the debugging symbol packages, and this link may be helpful.

In short, the following script may help to install bpftrace-dbgsym.

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt install ubuntu-dbgsym-keyring
sudo apt update
sudo apt install bpftrace-dbgsym

[kobi-ca]

thanks @zhihaop - I'll give it a shot!

[danobi]

FWIW you will no longer need debug syms (really just symbols) after the next release. See #2264

[kobi-ca]

thanks @zhihaop - I'll give it a shot!

@zhihaop - thank you again, it worked great. I can run opensnoop.bt

[viktormalik]

I believe that this is now resolved and can be closed.

[inliquid]

I have same issue on latest Ubuntu 22.04 @ WSL2

# opensnoop.bt
Attaching 6 probes...
ERROR: Could not resolve symbol: /proc/self/exe:BEGIN_trigger

# bpftrace --version
bpftrace v0.14.0

# apt list --installed "*bpf*"
Listing... Done
bpftrace/jammy,now 0.14.0-1 amd64 [installed]
libbpf-dev/jammy-updates,jammy-security,now 1:0.5.0-1ubuntu22.04.1 amd64 [installed]
libbpf0/jammy-updates,jammy-security,now 1:0.5.0-1ubuntu22.04.1 amd64 [installed,automatic]
libbpfcc-dev/jammy,now 0.18.0+ds-2 amd64 [installed]
libbpfcc/jammy,now 0.18.0+ds-2 amd64 [installed,automatic]
libpfm4/jammy-updates,now 4.11.1+git32-gd0b85fb-1ubuntu0.1 amd64 [installed,automatic]

[danobi]

Hi @inliquid , did you try following the steps outlined in #2168 (comment) ?

Also starting in bpftrace v0.16 it is no longer necessary to install bpftrace with symbols.

[inliquid]

Thanks @danobi! So would it be correct to say that that preferred way is to uninstall Ubuntu's v0.14.0 of bpftrace and build it from source instead?