kstack can't resolve symbols when using snap

[ivankelly]

bpftrace --info

~# bpftrace --info
System
  OS: Linux 5.4.0-1021-gcp #21~18.04.1-Ubuntu SMP Mon Jul 13 03:31:28 UTC 2020
  Arch: x86_64

Build
  version: v0.10.0-193-g2420
  LLVM: 7
  foreach_sym: no
  unsafe uprobe: no
  bfd: yes
  bpf_attach_kfunc: no
  bcc_usdt_addsem: no
  libbpf: no
  libbpf btf dump: no
  libbpf btf dump type decl: no

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: no
  probe_read_user_str: no
  probe_read_kernel: no
  probe_read_kernel_str: no
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: yes

Kernel features
  Instruction limit: 1000000
  Loop support: yes
  btf: no

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: no

What reproduces the bug?

#!/usr/bin/env bpftrace

kprobe:md_make_request {
    @[kstack(8)] = count();
}

Output is missing symbol resolution.

~# bpftrace kstacks.bt                                                                                                                                                            
Attaching 1 probe...
^C

@[
    0xffffffffa028c9e1
    0xffffffff9fedbba6
    0xffffffff9fdc286d
    0xffffffff9fda6ad4
    0xffffffff9fc26fdb
    0xffffffff9fc1d93b
    0xffffffff9fc1ed94
    0xffffffff9fd9604e
]: 1
@[
    0xffffffffa028c9e1
    0xffffffff9fedbba6
    0xffffffff9fd19452
    0xffffffff9fd19b13
    0xffffffff9fde970f
    0xffffffff9fdea995
    0xffffffff9fdef348
    0xffffffff9fac17f1
]: 2
...

When using bpftrace from the docker image (https://github.com/iovisor/bpftrace/blob/master/INSTALL.md#copying-bpftrace-binary-from-docker) it resolves as expected.

~# ./bpftrace kstacks.bt                                                                                                                                                          
Attaching 1 probe...
^C

@[
    md_make_request+1
    submit_bio+70
    submit_bh_wbc+386
    submit_bh+19
    journal_submit_commit_record.part.22+431
    jbd2_journal_commit_transaction+4693
    kjournald2+200
    kthread+289
]: 2
@[
    md_make_request+1
    submit_bio+70
    submit_bio_wait+89
    blkdev_issue_flush+142
    ext4_sync_file+838
    vfs_fsync_range+72
    do_fsync+61
    __x64_sys_fdatasync+23
]: 2
@[
    md_make_request+1
    submit_bio+70
    submit_bh_wbc+386
    write_dirty_buffer+72
    flush_descriptor.part.13+66
    jbd2_journal_write_revoke_records+619
    jbd2_journal_commit_transaction+1189
    kjournald2+200
]: 2
...

[danobi]

So bpftrace uses bcc to resolve ksym addrs. bcc gets all the information from /proc/kallsyms. I'm not too familiar with snap but I wonder if it's preventing bpftrace from reading /proc/kallsyms

[fbs]

Maybe we should stop listing the snap build. Our docker build works well, if we make the binary a bit easier to access (curl instead of docker) it should be a good replacement. This is not the first time an issue like this popped up. I think snaps idea of confinement just conflicts too much with a tool like this.

[mmisono]

Did you install bpftrace with snap install --devmode? (cf. https://github.com/iovisor/bpftrace/blob/master/INSTALL.md#ubuntu-packages)

[ivankelly]

@mmisono no, didn't specify devmode. I followed the instructions on https://snapcraft.io/bpftrace, which does not have it.

[mmisono]

I don't know much about snap, but apparently --devmode is needed (#662). Maybe the instruction should be updated. @ColinIanKing