bpf_printk doesn't create data on trace_pipe

[andreybleme]

My minimal app executes correctly:

But when I cat the "/sys/kernel/debug/tracing/trace_pipe" file it looks empty:

I'm running a Ubuntu 22.04 LTS docker container on privileged mode.

This is how I have manually mounted my debugfs:
mount -t debugfs none /sys/kernel/debug

Am I missing anything? Appreciate your help.

[anakryiko]

Not really, but I've had cases when something in kernel "broke" w.r.t. trace_pipe and BPF program stopped emitting anything into it. Can you try rebooting and trying ./minimal again before you do anything else with the system?

I also see

mount | rg debug
debugfs on /sys/kernel/debug type debugfs (rw,relatime,mode=755)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)

So maybe try mounting tracefs as well?

[andreybleme]

Thanks for replying @anakryiko!
After mounting tracefs mount -t tracefs nodev /sys/kernel/tracing, I see:

$ mount | rg debug
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)

And the BPF program still doesn't write into /sys/kernel/debug/tracing/trace_pipe.
Interesting that when I cat, tail, or try to open the trace_pipe file with vi, the process doesn't finish with the empty output as it does with regular empty files. It actually gets stuck:

The file stat is:

File: /sys/kernel/debug/tracing/trace_pipe
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: ah/10d	Inode: 12206       Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-05-23 23:07:10.643999999 +0000
Modify: 2022-05-23 23:07:10.643999999 +0000
Change: 2022-05-24 00:30:11.248272170 +0000
 Birth: -

All files from debugfs looking normal (permissions):

Already tried mounting debugfs and tracefs again, restarting the container etc... Any additional ideas?

[anakryiko]

I have no idea, tbh. If you are trying it from inside the container, have you tried reading that file from outside any container? Also make sure that you don't have any other process opening this file, because I think trace_pipe doesn't support multiple readers reading it simultaneously.

[andreybleme]

I see. I'm executing this by running "./minimal", and after I see the dots from the userspace application indicating the BPF program is running, I stop the processes (CTRL + C) and then run the cat command. Should I be checking for debug messages in a different way?

[anakryiko]

you can run cat in parallel, but even if you run afterwards you should still see previous output.

Do you have a chance to try this on some other system? It might be some (unknown to me) kernel config dependencies. Other than that, I'm not sure how to help you.

[andreybleme]

This was simply a mistake in the eBPF program. Having the if statement:

if (pid != my_pid)
        return 0;

With this initial declaration:

int my_pid = 0;

This causes the BPF program never to actually execute the bpf_printk. When I removed the if statement, the cat command worked.

Thanks again for your time @anakryiko !

[Arvintian]

This may causes by the pid namespace. I had the same problem in wsl microsoft/WSL#8461. The same as you are run the BPF program in the container, it pid namespace not is global namespace. Tansks @X-QIANG.

[lzh-lab]

I have the same problem in a host and reboot solved it.

[novitoll]

Make sure that tracing_on is enabled.

echo 1 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace_pipe