failed to find skeleton map '.rodata.str1.1'

[rainkin1993]

Code like bpf_printk("hello\n") will generate a section '.rodata.str1.1'

When I compile the above code with old version bpftool (the bpftool is installed by apt install in Ubuntu 20.10 kernel 5.8), the generated binary works well.

But when I use the same environment with the latest bpftool version to compile the same code, the generated binary report the error "libbpf: failed to find skeleton map '.rodata.str1.1'".

[qmonnet]

Hi and thanks for the report!

I'm not sure if this is a change in bpftool or libbpf that causes this error. As far as I know .rodata.str1.1 has never really been supported by libbpf, I'm not sure why the string ends up in there and why your program was working with an earlier version.

I would appreciate if you could please do the following:

• Confirm that you used the same clang/llvm version for the two tests (with both the old and the new bpftool versions),
• Specify your clang version,
• Provide a reproducer (a minimal eBPF program which fails to load with bpftool), and the command that you run to compile it.

In the meantime, if you need your program to load, one workaround seems to be to declare the string const as a global, see this example.

[rainkin1993]

Hi and thanks for the report!

I'm not sure if this is a change in bpftool or libbpf that causes this error. As far as I know .rodata.str1.1 has never really been supported by libbpf, I'm not sure why the string ends up in there and why your program was working with an earlier version.

I would appreciate if you could please do the following:

• Confirm that you used the same clang/llvm version for the two tests (with both the old and the new bpftool versions),

Yes. Two tests used the same clang/llvm version.

• Specify your clang version,

Ubuntu clang version 11.0.0-2
Target: x86_64

My testing environment is Ubuntu 20.10 kernel version is 5.8.

• Provide a reproducer (a minimal eBPF program which fails to load with bpftool), and the command that you run to compile it.

Here is a simple eBPF program based on libbpf-bootstrap which can reproduce the error:

SEC("kprobe/vfs_read")
int BPF_KPROBE(vfs_read_entry)
{
    bpf_printk("hello  world\n");
    return 0;
}

In the meantime, if you need your program to load, one workaround seems to be to declare the string const as a global, see this example.

Thanks for the example.

[qmonnet]

I'm sorry, I don't manage to recreate the object file with the .rodata.str1.1 section. Would you be able to attach the ELF file directly please?

I tried to:

1. Install a Ubuntu 20.10 VM, install gcc/clang/llvm etc., clang 11.0.0-2
2. Clone the libbpf-boostrap repository
3. Replace e.g. libbpf-boostrap/examples/c/kprobe.bpf.c with your example:

   #include "vmlinux.h"
   #include <bpf/bpf_helpers.h>
   
   char LICENSE[] SEC("license") = "Dual BSD/GPL";
   
   SEC("kprobe/vfs_read")
   int BPF_KPROBE(vfs_read_entry)
   {
       bpf_printk("hello  world\n");
       return 0;
   }

4. Compile: make -j kprobe
5. I can't see the section:

   $ readelf -SW .output/kprobe.bpf.o 
   There are 13 section headers, starting at offset 0x430:
   
   Section Headers:
     [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
     [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
     [ 1] .text             PROGBITS        0000000000000000 000040 000000 00  AX  0   0  4
     [ 2] kprobe/vfs_read   PROGBITS        0000000000000000 000040 000030 00  AX  0   0  8
     [ 3] license           PROGBITS        0000000000000000 000070 00000d 00  WA  0   0  1
     [ 4] .rodata           PROGBITS        0000000000000000 00007d 00000e 00   A  0   0  1
     [ 5] .BTF              PROGBITS        0000000000000000 00008b 0001cc 00      0   0  1
     [ 6] .BTF.ext          PROGBITS        0000000000000000 000257 000060 00      0   0  1
     [ 7] .symtab           SYMTAB          0000000000000000 0002b8 000090 18     12   4  8
     [ 8] .relkprobe/vfs_read REL             0000000000000000 000348 000010 10      7   2  8
     [ 9] .rel.BTF          REL             0000000000000000 000358 000020 10      7   5  8
     [10] .rel.BTF.ext      REL             0000000000000000 000378 000030 10      7   6  8
     [11] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 0003a8 000003 00   E  0   0  1
     [12] .strtab           STRTAB          0000000000000000 0003ab 000085 00      0   0  1
   

[rainkin1993]

Thanks for your reply.
I make a clear machine and do the same steps as you, and it works!
I compare the environment (including clang version, libbpf...) with my previous test, I find that the difference is libbpf version.
My previous test that returns errors use libbpf 0.4 version while the new test uses the latest version. When I update the libbpf from 0.4 to latest, the .rodata.str1.1 section disappears and the previous test also works.

But I am curious why the old libbpf version generates the .rodata.str1.1 section

[qmonnet]

Great, thanks for testing! I want to investigate more on this but didn't have time this week, and will be off for the next ten days or so.

Regarding your findings: I wonder if we're talking of the same thing. Was it on kprobe.bpf.o that you observe the .rodata.str1.1 section? I'm asking because libbpf is not involved in compiling from C into this file. It's only involved later, when parsing this ELF file to extract the eBPF bytecode and load it into the kernel. So the libbpf version should not have any influence on what sections are present in the ELF file - only on whether it can work with them when trying to parse and load. I didn't have time to check, but from the error you get, failed to find skeleton map, I wonder if this might come from the skeleton generated by bpftool rather than from the eBPF program itself. I'll need to dig into a it a bit more.

[rainkin1993]

Great, thanks for testing! I want to investigate more on this but didn't have time this week, and will be off for the next ten days or so.

Regarding your findings: I wonder if we're talking of the same thing. Was it on kprobe.bpf.o that you observe the .rodata.str1.1 section?

Yes.
When I compile with the libbpf 0.4 version, the generated binary fails. I run the command llvm-objdump --headers .output/xx.bpf.o and see the .rodata.str1.1 section printed in the screen.
When I just update libbpf from 0.4 to latest version without modifying other content, the generated binary works well and the above command print .rodata section instead of ordata.str1.1.

The bpftool version is

bpftool v6.8.0
using libbpf v0.8
features: libbfd, libbpf_strict

I'm asking because libbpf is not involved in compiling from C into this file. It's only involved later, when parsing this ELF file to extract the eBPF bytecode and load it into the kernel. So the libbpf version should not have any influence on what sections are present in the ELF file - only on whether it can work with them when trying to parse and load. I didn't have time to check, but from the error you get, failed to find skeleton map, I wonder if this might come from the skeleton generated by bpftool rather than from the eBPF program itself. I'll need to dig into a it a bit more.

I agree with you. I am also curious why the libbpf version affects the .rodata.str1.1 section

[chenhengqi]

FYI, see libbpf/libbpf#274 .

[qmonnet]

Thanks @chenhengqi! If I understand correctly, the issue should not happen now with recent clang and libbpf, which matches the observations here.