Module loading fails with DEBUG_INFO_BTF_MODULES

[YakoYakoYokuYoku]

Description

While testing the v5 patchset with the Arch Linux config (see below) I've stumbled upon module loading problems. I could reproduce the issue with the v5.16 series but more relevantly the v5.17 series. Both QEMU and bare metal don't want to work with it. Builds with CONFIG_DEBUG_INFO_BTF disabled load the modules as expected.

Stems from #695.

Architecture(s)

• x86_64

Toolchain versions

• rustc: 1.59.0
• bindgen: 0.56.0
• LLVM/Clang: 13.0.1
• GCC: 11.2.0
• QEMU: 6.2.0

Kernel log

[    0.642535] BPF:     width type_id=54975 bits_offset=0
[    0.642932] BPF:  
[    0.643100] BPF: Invalid member bits_offset
[    0.643413] BPF:

Kernel config

#
# Compile-time checks and compiler options
#
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_PAHOLE_HAS_SPLIT_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
CONFIG_GDB_SCRIPTS=y
CONFIG_FRAME_WARN=2048
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_HEADERS_INSTALL is not set
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
CONFIG_STACK_VALIDATION=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# end of Compile-time checks and compiler options

[YakoYakoYokuYoku]

Relevant modinfo and modprobe command output:

:/root# modinfo dm_mod
filename:       /lib/modules/5.17.1-arch1-1/kernel/drivers/md/dm-mod.ko.zst
description:    device-mapper driver
author:         Joe Thornber <dm-devel@redhat.com>
license:        GPL
alias:          char-major-10-236
alias:          devname:mapper/control
vermagic:       5.17.1-arch1-1 SMP preempt mod_unload 
name:           dm_mod
intree:         Y
retpoline:      Y
depends:        
srcversion:     C20888C8A5A2565787844BA
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        01:08:1A:6D:84:07:65:49:3C:B8:31:0C:05:F1:3D:AF:6A:38:11:23
sig_hashalgo:   sha512
signature:      30:65:02:31:00:EA:FF:DC:C6:5B:73:6F:3C:72:6D:45:C2:9F:29:B0:
                77:7F:8C:34:93:3B:F8:E7:35:FE:24:D0:70:93:E0:89:83:01:DD:AB:
                62:CA:84:70:4B:8B:CE:1F:80:CA:73:DB:46:02:30:23:83:50:F0:BD:
                B7:CC:A8:9A:F5:2C:95:94:C6:41:13:9C:68:86:51:D4:F8:E7:86:93:
                4F:9D:B6:11:A9:CD:F6:C6:53:7A:58:15:23:CF:00:93:8C:FD:54:31:
                85:9D:47
parm:           dm_mq_queue_depth:Queue depth for request-based dm-mq devices (uint)
parm:           dm_mq_nr_hw_queues:Number of hardware queues for request-based dm-mq devices (uint)
parm:           use_blk_mq:Use block multiqueue for request-based DM devices (bool)
parm:           reserved_rq_based_ios:Reserved IOs in request-based mempools (uint)
parm:           stats_current_allocated_bytes:Memory currently used by statistics (ulong)
parm:           kcopyd_subjob_size_kb:Sub-job size for dm-kcopyd clients (uint)
parm:           swap_bios:Maximum allowed inflight swap IOs (int)
parm:           dm_numa_node:NUMA node for DM device memory allocations (int)
parm:           reserved_bio_based_ios:Reserved IOs in bio-based mempools (uint)
parm:           major:The major number of the device mapper (uint)
:/root# modprobe dm_mod
modprobe: ERROR: could not insert 'dm_mod': Invalid argument

[bjorn3]

Shouldn't BTF generation be skipped for rust modules @ojeda?

[YakoYakoYokuYoku]

Shouldn't BTF generation be skipped for rust modules ojeda?

This may be unrelated to skipping generation with Rust mods as dm_mod is a C module. I speculate that this could be a problem with a source in kernel or an error while processing the modules from an script in scripts.

[ojeda]

I have reproduced the problem. Yes, it seems related to BTF generation for built-in Rust code -- it uses some unsupported features in pahole. We could disallow CONFIG_DEBUG_INFO_BTF for the moment, but I would like to give a try to solve it in a different way without forcing to disable it.

[ojeda]

A summary of news on this...

I took a look at what was going wrong. In the kernel side, BTF checking fails on a couple of asserts since:

• Rust struct members may be reordered, unlike C.
• Rust type names use quite a few more characters (at least these [1]).

This is triggered when loading C modules (rather than Rust ones) because it checks the debug info for vmlinux too (base BTF) the first time this is needed.

To solve this without making the config options exclusive, one possibility is to skip the BTF generation for Rust CUs to begin with. There is the --cu_exclude= flag in pahole but that only allows 1 prefix, rather than a list. I gave a try at doing it with a hack like [2], and that seemed to do the trick (so if somebody really needs it right now, you could play with that).

I asked the pahole and BPF maintainers about their opinion on the different possibilities, and Arnaldo (@acmel) agreed it would be nice to have more flexible options to skip CUs. I suggested something like a --cu_exclude_rust flag, and then he suggested pahole could use the DW_AT_language attribute of DW_TAG_compile_unit to implement a --cu_exclude_lang flag, which would be very nice. He was kind enough to start working on it already. He may also take a look at accepting several prefixes in --cu_exclude= (although with the language one we would not need it, but it could be good for users nevertheless).

I also noticed a few other issues when using pahole with Rust-generated code -- Arnaldo also kindly offered to take a look at those too. It would be very useful to have pahole/dwarves supporting Rust better! (even if the Rust compiler reorders structs in the default representation).

As for working around it on the kernel side, there are some assumptions about the ordering of members etc. in the kernel code and libbpf according to Andrii (@anakryiko), thus relaxing that would not be straightforward. In any case, having pahole skip the BTF generation for Rust CUs is simpler (and conceptually better).

[1]

       c == '(' || c == ')'              /* () */
    || c == '[' || c == ']'              /* [T] */
    || c == ';' || c == ' '              /* [T; 1] */
    || c == '&'                          /* &T */
    || c == '*'                          /* *mut T */
    || c == ':'                          /* m::T */
    || c == '<' || c == '>' || c == ','  /* S<T, T> */
    || c == '-'                          /* fn() -> T */
    || c == '+'                          /* dyn T + U */
    || c == '"'                          /* extern "C" */
    || c == '{' || c == '}' || c == '#'  /* {closure#0} */

[2]

diff --git a/pahole.c b/pahole.c
index f3a51cb..af3b0ff 100644
--- a/pahole.c
+++ b/pahole.c
@@ -600,6 +600,9 @@ static void print_ordered_classes(void)

 static struct cu *cu__filter(struct cu *cu)
 {
+       if (strstr(cu->name, "/@/") != NULL)
+               return NULL;
+
        if (cu__exclude_prefix != NULL &&
            (cu->name == NULL ||
             strncmp(cu__exclude_prefix, cu->name,

[YakoYakoYokuYoku]

I have a rundown of how to fix this. First pahole needs to be patched to perform exclusions of Rust compilation units, I do recommend my patch over the above snippet due to cu->name being nullable, then logic for checking if pahole supports those exclusions needs to be added to Kconfig or another part of the build system, subsequently #743 might need to be reverted and lastly scripts/pahole-flags.sh should be modified to pass the configuration, much like this

diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
index e6093adf4c06..46471c8d2f91 100755
--- a/scripts/pahole-flags.sh
+++ b/scripts/pahole-flags.sh
@@ -16,5 +16,8 @@ fi
 if [ "${pahole_ver}" -ge "121" ]; then
 	extra_paholeopt="${extra_paholeopt} --btf_gen_floats"
 fi
+if [ "${pahole_ver}" -ge "123" ]; then
+	extra_paholeopt="${extra_paholeopt} --cu_exclude_lang=Rust"
+fi
 
 echo ${extra_paholeopt}

[ojeda]

Yeah, Arnaldo (the author of pahole/dwarves) was working on adding the flag last week (see above), so he may have it already ready; i.e. the intention was to wait for him to have it working and released (of course, it is great that you took the time to also look at it!).

(By the way, to be clear, the snippet was just a hack to test the Rust CU exclusion; not intended as the actual change for pahole).

[acmel]

Oh well, I just finished working on --lang and --lang_exclude as a cu filter, went to look for the message from Miguel to reply to and found out about this patch :-\ Please take a look at the "next" branch:

https://github.com/acmel/dwarves/commits/next

[YakoYakoYokuYoku]

No problem @acmel, if not, I'm glad that you've submitted the changes in next and I do really appreciate it 😃