ld.lld: error: ld-temp.o <inline asm>:14577:1: symbol '__BTF_ID__struct__cgroup__624' is already defined

[ms178]

Hi, with the following config, Clang-18 (52aa6bd3fd6f48f583beaadcbb53edc0f3def4a1), Full-LTO, several SRSO-patches applied as well as "CONFIG_DEBUG_LIST_MINIMAL"-patches (see the attached ms178.patch file), I now see the following on 6.4.9:

  AR      vmlinux.a
  GEN     .tmp_initcalls.lds
  LD      vmlinux.o
ld.lld: error: ld-temp.o <inline asm>:14577:1: symbol '__BTF_ID__struct__cgroup__624' is already defined
__BTF_ID__struct__cgroup__624:                                     
^

config.txt
0001-ms178.patch.txt

[edit:] As the question of pahole came up, I am on a very recent snapshot: 1.25.r7.g37e3bf0

[ms178]

ThinLTO with the same patches and config doesn't error out, by the way.

[nickdesaulniers]

cc @melver

[melver]

Does the same happen with v4 (renamed to CONFIG_LIST_HARDENED and some other fixes)? https://lkml.kernel.org/r/20230811151847.1594958-1-elver@google.com
(I suspect yes, but just double checking.)

[nathanchance]

This is reproducible for me with just the two SRSO patches on top of 6.4.10, so I think the other patches are a red herring.

$ clang --version | head -1
ClangBuiltLinux clang version 18.0.0 (https://github.com/llvm/llvm-project 989f25001aee3457738c431ba1e977a8140726a0)

$ git log --oneline @{u}^..
897a51cdd98f x86/retpoline,kprobes: Fix position of thunk sections with CONFIG_LTO_CLANG
371970c7dd86 x86/srso: fix build breakage for LD=ld.lld
b269b0268d41 Linux 6.4.10

$ curl -LSso .config https://github.com/ClangBuiltLinux/linux/files/12321077/config.txt

$ scripts/config -d LTO_CLANG_THIN -e LTO_CLANG_FULL

$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig all
...
ld.lld: error: ld-temp.o <inline asm>:14542:1: symbol '__BTF_ID__struct__cgroup__624' is already defined
__BTF_ID__struct__cgroup__624:
^

I'll see if this is an LLVM regression.

[ms178]

Sorry, I did not want to imply that Marco's series was causal in any sort, as I was unsure which of the new patches caused it, I just wanted to mention that I carried Marco's series as well. Great that Nathan could reproduce the issue and happy hunting for a fix.

Reported-by: Marcus Seyfarth <m.seyfarth@gmail.com>

[nathanchance]

It reproduces for me with LLVM 17.0.0-rc2 and 16.0.6 as well, so this is probably something configuration related if it is not a recent Linux regression. I don't know how much more time I will have today to look at this so I might not be able to get back to it until next week.

[nickdesaulniers]

Strange, I could not reproduce:

$ git log --oneline @{u}^.. | cat
b269b0268d41 Linux 6.4.10
$ clang --version | head -n 1
clang version 18.0.0 (git@github.com:llvm/llvm-project.git 2105c32b32bf3360f524f1f3a1e5af605821eff8
$ curl -LSso .config https://github.com/ClangBuiltLinux/linux/files/12321077/config.txt
$ scripts/config -d LTO_CLANG_THIN -e LTO_CLANG_FULL
$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig all
ld.lld: warning: drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn31/display_mode_vba_31.c:3798:0: stack frame size (2440) exceeds limit (2048) in function 'dml31_ModeSupportAndSystemConfigurationFull'
ld.lld: warning: drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn314/display_mode_vba_314.c:3893:0: stack frame size (2472) exceeds limit (2048) in function 'dml314_ModeSupportAndSystemConfigurationFull'

[nathanchance]

Hmmm... Could this pahole related?

$ pahole --version
v1.25

[nathanchance]

Interestingly, this is not visible in mainline with the same configuration. I reverse bisected to b5ad4cdc46c7 ("bpf: Add bpf_task_under_cgroup() kfunc") as the "fix" but that does not seem like a real fix to me. I'll see if there is something about the configuration that specifically triggers this.

[nathanchance]

I am able to reproduce this with defconfig but the configuration changes do not make it obvious what is happening here... specifically, it only appears for me when CONFIG_KEXEC is disabled.

$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper defconfig

$ scripts/config \
    -d DEBUG_INFO_NONE \
    -d KEXEC \
    -d LTO_NONE \
    -e BPF_SYSCALL \
    -e DEBUG_INFO_BTF \
    -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
    -e LTO_CLANG_FULL

$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig vmlinux
ld.lld: error: ld-temp.o <inline asm>:27367:1: symbol '__BTF_ID__struct__cgroup__591' is already defined
__BTF_ID__struct__cgroup__591:
^
...

[nickdesaulniers]

Strange, I could not reproduce:
Hmmm... Could this pahole related?

No; I'm an idiot; after reimaging my machine, I lost my previous build of pahole. With that rebuilt+updated, I can reproduce on mainline with b5ad4cd reverted).

Looks like Android is now hitting this too, bumping this issue up in my priority.

I didn't see any similar reports in recent pahole or libbpf commits.

An identifier like __BTF_ID__struct__cgroup__591 comes from the use of BTF_ID macro in include/linux/btf_ids.h. It uses a macro __ID which "pastes" __COUNTER__ on the end of the identifier.

I almost wonder if there is a slight chance that given some esoteric set of configs, it is possible to generate a conflict between the usage of BTF_ID in two different translation units. For example:

kernel/bpf/helpers.c:2460:BTF_ID(struct, cgroup)
kernel/bpf/verifier.c:5075:BTF_ID(struct, cgroup)

Surely the probability of that is very low, but perhaps non-zero? Should one of those be deleted?

[nickdesaulniers]

With that rebuilt+updated, I can reproduce on mainline with b5ad4cd reverted).

Sorry, that's not true. I still am not able to reproduce on mainline @ dd1386d w/ or w/o b5ad4cd reverted, w @ms178 's .config or @nathanchance 's STR.

I think this lends to my theory of "near zero but not impossible probability of COUNTER resulting in collision." Will mail the BPF mailing list.

[nathanchance]

FWIW, I do not know that I was ever able to reproduce this on mainline... Did you try b5ad4cdc46c7d6e7f8d2c9e24b6c9a1edec95154^? I can still reproduce it with my simplified reproducer above on 6.4.15. I agree with you that this is likely a __COUNTER__ collision. I bet the fix commit that I landed on changes the __COUNTER__ value for the BTF_ID(struct, cgroup) in kernel/bpf/helpers.c, helping avoid this issue.

[nickdesaulniers]

thread: https://lore.kernel.org/llvm/CAKwvOdnaEakT_y8TA9b_nMY3kMp=xxqKpGQPc2drNqRdV39RQw@mail.gmail.com/

will test @ b5ad4cd.

[nickdesaulniers]

Mainline:

$ git checkout b5ad4cd
$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper defconfig
$ scripts/config \
    -d DEBUG_INFO_NONE \
    -d KEXEC \
    -d LTO_NONE \
    -e BPF_SYSCALL \
    -e DEBUG_INFO_BTF \
    -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
    -e LTO_CLANG_FULL
$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig vmlinux
WARN: resolve_btfids: unresolved symbol bpf_cpumask
$ echo $?
130
$

Stable:

$ git log -1 --format=oneline | cat
f60d5fd5e950c89a38578ae6f25877de511bb031 Linux 6.4.15
$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper defconfig
$ scripts/config \
    -d DEBUG_INFO_NONE \
    -d KEXEC \
    -d LTO_NONE \
    -e BPF_SYSCALL \
    -e DEBUG_INFO_BTF \
    -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
    -e LTO_CLANG_FULL
$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 olddefconfig vmlinux
ld.lld: error: ld-temp.o <inline asm>:27411:1: symbol '__BTF_ID__struct__cgroup__598' is already defined
__BTF_ID__struct__cgroup__598:                                     
^
$ llvm-nm vmlinux.a 2>/dev/null | grep __BTF_ID__ | grep cgroup
...
---------------- t __BTF_ID__struct__cgroup__598
...
---------------- t __BTF_ID__struct__cgroup__598
...

aha! Thanks for the reproducer! Ok, that should make debugging this faster.

[nickdesaulniers]

Testing:

diff --git a/tools/include/linux/btf_ids.h b/tools/include/linux/btf_ids.h
index 71e54b1e3796..30e920b96a18 100644
--- a/tools/include/linux/btf_ids.h
+++ b/tools/include/linux/btf_ids.h
@@ -38,7 +38,7 @@ asm(                                                  \
        ____BTF_ID(symbol)
 
 #define __ID(prefix) \
-       __PASTE(prefix, __COUNTER__)
+       __PASTE(prefix, __COUNTER__ __LINE__)
 
 /*
  * The BTF_ID defines unique symbol for each ID pointing

[ms178]

I don't know why Nick's patch from above doesn't apply on my 6.4.15 Kernel, maybe there was some whitespace issue? This one does apply though:

diff --git a/tools/include/linux/btf_ids.h b/tools/include/linux/btf_ids.h
index 71e54b1e3796..30e920b96a18 100644
--- a/tools/include/linux/btf_ids.h
+++ b/tools/include/linux/btf_ids.h
@@ -38,7 +38,7 @@ asm(							\
 	____BTF_ID(symbol)

 #define __ID(prefix) \
-	__PASTE(prefix, __COUNTER__)
+	__PASTE(prefix, __COUNTER__ __LINE__)

 /*
  * The BTF_ID defines unique symbol for each ID pointing

[ms178]

The issue is gone with that patch on 6.4.15 with Full LTO and my Kernel. Tested-by: Marcus Seyfarth <m.seyfarth@gmail.com>

[nickdesaulniers]

https://lore.kernel.org/llvm/ZPsJ4AAqNMchvms%2F@krava/ is the latest patch from Jiri.