Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kirkstone: OpenMP fails to build with thin LTO enabled. #708

Open
jaskij opened this issue Dec 17, 2022 · 10 comments
Open

Kirkstone: OpenMP fails to build with thin LTO enabled. #708

jaskij opened this issue Dec 17, 2022 · 10 comments
Labels

Comments

@jaskij
Copy link

jaskij commented Dec 17, 2022

When using clang as the default toolchain and thin-lto is enabeld in distro configuration, OpenMP fails to build.

The error given is:

/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-ld: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a: error adding symbols: file format not recognized

After unpacking libLLVMSupport.a I found out that the object files are, in fact, LLVM IR:

$ file Atomic.cpp.o
Atomic.cpp.o: LLVM IR bitcode

I'm not actually using any LLVM libs on the target, so for now have disabled LTO for clang as a workaround.

@kraj
Copy link
Owner

kraj commented Dec 17, 2022

Thanks for the report @jaskij yes I know there are some issues with LTO enabling with meta-clang. I think the linker plugin is not configured correctly when linking with lto objects. It needs to be root caused a bit more.

@kraj kraj added the bug label Dec 17, 2022
@jaskij
Copy link
Author

jaskij commented Dec 17, 2022

@kraj can you direct me on how to dig into it?

@kraj
Copy link
Owner

kraj commented Dec 17, 2022

@kraj can you direct me on how to dig into it?

you can start by looking at the linking step and extract it out of build and try to do manual link. Then you can add -v option to enable verbosity. This will give more info on what compiler is doing when linking. Perhaps its not calling the linker plugin from right directory in cross-toolchain installation tree or some such.

@jaskij
Copy link
Author

jaskij commented Dec 17, 2022

Looks like it's picking up the GNU linker.

root@yocto:~/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/build# aarch64-my_distro-linux-ld --version
GNU ld (GNU Binutils) 2.38.20220708
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-clang++ -v -fPIC -target aarch64-my_distro-linux  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -mlittle-endian -Qunused-arguments -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security  --sysroot=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot  -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native=  -fvisibility-inlines-hidden  -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-extra -Wno-pedantic -std=c++14 -DNDEBUG  -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fmacro-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native=  -Wl,-z,relro,-z,now -shared -Wl,-soname,libomptarget.so -o libomptarget/libomptarget.so libomptarget/src/CMakeFiles/omptarget.dir/api.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/device.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interface.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interop.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/omptarget.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o  -Wl,-rpath,"\$ORIGIN:"  /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a  -ldl  -Wl,--version-script=/home/jaskij/yocto/build/tmp-glibc/work-shared/llvm-project-source-14.0.3-r0/git/openmp/libomptarget/src/exports  -lrt  -ldl  -lm  /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libz.so  /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMDemangle.a
clang version 14.0.3 (https://github.com/llvm/llvm-project 1f9140064dfbfb0bbda8e51306ea51080b2f7aac)
Target: aarch64-my_distro-linux
Thread model: posix
InstalledDir: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux
Found candidate GCC installation: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0
Selected GCC installation: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-ld" --sysroot=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot -EL --hash-style=both --eh-frame-hdr -m aarch64linux -shared -o libomptarget/libomptarget.so /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//usr/lib/crti.o /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0/crtbeginS.o -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/../lib/aarch64-my_distro-linux -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0 -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//lib -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//usr/lib -O1 --hash-style=gnu --as-needed -z relro -z now -soname libomptarget.so libomptarget/src/CMakeFiles/omptarget.dir/api.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/device.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interface.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interop.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/omptarget.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o -rpath "\$ORIGIN:" /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a -ldl --version-script=/home/jaskij/yocto/build/tmp-glibc/work-shared/llvm-project-source-14.0.3-r0/git/openmp/libomptarget/src/exports -lrt -ldl -lm /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libz.so /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMDemangle.a -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0/crtendS.o /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//usr/lib/crtn.o
/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-ld: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a: error adding symbols: file format not recognized
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

@jaskij
Copy link
Author

jaskij commented Dec 17, 2022

You set me on the right track. Knowing that it's the wrong linker, I tested some stuff and adding -fuse-ld=lld to OpenMP recipe worked.

LDFLAGS:append = "-fuse-ld=lld"

Now, why is it using the GNU linker?

@jaskij
Copy link
Author

jaskij commented Dec 17, 2022

Sorry for the comment spam, but I seem to have found it - using lld distrowide is hidden behind a separate distro features, ld-is-lld. Which is not documented anywhere - search text for ld-is-lld in the repo only returns the one hit in clang.bbclass.

LDFLAGS:append:toolchain-clang = "${@bb.utils.contains('DISTRO_FEATURES', 'ld-is-lld', ' -fuse-ld=lld', '', d)}"

@kraj
Copy link
Owner

kraj commented Dec 17, 2022

@jaskij good ! yes ld-is-lld is only available with meta-clang and its disabled by default because we want the layer to be as inert as possible when added and give knobs to users choice to enable more clang'ness :)

You are right about documenting this distro feature. Perhaps its worth adding it to README. Patches welcome!!

@jaskij
Copy link
Author

jaskij commented Dec 17, 2022

@kraj LDFLAGS:append and documentation both against master and then cherrypick the append to Kirkstone? Or how do you want those patches?

@kraj
Copy link
Owner

kraj commented Dec 18, 2022

yes

jaskij added a commit to jaskij/meta-clang that referenced this issue Dec 19, 2022
When built with thin LTO enabled, LLVM produces object files containing
LLVM IR, which ld can't understand. Since we're forcing the toolchain
to clang anyway, let's also force using lld.

Fixes GitHub kraj#708

Signed-off-by: Jan Dorniak <jaskij@gmail.com>
kraj pushed a commit that referenced this issue Dec 27, 2022
When built with thin LTO enabled, LLVM produces object files containing
LLVM IR, which ld can't understand. Since we're forcing the toolchain
to clang anyway, let's also force using lld.

Fixes GitHub #708

Signed-off-by: Jan Dorniak <jaskij@gmail.com>
@fcuzzocrea
Copy link

With the proposed patch (Which has been picked in the kirkstone branch) openmp fails to build here: https://pastebin.com/m00Recv6

Reverting the commit makes the build pass again. Is there something which I am missing?

rakuco pushed a commit to rakuco/meta-clang that referenced this issue Jul 31, 2023
When built with thin LTO enabled, LLVM produces object files containing
LLVM IR, which ld can't understand. Since we're forcing the toolchain
to clang anyway, let's also force using lld.

Fixes GitHub kraj#708

Signed-off-by: Jan Dorniak <jaskij@gmail.com>
(cherry picked from commit bc8388c)
kraj pushed a commit that referenced this issue Sep 8, 2023
When built with thin LTO enabled, LLVM produces object files containing
LLVM IR, which ld can't understand. Since we're forcing the toolchain
to clang anyway, let's also force using lld.

Fixes GitHub #708

Signed-off-by: Jan Dorniak <jaskij@gmail.com>
(cherry picked from commit bc8388c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants