Two stage builds and ThinLTO

[nathanchance]

I am currently implementing the two stage build style that @stephenhines mentioned in #2, building a smaller clang + ld.lld and using those to build the full toolchain. I have added the ability to use ThinLTO in the second stage to provide a little bit of speed up.

However, the issue I am running into is there are points in the build where GNU ld is used and my system symlinks /usr/lib/LLVMgold.so to /usr/lib/bfd-plugins, which is then used instead of the LLVMgold.so that is built as a part of stage 1, resulting in really ugly errors:

bfd plugin: LLVM gold plugin has failed to create LTO module: Unknown attribute kind (60) (Producer: 'LLVM9.0.0svn' Reader: 'LLVM 8.0.0')
bfd plugin: LLVM gold plugin has failed to create LTO module: Unknown attribute kind (60) (Producer: 'LLVM9.0.0svn' Reader: 'LLVM 8.0.0')
bfd plugin: LLVM gold plugin has failed to create LTO module: Unknown attribute kind (60) (Producer: 'LLVM9.0.0svn' Reader: 'LLVM 8.0.0')
bfd plugin: LLVM gold plugin has failed to create LTO module: Unknown attribute kind (60) (Producer: 'LLVM9.0.0svn' Reader: 'LLVM 8.0.0')
bfd plugin: LLVM gold plugin has failed to create LTO module: Unknown attribute kind (60) (Producer: 'LLVM9.0.0svn' Reader: 'LLVM 8.0.0')
bfd plugin: LLVM gold plugin has failed to create LTO module: Unknown attribute kind (60) (Producer: 'LLVM9.0.0svn' Reader: 'LLVM 8.0.0')

Has anyone ever run into this and overcome it? I can't find anything on overriding bfd-plugins. I've tried to add -L <stage1_lib> to the linker flags via cmake variables but that doesn't help.

At this point, the only viable workaround I can think of is just building a copy of ld and ld.gold standalone to ensure that everything works properly regardless of host setup.

[bwendling]

It looks like a version mismatch between clang and the plugin. Make sure that LLVMgold.so and the clang you're using are the save versions.

[nathanchance]

Correct. My system's LLVMgold.so is 8.0.0 but I'm trying to force the stage 2 build to use the LLVMgold.so that was just built in stage 1 to avoid this mismatch.

[bwendling]

Ah! Sorry about that. You may need to modify the script/link-vmlinux.sh script to point to the correct one. Do you set an environment variable during the second stage?

[nathanchance]

I think I figured it out. Apparently, the command that first causes that spew is:

: && /usr/bin/cmake -E remove lib/libLLVMDemangle.a && /usr/bin/ar qc lib/libLLVMDemangle.a  lib/Demangle/CMakeFiles/LLVMDemangle.dir/Demangle.cpp.o lib/Demangle/CMakeFiles/LLVMDemangle.dir/ItaniumDemangle.cpp.o lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangleNodes.cpp.o && /usr/bin/ranlib lib/libLLVMDemangle.a && :

which I got from ninja --verbose. I always figured it was ld that was complaining, not ar/ranlib... Using llvm-ar and llvm-ranlib alleviates the problem.