PGO clang crashes with BOLT instrumentation

[nathanchance]

I am attempting to wire up BOLT support into our toolchain build script. However, when clang is compiled with profile guided optimization, it crashes after it has been instrumented with BOLT. I noticed this when building scripts/dtc/srcpos.c in the Linux kernel, which I reduced below. I see #53994 but the crash is different so I figured I would report it and let someone else mark it as a duplicate.

I was able to reproduce at bff8356 with the following steps on two different x86_64 machines. If assertions are enabled (-DLLVM_ENABLE_ASSERTIONS=ON on all stages), there is no crash.

1. Compile stage 1

$ cmake \
-B build/stage1 \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/clang \
-DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
-DLLVM_ENABLE_PROJECTS="bolt;clang;compiler-rt;lld" \
-DLLVM_ENABLE_TERMINFO=OFF \
-DLLVM_TARGETS_TO_BUILD=host \
-DLLVM_USE_LINKER=/usr/bin/ld.lld \
-S llvm

$ ninja -C build/stage1

2. Compile stage 2

$ cmake \
-B build/stage2 \
-G Ninja \
-DCLANG_TABLEGEN=$PWD/build/stage1/bin/clang-tblgen \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$PWD/build/stage1/bin/clang \
-DCMAKE_CXX_COMPILER=$PWD/build/stage1/bin/clang++ \
-DLLVM_BUILD_INSTRUMENTED=IR \
-DLLVM_BUILD_RUNTIME=OFF \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_ENABLE_TERMINFO=OFF \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_TABLEGEN=$PWD/build/stage1/bin/llvm-tblgen \
-DLLVM_TARGETS_TO_BUILD=host \
-DLLVM_USE_LINKER=$PWD/build/stage1/bin/ld.lld \
-DLLVM_VP_COUNTERS_PER_SITE=6 \
-S llvm

$ ninja -C build/stage2

3. Run stage 2 against LLVM to generate profiles

$ cmake \
-B build/pgo \
-G Ninja \
-DCLANG_TABLEGEN=$PWD/build/stage1/bin/clang-tblgen \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$PWD/build/stage2/bin/clang \
-DCMAKE_CXX_COMPILER=$PWD/build/stage2/bin/clang++ \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_ENABLE_TERMINFO=OFF \
-DLLVM_TABLEGEN=$PWD/build/stage1/bin/llvm-tblgen \
-DLLVM_TARGETS_TO_BUILD=host \
-DLLVM_USE_LINKER=$PWD/build/stage2/bin/ld.lld \
-S llvm

$ ninja -C build/pgo check-{clang,lld,llvm,llvm-unit}

4. Build .prof file from raw profiles

$ build/stage1/bin/llvm-profdata merge \
-output=$PWD/build/profdata.prof \
build/stage2/profiles/*.profraw

5. Build final compiler (stage 3)

$ cmake \
-B build/stage3 \
-G Ninja \
-DCLANG_TABLEGEN=$PWD/build/stage1/bin/clang-tblgen \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$PWD/build/stage1/bin/clang \
-DCMAKE_CXX_COMPILER=$PWD/build/stage1/bin/clang++ \
-DCMAKE_EXE_LINKER_FLAGS=-Wl,-q \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_ENABLE_TERMINFO=OFF \
-DLLVM_PROFDATA_FILE=$PWD/build/profdata.prof \
-DLLVM_TABLEGEN=$PWD/build/stage1/bin/llvm-tblgen \
-DLLVM_TARGETS_TO_BUILD=host \
-DLLVM_USE_LINKER=$PWD/build/stage1/bin/ld.lld \
-S llvm

$ ninja -C build/stage3

6. Instrument clang binary with llvm-bolt

$ build/stage1/bin/llvm-bolt \
--instrument \
--instrumentation-file=$PWD/build/clang.fdata \
-o build/stage3/bin/clang.inst \
build/stage3/bin/clang-15
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: bff8356b1969d2edd02e22c73d1c3d386f862937
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x5400000, offset 0x5400000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling -align-macro-fusion=all since no profile was specified
BOLT-INFO: enabling lite mode
BOLT-WARNING: Failed to analyze 1194 relocations
BOLT-WARNING: 2 collisions detected while hashing binary objects. Use -v=1 to see the list.
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 39000
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 135258
BOLT-INSTRUMENTER: Number of function descriptors: 135258
BOLT-INSTRUMENTER: Number of branch counters: 1270794
BOLT-INSTRUMENTER: Number of ST leaf node counters: 641245
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 1912039
BOLT-INSTRUMENTER: Total size of counters: 15296312 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 17096121 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 136151340 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /home/nathan/cbl/src/llvm-project/build/clang.fdata
BOLT-INFO: 0 out of 136234 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: the input contains 16376 (dynamic count : 0) opportunities for macro-fusion optimization that are going to be fixed
BOLT-INFO: 4660727 instructions were shortened
BOLT-INFO: removed 126 empty blocks
BOLT-INFO: UCE removed 1077 blocks and 73981 bytes of code.
BOLT-INFO: SCTC: patched 0 tail calls (0 forward) tail calls (0 backward) from a total of 0 while removing 0 double jumps and removing 0 basic blocks totalling 0 bytes of code. CTCs total execution count is 0 and the number of times CTCs are taken is 0.
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0xe84aaf0
BOLT-INFO: clear procedure is 0xe8463e0

7. Compile test file. The BOLT instrumented clang will crash but the original will not.

srcpos.i:

# 1 "" 3
int initial_pathlen_i, initial_pathlen_j;
initial_pathlen() {
  int diff = initial_pathlen;
  char *res = initial_pathlen_i = 0;
  for (; initial_pathlen_i != diff; initial_pathlen_i++)
    res[initial_pathlen_j++] = res[initial_pathlen_j++] = '/';
}

$ build/stage3/bin/clang-15 -O2 -fomit-frame-pointer -std=gnu89 -c -o /dev/null srcpos.i

$ build/stage3/bin/clang.inst -O2 -fomit-frame-pointer -std=gnu89 -c -o /dev/null srcpos.i
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: build/stage3/bin/clang.inst -O2 -fomit-frame-pointer -std=gnu89 -c -o /dev/null srcpos.i
1.      <eof> parser at end of file
2.      Optimizer
 #0 0x0000560576935564 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (build/stage3/bin/clang.inst+0x9535564)
 #1 0x0000560576934f01 llvm::sys::CleanupOnSignal(unsigned long) (build/stage3/bin/clang.inst+0x9534f01)
 #2 0x0000560576895f22 (anonymous namespace)::CrashRecoveryContextImpl::HandleCrash(int, unsigned long) CrashRecoveryContext.cpp:0:0
 #3 0x000056057689624c CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0
 #4 0x00007f7246997560 __restore_rt libc_sigaction.c:0:0
 #5 0x00005605747036e7 unsigned int std::__find_if<llvm::CostTblEntryT<unsigned int> const*, __gnu_cxx::__ops::_Iter_pred<llvm::CostTblEntryT<unsigned int> const* llvm::CostTableLookup<unsigned int>(llvm::ArrayRef<llvm::CostTblEntryT<unsigned int> >, int, llvm::MVT)::'lambda'(llvm::CostTblEntryT<unsigned int> const&)> >(unsigned int, unsigned int, __gnu_cxx::__ops::_Iter_pred<llvm::CostTblEntryT<unsigned int> const* llvm::CostTableLookup<unsigned int>(llvm::ArrayRef<llvm::CostTblEntryT<unsigned int> >, int, llvm::MVT)::'lambda'(llvm::CostTblEntryT<unsigned int> const&)>, std::random_access_iterator_tag) X86TargetTransformInfo.cpp:0:0
 #6 0x00005605756cb6ac llvm::X86TTIImpl::getInterleavedMemoryOpCost(unsigned int, llvm::Type*, unsigned int, llvm::ArrayRef<unsigned int>, llvm::Align, unsigned int, llvm::TargetTransformInfo::TargetCostKind, bool, bool) X86TargetTransformInfo.cpp:0:0
 #7 0x0000560574435595 llvm::LoopVectorizationCostModel::getInterleaveGroupCost(llvm::Instruction*, llvm::ElementCount) (build/stage3/bin/clang.inst+0x7035595)
 #8 0x0000560574424bda llvm::LoopVectorizationCostModel::setCostBasedWideningDecision(llvm::ElementCount) (build/stage3/bin/clang.inst+0x7024bda)
 #9 0x00005605744233a2 llvm::LoopVectorizationCostModel::collectUniformsAndScalars(llvm::ElementCount) LoopVectorize.cpp:0:0
#10 0x00005605743b61ee llvm::LoopVectorizationPlanner::plan(llvm::ElementCount, unsigned int) (build/stage3/bin/clang.inst+0x6fb61ee)
#11 0x0000560574394b31 llvm::LoopVectorizePass::processLoop(llvm::Loop*) (build/stage3/bin/clang.inst+0x6f94b31)
#12 0x0000560574393b5d llvm::LoopVectorizePass::runImpl(llvm::Function&, llvm::ScalarEvolution&, llvm::LoopInfo&, llvm::TargetTransformInfo&, llvm::DominatorTree&, llvm::BlockFrequencyInfo&, llvm::TargetLibraryInfo*, llvm::DemandedBits&, llvm::AAResults&, llvm::AssumptionCache&, std::function<llvm::LoopAccessInfo const& (llvm::Loop&)>&, llvm::OptimizationRemarkEmitter&, llvm::ProfileSummaryInfo*) (build/stage3/bin/clang.inst+0x6f93b5d)
#13 0x00005605743933f4 llvm::LoopVectorizePass::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) (build/stage3/bin/clang.inst+0x6f933f4)
#14 0x00005605743930cd llvm::detail::PassModel<llvm::Function, llvm::LoopVectorizePass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function> >::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) PassBuilder.cpp:0:0
#15 0x0000560573161327 llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function> >::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) (build/stage3/bin/clang.inst+0x5d61327)
#16 0x0000560573160e47 llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function> >, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function> >::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) BackendUtil.cpp:0:0
#17 0x00005605753b6d25 llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (build/stage3/bin/clang.inst+0x7fb6d25)
#18 0x00005605753b6937 llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module> >::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) BackendUtil.cpp:0:0
#19 0x00005605750a924f llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module> >::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (build/stage3/bin/clang.inst+0x7ca924f)
#20 0x00005605750a05c1 (anonymous namespace)::EmitAssemblyHelper::RunOptimizationPipeline(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >&, std::unique_ptr<llvm::ToolOutputFile, std::default_delete<llvm::ToolOutputFile> >&) BackendUtil.cpp:0:0
#21 0x000056057508f6a2 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (build/stage3/bin/clang.inst+0x7c8f6a2)
#22 0x000056057508a752 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) CodeGenAction.cpp:0:0
#23 0x00005605752da080 clang::ParseAST(clang::Sema&, bool, bool) (build/stage3/bin/clang.inst+0x7eda080)
#24 0x0000560573041d85 clang::FrontendAction::Execute() (build/stage3/bin/clang.inst+0x5c41d85)
#25 0x0000560573041233 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (build/stage3/bin/clang.inst+0x5c41233)
#26 0x000056057303d806 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (build/stage3/bin/clang.inst+0x5c3d806)
#27 0x000056057303b840 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (build/stage3/bin/clang.inst+0x5c3b840)
#28 0x000056057303a6c5 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) driver.cpp:0:0
#29 0x00005605770989d8 void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool*) const::$_1>(long) Job.cpp:0:0
#30 0x0000560572a465db llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (build/stage3/bin/clang.inst+0x56465db)
#31 0x0000560572a45f51 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool*) const (build/stage3/bin/clang.inst+0x5645f51)
#32 0x0000560572fc01c3 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (build/stage3/bin/clang.inst+0x5bc01c3)
#33 0x0000560572fbf65f clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) const (build/stage3/bin/clang.inst+0x5bbf65f)
#34 0x0000560572fbec16 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) (build/stage3/bin/clang.inst+0x5bbec16)
#35 0x0000560572faeffc main (build/stage3/bin/clang.inst+0x5baeffc)
#36 0x00007f7246982310 __libc_start_call_main libc-start.c:0:0
#37 0x00007f72469823c1 __libc_start_main@GLIBC_2.2.5 (/usr/lib/libc.so.6+0x2d3c1)
#38 0x0000560575a49cf1 _start (build/stage3/bin/clang.inst+0x8649cf1)
clang: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 15.0.0 (https://github.com/llvm/llvm-project bff8356b1969d2edd02e22c73d1c3d386f862937)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/nathan/cbl/src/llvm-project/build/stage3/bin
clang: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

[llvmbot]

@llvm/issue-subscribers-bolt

[maksfb]

Thanks for sending the detailed repro. On the surface indeed it looks similar to the other instrumentation bug (#53994).

[aaupov]

@nathanchance
Can you please clarify what OS and versions of clang/lld were used here? We couldn't reproduce the assertion but that might be due to some system differences.

[nathanchance]

@aaupov I believe I would have reproduced this on Arch Linux, as that is my primary distribution, which currently has the following versions:

$ /usr/bin/clang --version
clang version 13.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

$ /usr/bin/ld.lld --version
LLD 13.0.1 (compatible with GNU linkers)

[aaupov]

@nathanchance
Thanks! Was able to repro the bug now with Ubuntu 20.04 and clang/lld 13.

[Kobzol]

Hi @aaupov, we're trying to use BOLT for optimizing the Rust compiler and we're hitting similar issues when trying to BOLTify LLVM. I wonder if there are any ongoing activities/investigation regarding the PGO clang/llvm crash? Thanks!

[maksfb]

Will put up the fix soon. Let’s see if it solves your problem too.

[Kobzol]

Any updates? I tried it with 15.0.0-rc1, but unfortunately the instrumented libLLVM.so still segfaults for us.

[rafaelauler]

I was able to repro and find the root cause. The symbol that represents the end of a table in .rodata is being colocated with the start of a jump table from another function, and BOLT moves that jump table. This causes the symbol representing the end of the table to be moved as well. The new location is a few MB away in distance, significantly increasing the size of this table as perceived by the application. The application (clang) then crashes scanning values in the table -- because it has the wrong end-of-table address, the loop that scans this table goes out of bounds until it reaches an unmapped address in memory and then segfaults.

We're working on a fix.