Clang plugin segfaulting with unregistered passes

[aaronmondal]

Bug summary
We are trying to get hipsycl to work with upstream Clang/LLVM. The fiber cpu backend seems to work, but when trying to use the plugin (-fpass-plugin) the compiler segfaults:

clang++:
.../llvm/include/llvm/IR/PassManager.h:782:
typename PassT::Result &llvm::AnalysisManager<llvm::Module>::getResult(IRUnitT &, ExtraArgTs...)
[IRUnitT = llvm::Module, ExtraArgTs = <>, PassT = llvm::PassInstrumentationAnalysis]:
  Assertion `AnalysisPasses.count(PassT::ID()) && "This analysis pass was not registered prior to being queried"' failed.

clang: error: clang frontend command failed with exit code 134

To Reproduce
Install upstream Clang/LLVM and try to use the hipSYCL plugin with HIPSYCL_USE_ACCELERATED_CPU for an OMPHOST target so that the plugin is used.

Expected behavior
No crash 🥲

Describe your setup

• Using the branch from [CBS] Fix runtime issues with opaque pointers #765 which fixed compile-time issues with upstream Clang/LLVM.
• We are building hipsycl using a custom Bazel overlay which we will push into https://github.com/eomii/rules_ll as soon as we get this to work. For the plugin itself we are using LLVM_ON_UNIX, HIPSYCL_WITH_ACCELERATED_CPU, HIPSYCL_COMPILER_COMPONENT.
• For executables we emulate the syclcc-clang python script by adding -D__HIPSYCL_OMPHOST_TARGET__, -fpass-plugin=..., -D__HIPSYCL_USE_ACCELERATED_CPU__.

Optional additional diagnostic information
When using only -fpass-plugin, we get the error above. With -mllvm -opt-bisect-limit=<various limits> (and trying various optimization levels) we only get

Additional context

• With -fplugin, the compiler just segfaults without any warnings, backtrace or anything. Just says (Segmentation fault): clang++ failed: error executing command ....
• With -fpass-plugin we get the error posted at the top and a debug stacktrace (trying to make this somewhat readable 😅)

clang++:
  external/@rules_ll.override.llvm_project_overlay.llvm-project/llvm/include/llvm/IR/PassManager.h:782:
    typename PassT::Result&llvm::AnalysisManager<llvm::Module>::getResult(IRUnitT &, ExtraArgTs...) [IRUnitT = llvm::Module, ExtraArgTs = <>, PassT = llvm::PassInstrumentationAnalysis]:
      Assertion `AnalysisPasses.count(PassT::ID()) && "This analysis pass was not registered prior to being queried"' failed.


0.     Program arguments: bazel-out/k8-dbg-ST-d2d72091753b/bin/external/@rules_ll.override.llvm_project_overlay.llvm-project/clang/clang++ 
-fcolor-diagnostics -v -fdebug-default-version=5 -fdebug-compilation-dir=. -Xarch_host -glldb 
-c -fPIC 
-D__HIPSYCL_ENABLE_OMPHOST_TARGET__ 
-fpass-plugin=bazel-out/k8-dbg/bin/external/@hipsycl.override/hipsycl-plugin.so 
-D__HIPSYCL_USE_ACCELERATED_CPU__ 
-Xarch_host -MJbazel-out/k8-dbg-ST-0fdaae6d89e0/bin/sycl_example/main.cdf 
-nostdlib++
-nostdinc++ 
-nostdlib 
--gcc-toolchain=NONE 
-resource-dir=bazel-out/k8-dbg-ST-0fdaae6d89e0/bin/external/@rules_ll.override.llvm_project_overlay.llvm-project/clang/staging 
-isystembazel-out/k8-dbg-ST-0fdaae6d89e0/bin/external/@rules_ll.override.llvm_project_overlay.llvm-project/libcxx/include 
-isystembazel-out/k8-dbg-ST-0fdaae6d89e0/bin/external/@rules_ll.override.llvm_project_overlay.llvm-project/libcxxabi/include 
-isystembazel-out/k8-dbg-ST-0fdaae6d89e0/bin/externa//@rules_ll.override.llvm_project_overlay.llvm-project/libunwind/include 
-isystembazel-out/k8-dbg-ST-0fdaae6d89e0/bin/external/@rules_ll.override.llvm_project_overlay.llvm-project/clang/staging 
-isystemexternal/@rules_ll.override.llvm_project_overlay.llvm-project/libcxx/include 
-isystemexternal/@rules_ll.override.llvm_project_overlay.llvm-project/libcxxabi/include 
-isystemexternal/@rules_ll.override.llvm_project_overlay.llvm-project/libunwind/include 
-std=c++17 
-I/mnt/hipSYCL/include -g -O0 
-fno-omit-frame-pointer sycl_example/main.cpp 
-o bazel-out/k8-dbg-ST-0fdaae6d89e0/bin/sycl_example/main.o

1.      <eof> parser at end of file
                                                                                                                                             
2.      Optimizer                                                                                                                                                               
 #0 0x0000558af98250fa llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) .../llvm/lib/Support/Unix/Signals.inc:569:11                                                                                                                                                                
 #1 0x0000558af98252ab PrintStackTraceSignalHandler(void*) .../llvm/lib/Support/Unix/Signals.inc:636:1 
 #2 0x0000558af98232b6 llvm::sys::RunSignalHandlers() .../llvm/lib/Support/Signals.cpp:103:5           
 #3 0x0000558af982493e llvm::sys::CleanupOnSignal(unsigned long) .../llvm/lib/Support/Unix/Signals.inc:362:1                                                                                                                                                                           
 #4 0x0000558af9757ffe (anonymous namespace)::CrashRecoveryContextImpl::HandleCrash(int, unsigned long) .../llvm/lib/Support/CrashRecoveryContext.cpp:0:7                                                                                                                              
 #5 0x0000558af9758463 CrashRecoverySignalHandler(int) .../llvm/lib/Support/CrashRecoveryContext.cpp:390:1                                                                                                                                                                             
 #6 0x00007f62f65111a0 (/lib64/libc.so.6+0x3e1a0)                                                                                                                               
 #7 0x00007f62f6560f9c __pthread_kill_implementation /usr/src/debug/sys-libs/glibc-2.34-r13/glibc-2.34/nptl/pthread_kill.c:44:76                                                
 #8 0x00007f62f65110f2 gsignal /usr/src/debug/sys-libs/glibc-2.34-r13/glibc-2.34/signal/../sysdeps/posix/raise.c:27:6                                                           
 #9 0x00007f62f64fb46a abort /usr/src/debug/sys-libs/glibc-2.34-r13/glibc-2.34/stdlib/abort.c:81:7                                                                              
#10 0x00007f62f64fb395 _nl_load_domain /usr/src/debug/sys-libs/glibc-2.34-r13/glibc-2.34/intl/loadmsgcat.c:1177:9                                                               
#11 0x00007f62f6509df2 (/lib64/libc.so.6+0x36df2)                                                                                                                               
#12 0x00007f62f00e1f6b llvm::PassInstrumentationAnalysis::Result& llvm::AnalysisManager<llvm::Module>::getResult<llvm::PassInstrumentationAnalysis>(llvm::Module&) .../llvm/include/llvm/IR/PassManager.h:784:23

clang: error: clang frontend command failed with exit code 134 (use -v to see invocation)

I will try to create a reproducer ready with CMake and upstream Clang/LLVM to make sure this is not coming from our custom toolchain. I believe this issue comes somewhere from changes in the passmanager implementations in upstream. Seems like pass registration is somehow broken.

I will investigate this further and post any findings.

[aaronmondal]

cc @fodinabor

[fodinabor]

Hi @aaronmondal,
thanks for tracking this.

As a reference, with a "typical" setup, it seems to work quite fine: https://github.com/fodinabor/hipSYCL/runs/7296288804?check_suite_focus=true
https://github.com/fodinabor/hipSYCL/runs/7296289502?check_suite_focus=true

I'm not too familiar with bazel and thus don't really know where to look for it, so could you maybe share a summary of what flags LLVM and hipSYCL are being built with? (usually I'd ask for ninja -v or so, to see the command line)
Things to look out for: is LTO enabled? That possibly removes some registration magic (although that should only impact the legacy pass manager..).

From the stack trace I also can't make out that hipSYCL is involved in the crash at all?

I guess that assertion should not happen in the first place, but since hipSYCL is running into some overly cautious assertions in the Clang frontend, hipSYCL only supports LLVM builds with assertions disabled anyways. Could you test whether it fails with LLVM built with assertions disabled?
Also make sure that both, the hipSYCL plugin and LLVM are built with the same RTTI / exception settings. These settings are typically inherited when using the CMake integration, but not sure whether that's correctly generated when not using CMake primarily?

Maybe some of these points already help you tracking the issue down, otherwise more information on what the actual LLVM / hipSYCL plugin build commands are would be great. (And don't worry about formatting, I'm kinda used to those traces ;-))

[aaronmondal]

@fodinabor Thanks a lot for the pointers! I went over our build configurations was able to get it almost working. I am now fairly certain that this is an issue related to the passes that are enabled when using HIPSYCL_WITH_ACCELERATED_CPU and the new pass manager via -fpass-plugin.

I checked LTO, RTTI and exception settings, but these don't appear to be causing this. The compilation arguments for building the plugin sources look like this:

clang -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -main-file-name Vector
Shape.cpp -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwi
nd-tables=2 -target-cpu x86-64 -tune-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=. -nostdinc++ -resource-dir bazel-out/k8-fastbuild/bin/external/@rules_ll.override.llvm_proje
ct_overlay.llvm-project/clang/staging

<includes omitted>

-D LLVM_ON_UNIX -D HIPSYCL_COMPILER_COMPONENT -D HIPSYCL_WITH_ACCELERATED_CPU -std=c++17 -fdeprecated-macro -fdebug-compilation-dir=. -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o bazel-out/k8-fastbuild/bin/external/@hipsycl.override/sr
c/compiler/cbs/VectorShape.o -x c++ external/@hipsycl.override/src/compiler/cbs/VectorShape.cpp

To link the plugin we use

-dynamic-linker=/lib64/ld-linux-x86-64.so.2 --shared --nostdlib -L/usr/lib64 -lm -ldl -lpthread -ltinfo -lz -lc --verbose --no-undefined -znodelete

✅ Without the accelerated CPU flag, the plugin does not crash and works with both -fplugin and -fpass-plugin.
✅ With the accelerated CPU flag, the -fplugin option works
❌ With the accelerated CPU flag, the -fpass-plugin option triggers the "missing PassInstrumentationAnalysis" error above.

Trying to bandaid the missing pass (without really knowing what I'm doing 😅) by adding

diff --git a/src/compiler/HipsyclClangPlugin.cpp b/src/compiler/HipsyclClangPlugin.cpp
index 24b0f9f..07da4de 100644
--- a/src/compiler/HipsyclClangPlugin.cpp
+++ b/src/compiler/HipsyclClangPlugin.cpp
@@ -103,6 +104,7 @@ extern "C" LLVM_ATTRIBUTE_WEAK ::llvm::PassPluginLibraryInfo llvmGetPassPluginIn
 
 #ifdef HIPSYCL_WITH_ACCELERATED_CPU
           PB.registerAnalysisRegistrationCallback([](llvm::ModuleAnalysisManager &MAM) {
+            MAM.registerPass([] { return llvm::PassInstrumentationAnalysis{}; });
             MAM.registerPass([] { return SplitterAnnotationAnalysis{}; });
           });
 #if LLVM_VERSION_MAJOR < 12

changes the error to

llvm/include/llvm/IR/PassManager.h:782: typename PassT::Result &llvm::AnalysisManager<llvm::Module>::getR
esult(IRUnitT &, ExtraArgTs...) [IRUnitT = llvm::Module, ExtraArgTs = <>, PassT = llvm::InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>]: Assertion `Anal
ysisPasses.count(PassT::ID()) && "This analysis pass was not registered prior to being queried"' failed. 

Now, I am not sure whether this is because PassInstrumentationAnalysis requires FunctionAnalysisManager, or because FunctionAnalysisManager is another requirement for the CBS-related new-PM passes, but at least we know that this code is reached 🤣

I think im close to the solution 😃

[aaronmondal]

@fodinabor Ok I finally got around to playing around with this again. I'm still not very familiar with plugins, but I think I was able to get a better understanding of what is happening

Two lines appear to cause issues. The first one is this:
https://github.com/illuhad/hipSYCL/blob/f3177b86c7731c27805dfff51c1efa2e7b96cb1f/src/compiler/cbs/PipelineBuilder.cpp#L72

This causes llvm::PassInstrumentationAnalysis pass was not registered prior to being queried. This surprises me since that pass only seems to get results of SplitterAnnotationAnalysis, which I believe is being registered here:
https://github.com/illuhad/hipSYCL/blob/f3177b86c7731c27805dfff51c1efa2e7b96cb1f/src/compiler/HipsyclClangPlugin.cpp#L106-L108

Maybe it is being registered too late for some reason?

Commenting the first line out will then fail at this line:
https://github.com/illuhad/hipSYCL/blob/f3177b86c7731c27805dfff51c1efa2e7b96cb1f/src/compiler/cbs/PipelineBuilder.cpp#L107

With llvm::InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module> was not registered prior to being queried. I couldn't find anything related to InnerAnalysisManagerProxy or FunctionAnalysisManager in this repo, except for this comment:
https://github.com/illuhad/hipSYCL/blob/f3177b86c7731c27805dfff51c1efa2e7b96cb1f/src/compiler/cbs/UniformityAnalysis.cpp#L7-L9

Commenting both out makes -fpass-plugin work, but that is probably not a good idea 😅

[aaronmondal]

Edit: The patch in my first answer actually makes the first line pass. I think adding the missing passes is the way to fix this. I'll look into how to register the InnerAnalysisManagerProxy tomorrow.

Unfortunately i made a mistake in my second answer. Commenting out the second line is actually not enough. The next failure comes from:
https://github.com/illuhad/hipSYCL/blob/f3177b86c7731c27805dfff51c1efa2e7b96cb1f/src/compiler/HipsyclClangPlugin.cpp#L117-L121
with failure [IRUnitT = llvm::Function, ExtraArgTs = <>]: PI != AnalysisPasses.end() && "Analysis passes must be registered prior to being queried!.

Commenting out the whole block works, but again, that is hardly a solution.