Mixing Clang 20 and ROOT causes very sporadic crashes

[ktf]

Check duplicate issues.

• Checked for duplicates

Description

Since we bumped to Clang 20 (while keeping everything else unchanged), we see the following stacktrace:

#0  0x000014a519934570 in cling::Interpreter::runAndRemoveStaticDestructors() () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#1  0x000014a519843d4c in TCling::ResetGlobals (this=0x1d68cd0) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/metacling/src/TCling.cxx:3756                                                                                                                                                                                                                    
#2  0x000014a52aa5ba8b in TROOT::EndOfProcessCleanups (this=0x14a52acbb500 <ROOT::Internal::GetROOT1()::alloc>) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/base/src/TROOT.cxx:1257                                                                                                                                                                          
#3  0x000014a52ab59fcf in TUnixSystem::Exit (this=<optimized out>, code=139, mode=true) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/unix/src/TUnixSystem.cxx:2191                                                                                                                                                                                            
#4  0x000014a52ab6229b in TUnixSystem::DispatchSignals (this=0x19cf920, sig=<optimized out>) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/unix/src/TUnixSystem.cxx:3675                                                                                                                                                                                       
#5  <signal handler called>
#6  std::local_Rb_tree_rotate_left (__x=0x2d977c0, __root=@0x2e16a00: 0x26f75a0) at ../../../../../gcc/libstdc++-v3/src/c++98/tree.cc:138
#7  std::_Rb_tree_insert_and_rebalance (__insert_left=<optimized out>, __x=0x26f7620, __p=<optimized out>, __header=...) at ../../../../../gcc/libstdc++-v3/src/c++98/tree.cc:278
#8  0x000014a51c51ccc1 in llvm::TargetLoweringBase::initActions() () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#9  0x000014a51c51d12f in llvm::TargetLoweringBase::TargetLoweringBase(llvm::TargetMachine const&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#10 0x000014a51c11c449 in llvm::TargetLowering::TargetLowering(llvm::TargetMachine const&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#11 0x000014a51b773b32 in llvm::X86TargetLowering::X86TargetLowering(llvm::X86TargetMachine const&, llvm::X86Subtarget const&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#12 0x000014a51b8038b1 in llvm::X86Subtarget::X86Subtarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::StringRef, llvm::X86TargetMachine const&, llvm::MaybeAlign, unsigned int, unsigned int) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                                                                                                                                            
#13 0x000014a51b530202 in llvm::X86TargetMachine::getSubtargetImpl(llvm::Function const&) const () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#14 0x000014a51b5304ed in llvm::X86TargetMachine::getTargetTransformInfo(llvm::Function const&) const () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#15 0x000014a51c7bec8a in std::_Function_handler<llvm::TargetTransformInfo (llvm::Function const&), llvm::TargetMachine::getTargetIRAnalysis() const::{lambda(llvm::Function const&)#1}>::_M_invoke(std::_Any_data const&, llvm::Function const&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                                                                                                          
#16 0x000014a51d748892 in llvm::TargetIRAnalysis::run(llvm::Function const&, llvm::AnalysisManager<llvm::Function>&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#17 0x000014a51b3cbb7e in llvm::detail::AnalysisPassModel<llvm::Function, llvm::TargetIRAnalysis, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>::Invalidator>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                                                                                                               
#18 0x000014a51decc29c in llvm::AnalysisManager<llvm::Function>::getResultImpl(llvm::AnalysisKey*, llvm::Function&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#19 0x000014a51d04eb0c in llvm::SimplifyCFGPass::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#20 0x000014a51b3cca3d in llvm::detail::PassModel<llvm::Function, llvm::SimplifyCFGPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                                                                                                                                     
#21 0x000014a519d6e374 in llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                                                                                  
#22 0x000014a51ded0551 in llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#23 0x000014a519d6715d in llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                                                                                                                                 
#24 0x000014a519a0d8ee in cling::BackendPasses::runOnModule(llvm::Module&, int) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#25 0x000014a5199bd0a4 in cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#26 0x000014a519934a48 in cling::Interpreter::executeTransaction(cling::Transaction&) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so
#27 0x000014a5199cef33 in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                               
#28 0x000014a519939d54 in cling::Interpreter::Interpreter(int, char const* const*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, void*, bool, cling::Interpreter const*) () from /opt/alisw/el9/ROOT/v6-32-06-alice9-8/lib/libCling.so                                                                                                    
#29 0x000014a519879acf in cling::Interpreter::Interpreter (this=0x1a5bad0, argc=77, argv=0x1d69850, llvmdir=<optimized out>, moduleExtensions=..., extraLibHandle=0x1d33a50, noRuntime=false) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/interpreter/cling/include/cling/Interpreter/Interpreter.h:368                                                           
#30 std::make_unique<cling::Interpreter, unsigned long, char const**, TString&, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > >&, void*&> () at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-2/include/c++/14.2.0/bits/unique_ptr.h:1076                                                          
#31 TCling::TCling (this=0x1d68cd0, name=<optimized out>, title=<optimized out>, argv=<optimized out>, interpLibHandle=<optimized out>) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/metacling/src/TCling.cxx:1543                                                                                                                                            
#32 0x000014a51987b403 in CreateInterpreter (interpLibHandle=0x1d33a50, argv=argv@entry=0x7ffefbe30f80) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/metacling/src/TCling.cxx:611                                                                                                                                                                             
#33 0x000014a52aa5d50e in TROOT::InitInterpreter (this=0x14a52acbb500 <ROOT::Internal::GetROOT1()::alloc>) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/base/src/TROOT.cxx:2126                                                                                                                                                                               
#34 0x000014a52aa5d88f in ROOT::Internal::GetROOT2 () at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/base/src/TROOT.cxx:391
#35 0x000014a52ab05a37 in TClass::GetClass (typeinfo=..., load=load@entry=true, hint_pair_offset=hint_pair_offset@entry=0, hint_pair_size=hint_pair_size@entry=0) at /local/workspace/DailyBuilds/WeeklyO2Release/daily-tags.eyARqWDKEt/SOURCES/ROOT/v6-32-06-alice9/v6-32-06-alice9/core/meta/src/TClass.cxx:3256                                                                                                                       
#36 0x000014a52c763378 in o2::conf::ConfigurableParamHelper<o2::conf::KeyValParam>::putKeyValues (this=0x14a52c7fdd80 <o2::conf::KeyValParam::sInstance>, tree=0x1ad8c50) at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Common/Utils/include/CommonUtils/ConfigurableParamHelper.h:152                                                                                        
#37 o2::conf::ConfigurableParam::initPropertyTree () at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Common/Utils/src/ConfigurableParam.cxx:249
#38 0x000014a52c763483 in o2::conf::ConfigurableParam::initialize () at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Common/Utils/src/ConfigurableParam.cxx:341                                                                                                                                                                                                                 
#39 0x000014a52c76ab55 in o2::conf::ConfigurableParam::updateFromString (configString=...) at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Common/Utils/src/ConfigurableParam.cxx:436                                                                                                                                                                                           
#40 0x000000000040725d in defineDataProcessing (cfgc=...) at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Detectors/TRD/workflow/src/entropy-encoder-workflow.cxx:39                                                                                                                                                                                                            
#41 0x000000000040d942 in mainNoCatch (argc=<optimized out>, argv=<optimized out>) at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Framework/Core/include/Framework/runDataProcessing.h:197                                                                                                                                                                                     
#42 0x000014a52ef123b1 in callMain (argc=argc@entry=130, argv=argv@entry=0x7ffefbe31f48, mainNoCatch=mainNoCatch@entry=0x40d370 <mainNoCatch(int, char**)>) at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Framework/Core/src/runDataProcessing.cxx:202                                                                                                                        
#43 0x0000000000406429 in main (argc=130, argv=0x7ffefbe31f48) at /local/workspace/DailyBuilds/DailyO2epn/daily-tags.uPxjhOj5Sp/SOURCES/O2/epn-20250910/rc/epn-20250910/Framework/Core/include/Framework/runDataProcessing.h:218       

when running in an environment which has both ROOT 6.32.06 and Clang 20 in it (even if they should not interfere, as we understand it). Is there some new "cross talk" which might break things? Could you confirm that ROOT clang should be 100% hidden from outside interference?

Reproducer

Reproducer is complicated. It happens very rarely (say once every 10000 spawn processes) when we run in data taking. The problem is that this is preventing to reliably start the data taking topology with the new Clang.

ROOT version

6.32.06

Installation method

aliBuild

Operating system

Linux

Additional context

No response

[ktf]

@ehellbar @sawenzel

[ktf]

@silverweed

[silverweed]

cc @devajithvs @hahnjo

[hahnjo]

[...] when running in an environment which has both ROOT 6.32.06 and Clang 20 in it (even if they should not interfere, as we understand it).

Just to clarify, you have LLVM + Clang 20 linked into the executable / process, so it's not about compiling with Clang 20?

Is there some new "cross talk" which might break things? Could you confirm that ROOT clang should be 100% hidden from outside interference?

Yes, ROOT's LLVM + Clang should be "confined" into libCling.so and all symbols "hidden" from the outer world. Do you statically link LLVM + Cling into the executable or into a shared library? (We recently had a case where some of the hiding didn't work when some static initialization functions with C linkage were present in the executable because then they somehow took precedence (sometimes). I would have to dig up the details.)

Reproducer is complicated. It happens very rarely (say once every 10000 spawn processes) when we run in data taking. The problem is that this is preventing to reliably start the data taking topology with the new Clang.

This is even more curious. In principle, I would expect the dynamic linker to be reproducible and the choice of which function to execute not be subject to ASLR. I suppose you already looked with the usual tools such as Valgrind?

[ktf]

Just to clarify, you have LLVM + Clang 20 linked into the executable / process, so it's not about compiling with Clang 20?

Correct. LLVM gets loaded because we have something unrelated linking against it.

Do you statically link LLVM + Cling into the executable or into a shared library?

Cling is probably linked in the executable and in some shared libraries, our clang 20 LLVM should be in a shared library only.

[ktf]

This is even more curious. In principle, I would expect the dynamic linker to be reproducible and the choice of which function to execute not be subject to ASLR. I suppose you already looked with the usual tools such as Valgrind?

We can try with valgrind, however it's complicated to instrument the whole staging setup with it (and the crash, as I said, seems random). Maybe indeed we could see if echo 0 | sudo tee /proc/sys/kernel/randomize_va_space makes it fully reproducible...

[hahnjo]

Another debugging tool could be LD_DEBUG=bindings (potentially with LD_DEBUG_OUTPUT). It shows which symbol from which file is resolved to which library. If in that (long) dump you find a function that "crosses" LLVM versions, we have a problem...

[sawenzel]

@hahnjo : Following your suggestion, I ran a simple command and checked the output. Not sure if this is what you are after but it seems that libCling.so binds symbols from the external libLLVM.so.

ldd does not mention the dependency:

/workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib $> ldd libCling.so           
        linux-vdso.so.1 (0x00007ffe9658c000)
        libz.so.1 => /workdir/alisw/sw/slc9_x86-64/AliEn-Runtime/v2-19-le-20/lib/libz.so.1 (0x00007fe49c090000)
        libzstd.so.1 => /lib64/libzstd.so.1 (0x00007fe49bfb1000)
        libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007fe49bf81000)
        libstdc++.so.6 => /workdir/alisw/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib64/libstdc++.so.6 (0x00007fe49bd0f000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fe49bc32000)
        libgcc_s.so.1 => /workdir/alisw/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib64/libgcc_s.so.1 (0x00007fe49bc04000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe49b9fc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe4a2939000)

but the binding logs mention things like this :

 binding file /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so [0] to /workdir/alisw/sw/slc9_x86-64/Clang/v20.1.7-1/lib/libLLVM.so.20.1 [0]: normal symbol `_ZTSSt8functionIFviEE'
(many more symbols follow)

[sawenzel]

A valgrind run (valgrind o2-sim-serial -n 0 -m ITS -e TGeant3 -o bar) also show this mixing:

==29882== Conditional jump or move depends on uninitialised value(s)
==29882==    at 0x13D49F8C: std::_Rb_tree<std::pair<unsigned int, llvm::MVT::SimpleValueType>, std::pair<std::pair<unsigned int, llvm::MVT::SimpleValueType> const, llvm::MVT::SimpleValueType>, std::_Select1st<std::pair<std::pair<unsigned int, llvm::MVT::SimpleValueType> const, llvm::MVT::SimpleValueTyp
e> >, std::less<std::pair<unsigned int, llvm::MVT::SimpleValueType> >, std::allocator<std::pair<std::pair<unsigned int, llvm::MVT::SimpleValueType> const, llvm::MVT::SimpleValueType> > >::_M_get_insert_unique_pos(std::pair<unsigned int, llvm::MVT::SimpleValueType> const&) [clone .isra.0] (in /workdir/a
lisw/sw/slc9_x86-64/Clang/v20.1.7-1/lib/libLLVM.so.20.1)
==29882==    by 0x1CF04C99: llvm::TargetLoweringBase::initActions() (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1CF0512E: llvm::TargetLoweringBase::TargetLoweringBase(llvm::TargetMachine const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1CB04448: llvm::TargetLowering::TargetLowering(llvm::TargetMachine const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1C15BB31: llvm::X86TargetLowering::X86TargetLowering(llvm::X86TargetMachine const&, llvm::X86Subtarget const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1C1EB8B0: llvm::X86Subtarget::X86Subtarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::StringRef, llvm::X86TargetMachine const&, llvm::MaybeAlign, unsigned int, unsigned int) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1BF18201: llvm::X86TargetMachine::getSubtargetImpl(llvm::Function const&) const (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1BF184EC: llvm::X86TargetMachine::getTargetTransformInfo(llvm::Function const&) const (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1D1A6C89: std::_Function_handler<llvm::TargetTransformInfo (llvm::Function const&), llvm::TargetMachine::getTargetIRAnalysis() const::{lambda(llvm::Function const&)#1}>::_M_invoke(std::_Any_data const&, llvm::Function const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/li
b/libCling.so.6.32.06)
==29882==    by 0x1E130891: llvm::TargetIRAnalysis::run(llvm::Function const&, llvm::AnalysisManager<llvm::Function>&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1BDB3B7D: llvm::detail::AnalysisPassModel<llvm::Function, llvm::TargetIRAnalysis, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>::Invalidator>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/lib
Cling.so.6.32.06)
==29882==    by 0x1E8B429B: llvm::AnalysisManager<llvm::Function>::getResultImpl(llvm::AnalysisKey*, llvm::Function&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29

libCling calls a std::_Rb_tree<...> in libLLVM.

[ehellbar]

thanks @sawenzel , I was about to post the same (valgrind output). Running with a software stack that uses clang18, we don't see these messages from valgrind.

[hahnjo]

ldd does not mention the dependency:

Yes, libCling.so does not link against external libLLVM.so, it statically links in our version of LLVM and Clang that we build. And in principle its symbols should be completely hidden.

Following your suggestion, I ran a simple command and checked the output. Not sure if this is what you are after but it seems that libCling.so binds symbols from the external libLLVM.so.

[...] the binding logs mentions things like this :

 binding file /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so [0] to /workdir/alisw/sw/slc9_x86-64/Clang/v20.1.7-1/lib/libLLVM.so.20.1 [0]: normal symbol `_ZTSSt8functionIFviEE'
(many more symbols follow)

In principle, this specific symbol is fine because it's from the STL and doesn't depend on LLVM types.

A valgrind run (valgrind o2-sim-serial -n 0 -m ITS -e TGeant3 -o bar) also show this mixing:

==29882== Conditional jump or move depends on uninitialised value(s)
==29882==    at 0x13D49F8C: std::_Rb_tree<std::pair<unsigned int, llvm::MVT::SimpleValueType>, std::pair<std::pair<unsigned int, llvm::MVT::SimpleValueType> const, llvm::MVT::SimpleValueType>, std::_Select1st<std::pair<std::pair<unsigned int, llvm::MVT::SimpleValueType> const, llvm::MVT::SimpleValueTyp
e> >, std::less<std::pair<unsigned int, llvm::MVT::SimpleValueType> >, std::allocator<std::pair<std::pair<unsigned int, llvm::MVT::SimpleValueType> const, llvm::MVT::SimpleValueType> > >::_M_get_insert_unique_pos(std::pair<unsigned int, llvm::MVT::SimpleValueType> const&) [clone .isra.0] (in /workdir/a
lisw/sw/slc9_x86-64/Clang/v20.1.7-1/lib/libLLVM.so.20.1)
==29882==    by 0x1CF04C99: llvm::TargetLoweringBase::initActions() (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1CF0512E: llvm::TargetLoweringBase::TargetLoweringBase(llvm::TargetMachine const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1CB04448: llvm::TargetLowering::TargetLowering(llvm::TargetMachine const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1C15BB31: llvm::X86TargetLowering::X86TargetLowering(llvm::X86TargetMachine const&, llvm::X86Subtarget const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1C1EB8B0: llvm::X86Subtarget::X86Subtarget(llvm::Triple const&, llvm::StringRef, llvm::StringRef, llvm::StringRef, llvm::X86TargetMachine const&, llvm::MaybeAlign, unsigned int, unsigned int) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1BF18201: llvm::X86TargetMachine::getSubtargetImpl(llvm::Function const&) const (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1BF184EC: llvm::X86TargetMachine::getTargetTransformInfo(llvm::Function const&) const (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1D1A6C89: std::_Function_handler<llvm::TargetTransformInfo (llvm::Function const&), llvm::TargetMachine::getTargetIRAnalysis() const::{lambda(llvm::Function const&)#1}>::_M_invoke(std::_Any_data const&, llvm::Function const&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/li
b/libCling.so.6.32.06)
==29882==    by 0x1E130891: llvm::TargetIRAnalysis::run(llvm::Function const&, llvm::AnalysisManager<llvm::Function>&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29882==    by 0x1BDB3B7D: llvm::detail::AnalysisPassModel<llvm::Function, llvm::TargetIRAnalysis, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>::Invalidator>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/lib
Cling.so.6.32.06)
==29882==    by 0x1E8B429B: llvm::AnalysisManager<llvm::Function>::getResultImpl(llvm::AnalysisKey*, llvm::Function&) (in /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so.6.32.06)
==29

libCling calls a std::_Rb_tree<...> in libLLVM.

However, this STL function is not fine (if it also appears in the binding, can you check?): the value type llvm::MVT::SimpleValueType is an enum that changed its underlying type from std::uint8_t to std::uint16_t in commit llvm/llvm-project@a4c6ebe included in LLVM 20!! This explains the memory issues when mixing these STL instantiations between two LLVM versions...

[sawenzel]

@hahnjo : Yes, I also see these bindings in the LD_DEBUG log:

grep "libLLVM" ld_bindings.2*  | grep "tree" | grep "libClin"
ld_bindings.24798:     24798:   binding file /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so [0] to /workdir/alisw/sw/slc9_x86-64/Clang/v20.1.7-1/lib/libLLVM.so.20.1 [0]: normal symbol `_ZNSt8_Rb_treeISt4pairIjN4llvm3MVT15SimpleValueTypeEES0_IKS4_S3_ESt10_Select1stIS6_ESt4lessIS4_ESaIS6_EE29_M_get_insert_hint_unique_posESt23_Rb_tree_const_iteratorIS6_ERS5_'
ld_bindings.29833:     29833:   binding file /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so [0] to /workdir/alisw/sw/slc9_x86-64/Clang/v20.1.7-1/lib/libLLVM.so.20.1 [0]: normal symbol `_ZNSt8_Rb_treeISt4pairIjN4llvm3MVT15SimpleValueTypeEES0_IKS4_S3_ESt10_Select1stIS6_ESt4lessIS4_ESaIS6_EE29_M_get_insert_hint_unique_posESt23_Rb_tree_const_iteratorIS6_ERS5_'
ld_bindings.29882:     29882:   binding file /workdir/alisw/sw/slc9_x86-64/ROOT/v6-32-06-alice9-7/lib/libCling.so [0] to /workdir/alisw/sw/slc9_x86-64/Clang/v20.1.7-1/lib/libLLVM.so.20.1 [0]: normal symbol `_ZNSt8_Rb_treeISt4pairIjN4llvm3MVT15SimpleValueTypeEES0_IKS4_S3_ESt10_Select1stIS6_ESt4lessIS4_ESaIS6_EE29_M_get_insert_hint_unique_posESt23_Rb_tree_const_iteratorIS6_ERS5_'