Segfault on hsimple.C in user-mode qemu cross-platform linux/arm64/v8 docker installation

[wdconinc]

Check duplicate issues.

• Checked for duplicates

Description

As of e9d264e, building ROOT inside a docker container for arm64 architectures on a x86_64 host system using qemu emulation fails on the hsimple.C test:

Consolidate compiler generated dependencies of target GeomBuilder
make[2]: Entering directory '/mnt/root-build-ubuntu'
make[2]: Leaving directory '/mnt/root-build-ubuntu'
[100%] Generating etc/allDict.cxx.pch
make[2]: Entering directory '/mnt/root-build-ubuntu'
[100%] Linking CXX shared library ../../lib/libGeomBuilder.so

Generating PCH for core core/clingutils core/imt core/multiproc core/rint core/thread graf2d/gpad graf2d/graf graf2d/postscript graf3d/g3d gui/fitpanel gui/gui hist/hist hist/histpainter io/io math/genvector math/mathcore math/matrix math/minuit math/physics math/smatrix math/vecops net/net tree/tree tree/treeplayer tree/treeviewer

make[2]: Leaving directory '/mnt/root-build-ubuntu'
[100%] Built target GeomBuilder
make[2]: Leaving directory '/mnt/root-build-ubuntu'
[100%] Built target onepcm
make[2]: Entering directory '/mnt/root-build-ubuntu'
make[2]: Leaving directory '/mnt/root-build-ubuntu'
make[2]: Entering directory '/mnt/root-build-ubuntu'
[100%] Generating tutorials/hsimple.root

Processing hsimple.C...

 *** Break *** segmentation violation
 Generating stack trace...
 0x000000550424be2c in <unknown> from /mnt/root-build-ubuntu/lib/libCling.so
 0x000000550424c598 in <unknown> from /mnt/root-build-ubuntu/lib/libCling.so
 0x000000550432bef4 in <unknown> from /mnt/root-build-ubuntu/lib/libCling.so
 0x0000005504338200 in <unknown> from /mnt/root-build-ubuntu/lib/libCling.so
 0x000000550433a1d4 in <unknown> from /mnt/root-build-ubuntu/lib/libCling.so
 0x00000055043253ec in <unknown> from /mnt/root-build-ubuntu/lib/libCling.so
 0x000000550415c38c in <unknown> from /mnt/root-build-ubuntu/lib/libCling.so
 0x0000005504174cf4 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) + 0xdb4 from /mnt/root-build-ubuntu/lib/libCling.so
 0x000000550417514c in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) + 0xe8 from /mnt/root-build-ubuntu/lib/libCling.so
 0x0000005502a31224 in TApplication::ExecuteFile(char const*, int*, bool) at TApplication.cxx:? from /mnt/root-build-ubuntu/lib/libCore.so
 0x00000055028712f0 in TRint::ProcessLineNr(char const*, char const*, int*) + 0x7c from /mnt/root-build-ubuntu/lib/libRint.so
 0x00000055028731ec in TRint::Run(bool) + 0x2d8 from /mnt/root-build-ubuntu/lib/libRint.so
 0x0000005500000fe8 in main + 0x54 from /mnt/root-build-ubuntu/bin/root.exe
 0x00000055030c73fc in <unknown> from /lib/aarch64-linux-gnu/libc.so.6
 0x00000055030c74cc in __libc_start_main + 0x98 from /lib/aarch64-linux-gnu/libc.so.6
 0x0000005500001070 in _start + 0x30 from /mnt/root-build-ubuntu/bin/root.exe
Root > make[2]: *** [CMakeFiles/hsimple.dir/build.make:79: tutorials/hsimple.root] Error 129
make[2]: Target 'CMakeFiles/hsimple.dir/build' not remade because of errors.
make[2]: Leaving directory '/mnt/root-build-ubuntu'
make[1]: *** [CMakeFiles/Makefile2:7589: CMakeFiles/hsimple.dir/all] Error 2
make[1]: Target 'all' not remade because of errors.
make[1]: Leaving directory '/mnt/root-build-ubuntu'
make: *** [Makefile:156: all] Error 2
make: Target 'default_target' not remade because of errors.
make: Leaving directory '/mnt/root-build-ubuntu'

Reproducer

# Enable multiarch support for docker (https://hub.docker.com/r/multiarch/qemu-user-static/)
docker run --privileged --rm multiarch/qemu-user-static --reset --persistent yes
# Start ubuntu docker container 
docker run --platform linux/arm64/v8 --rm -it ubuntu:latest
# Install minimal dependencies
apt-get update
apt-get install build-essential git cmake python3
# Clone ROOT to commit just before problem
git clone https://github.com/root-project/root
git -C root reset --hard a886646
# Apply unrelated patch (see below)
sed 's/^    \([^s].*\.cxx\)/    src\/\1/g' hist/hist/CMakeLists.txt
# Configure and build (took 10 hours)
cmake -Broot-build -Sroot -Dasimage=OFF -Dclad=OFF -Ddataframe=OFF -Ddavix=OFF -Dfftw3=OFF -Dgdml=off -Dhttp=off -Dimt=OFF -Dmathmore=OFF -Dminuit2=OFF -Dmlp=OFF -Dopengl=OFF -Dpyroot=OFF -Dpythia8=OFF -Droofit=OFF -Dwebgui=OFF -Droot7=OFF -Dspectrum=OFF -Dsqlite=OFF -Dssl=OFF -Dtmva=OFF -Dtmva-cpu=OFF -Dvdt=OFF -Dx11=OFF -Dxml=OFF -Dxrootd=OFF -DCMAKE_INSTALL_PREFIX=root-install
make -Croot-build -j7 install
# Reset ROOT to next commit, which introduced the problem 
git -C root reset --hard e9d264e
# Apply unrelated patch (see below)
sed 's/^    \([^s].*\.cxx\)/    src\/\1/g' hist/hist/CMakeLists.txt
make -Croot-build -j7 install
# This last command fails

ROOT version

e9d264e, i.e. v6-27-02~243^2~17

Installation method

git clone, cmake, make

Operating system

Linux Ubuntu 23.04

Additional context

Why would you want to do that???

A valid question is why one would want to build ROOT inside a qemu/docker environment where a foreign architecture is emulated? Well, just for this PR in fact, to make it self-contained and (somewhat) debuggable. You can also build a container natively on ARM, upload to docker, and atempt to run ROOT inside that container on a qemu/docker arm64 emulation, with the same results.

Still, why would you want to do that???

We have provided ARM containers for our users at the Electron-Ion Collider for a while now. In the absence of ARM nodes on our container build cluster, we build these using emulation. Aside from a rare major dependency recompile, this works well enough. As of ROOT v6.28 we have had to stop providing ARM images for our users because the C++20 feature of the upgrade was more important than the continued ARM support. Nevertheless, we would like to bring back ARM support.

[wdconinc]

I forgot to add that there is a small patch to one CMakeLists.txt file needed:

sed 's/^    \([^s].*\.cxx\)/    src\/\1/g' hist/hist/CMakeLists.txt

This adds an explicit src/ before all cxx files in hist/hist/CMakeLists.txt where none are added. Without this, CMake fails to find the correct files. We are not using ROOT7, so this doens't affect any differences.

Patch included in reproducer above.

[Axel-Naumann]

Thanks, Wouter. I will look at this likely in 10 days. If you have a moment before, could you configure with -DCMAKE_BUILD_TYPE=Debug -DLLVM_BUILD_TYPE=Debug and post the updated backtrace?

I find it very suspicious that you need to patch hist/hist/CMakeLists.txt when all our builds work...

Does it help to build with the default -Druntime_cxxmodules=On?

[wdconinc]

Thanks for the response. The patching of the CMakeLists.txt is unrelated and also needed before this specific commit. It's essentially due to cmake often but not always failing to return any files on certain globs here when the file itself doesn't exist yet, but again only in qemu/docker. It's suspicious, not very reproducible, hard to debug, but pretty sure it is not an issue with ROOT.

I'll give the debug build a try today. I should have done that before reporting. Attaching debuggers inside a qemu emulated docker container is a bit limited unless one can run docker in privileged mode with cap ptrace added. Our usual build system doesn't have that capability so I'm used to this issue not resulting in any backtrace, but on my laptop I can enable that (as I did for this issue).

I'll also try with/without runtime_cxxmodules.

[Axel-Naumann]

You can just install gdb in your VM and we should get a useful backtrace. Thanks for being so supportive!

[wdconinc]

Apologies for the delay. Turning on LLVM debug really increases the build size, so I had to turn my test setup upside down.

In any case, building with LLVM debug in particular triggers an assert to fail before the segfault happens, both with and without runtime_cxxmodules, and both before and after the commit I had identified before:

make[2]: Entering directory '/data/root-build-runtime_cxxmodules-ON'
[100%] Generating lib/modules.idx
cd /mnt/root-build-runtime_cxxmodules-ON/lib && /usr/bin/cmake -E remove -f modules.idx modules.timestamp
cd /mnt/root-build-runtime_cxxmodules-ON/lib && LD_LIBRARY_PATH=/mnt/root-build-runtime_cxxmodules-ON/lib: ROOTIGNOREPREFIX=1 ROOT_HIST=0 /mnt/root-build-runtime_cxxmodules-ON/bin/root.exe -l -q -b
resolveAArch64Relocation, LocalAddress: 0x55849bc010 FinalAddress: 0x55849bc010 Value: 0x7f7a5f8eb000 Type: 0x113 Addend: 0x0
root.exe: /mnt/root/interpreter/llvm/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:488: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry&, uint64_t, uint64_t, uint32_t, int64_t): Assertion `isInt<33>(Result) && "overflow check failed for relocation"' failed.
Aborted (core dumped)
make[2]: *** [CMakeFiles/modules_idx.dir/build.make:145: lib/modules.idx] Error 134
make[2]: Leaving directory '/data/root-build-runtime_cxxmodules-ON'
make[1]: *** [CMakeFiles/Makefile2:7643: CMakeFiles/modules_idx.dir/all] Error 2
make[1]: Leaving directory '/data/root-build-runtime_cxxmodules-ON'
make: *** [Makefile:156: all] Error 2
make: Leaving directory '/data/root-build-runtime_cxxmodules-ON'

(The same happens with runtime_cxxmodules=OFF, just with different addresses.)

The resolveAArch64Relocation diagnostic output is added by (since I only wanted debugging for the failing assert):

diff --git a/interpreter/llvm/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp b/interpreter/llvm/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
index efe0b9cd61..9d7e5b896d 100644
--- a/interpreter/llvm/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
+++ b/interpreter/llvm/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
@@ -10,6 +10,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include <iostream>
 #include "RuntimeDyldELF.h"
 #include "RuntimeDyldCheckerImpl.h"
 #include "Targets/RuntimeDyldELFMips.h"
@@ -476,6 +477,14 @@ void RuntimeDyldELF::resolveAArch64Relocation(const SectionEntry &Section,
         ((Value + Addend) & ~0xfffULL) - (FinalAddress & ~0xfffULL);
 
     // Check that -2^32 <= X < 2^32
+    if (! isInt<33>(Result)) {
+      outs() << "resolveAArch64Relocation, LocalAddress: 0x"
+             << format("%llx", Section.getAddressWithOffset(Offset))
+             << " FinalAddress: 0x" << format("%llx", FinalAddress)
+             << " Value: 0x" << format("%llx", Value) << " Type: 0x"
+             << format("%x", Type) << " Addend: 0x"
+             << format("%llx", Addend) << "\n";
+    }
     assert(isInt<33>(Result) && "overflow check failed for relocation");
 
     // Immediate goes in bits 30:29 + 5:23 of ADRP instruction, taken

Using gdb proved to be fruitless in this environment.

• Just gdb fails due to the absence of ptrace, even with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined:

# ROOTIGNOREPREFIX=1 ROOT_HIST=0 gdb -ex run --args /mnt/root-build-runtime_cxxmodules-ON/bin/root.exe -l -q -b
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /mnt/root-build-runtime_cxxmodules-ON/bin/root.exe...
Starting program: /data/root-build-runtime_cxxmodules-ON/bin/root.exe -l -q -b
warning: Could not trace the inferior process.
warning: ptrace: Function not implemented
During startup program exited with code 127.

• After apt-get install qemu-user-static, there is a gdb hook in an explicit qemu call, and with docker cp sad_gauss:/lib/ . I can copy the necessary aarch64 libs over:

# ROOTIGNOREPREFIX=1 ROOT_HIST=0 qemu-aarch64-static -g 1234 /mnt/root-build-runtime_cxxmodules-ON/bin/root.exe -l -q -b
^C^\

but doesn't crash, and it just keep spinning at 100% CPU for a few hours, and requires a SIGINT to get a backtrace, which we can view outside the container:

$ gdb-multiarch -q --nh -ex 'file ../root-build-runtime_cxxmodules-ON/bin/root.exe' -ex 'set architecture aarch64' -ex 'set solib-search-path ../root-build-runtime_cxxmodules-ON/lib:./lib' -ex 'target remote 172.17.0.3:1234' -ex 'set auto-load safe-path /' -ex 'continue'
Reading symbols from ../root-build-runtime_cxxmodules-ON/bin/root.exe...
The target architecture is set to "aarch64".
Remote debugging using 172.17.0.3:1234
warning: remote target does not support file transfer, attempting to access files from local filesystem.
Reading symbols from /home/wdconinc/git/root/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) n
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
(No debugging symbols found in /home/wdconinc/git/root/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1)
0x00007f6925dc2c40 in ?? () from /home/wdconinc/git/root/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
Continuing.

Program received signal SIGINT, Interrupt.
0x00007f6908203f2c in llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >::getNumBuckets (this=0x7f6917ca5868)
    at /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h:1168
warning: 1168   /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h: No such file or directory
(gdb) bt
#0  0x00007f6908203f2c in llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >::getNumBuckets (this=0x7f6917ca5868)
    at /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h:1168
#1  0x00007f69081ee0e0 in llvm::DenseMapBase<llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >, clang::Decl*, unsigned int, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >::getNumBuckets (this=0x7f6917ca5868) at /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h:525
#2  0x00007f6908278b6c in llvm::DenseMapBase<llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >, clang::Decl*, unsigned int, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >::InsertIntoBucketImpl<clang::Decl*> (this=0x7f6917ca5868, Key=@0x7f6918c65f70: 0x7f68f85b4988, Lookup=@0x7f6918c65f70: 0x7f68f85b4988, TheBucket=0x7f68f9e9e0e0)
    at /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h:579
#3  0x00007f6908270f58 in llvm::DenseMapBase<llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >, clang::Decl*, unsigned int, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >::InsertIntoBucket<clang::Decl* const&, unsigned int const&> (this=0x7f6917ca5868, TheBucket=0x7f68f9e9e0e0, Key=@0x7f6918c65f70: 0x7f68f85b4988)
    at /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h:547
#4  0x00007f6908268c34 in llvm::DenseMapBase<llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >, clang::Decl*, unsigned int, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >::try_emplace<unsigned int const&> (this=0x7f6917ca5868, Key=@0x7f6918c65f70: 0x7f68f85b4988) at /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h:258
#5  0x00007f690825d3fc in llvm::DenseMapBase<llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >, clang::Decl*, unsigned int, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >::insert (this=0x7f6917ca5868, KV=...) at /mnt/root/interpreter/llvm/src/include/llvm/ADT/DenseMap.h:208
#6  0x00007f69082509ac in llvm::MapVector<clang::Decl*, unsigned long, llvm::SmallDenseMap<clang::Decl*, unsigned int, 4u, llvm::DenseMapInfo<clang::Decl*>, llvm::detail::DenseMapPair<clang::Decl*, unsigned int> >, llvm::SmallVector<std::pair<clang::Decl*, unsigned long>, 4u> >::operator[] (this=0x7f6917ca5868, Key=@0x7f6918c65fd8: 0x7f68f85b4988) at /mnt/root/interpreter/llvm/src/include/llvm/ADT/MapVector.h:100
#7  0x00007f69082263d8 in clang::ASTDeclReader::ReadFunctionDefinition (this=0x7f6918c66130, FD=0x7f68f85b4988) at /mnt/root/interpreter/llvm/src/tools/clang/lib/Serialization/ASTReaderDecl.cpp:529
#8  0x00007f6908226624 in clang::ASTDeclReader::Visit (this=0x7f6918c66130, D=0x7f68f85b4988) at /mnt/root/interpreter/llvm/src/tools/clang/lib/Serialization/ASTReaderDecl.cpp:564
#9  0x00007f6908245f10 in clang::ASTReader::ReadDeclRecord (this=0x7f6917ca49b0, ID=366988) at /mnt/root/interpreter/llvm/src/tools/clang/lib/Serialization/ASTReaderDecl.cpp:4120
#10 0x00007f6908176fa0 in clang::ASTReader::GetDecl (this=0x7f6917ca49b0, ID=366988) at /mnt/root/interpreter/llvm/src/tools/clang/lib/Serialization/ASTReader.cpp:7531
#11 0x00007f690819cda8 in clang::ASTReader::GetLocalDecl (this=0x7f6917ca49b0, F=..., LocalID=366988) at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/Serialization/ASTReader.h:1856
#12 0x00007f690817749c in operator() (__closure=0x7f6918c66568, M=0x7f69176b1f90, LexicalDecls=...) at /mnt/root/interpreter/llvm/src/tools/clang/lib/Serialization/ASTReader.cpp:7611
#13 0x00007f69081776ac in clang::ASTReader::FindExternalLexicalDecls(clang::DeclContext const*, llvm::function_ref<bool (clang::Decl::Kind)>, llvm::SmallVectorImpl<clang::Decl*>&) (this=0x7f6917ca49b0, DC=0x7f68fdb16850, 
    IsKindWeWant=..., Decls=...) at /mnt/root/interpreter/llvm/src/tools/clang/lib/Serialization/ASTReader.cpp:7625
#14 0x00007f690730267c in cling::ExternalASTSourceWrapper::FindExternalLexicalDecls(clang::DeclContext const*, llvm::function_ref<bool (clang::Decl::Kind)>, llvm::SmallVectorImpl<clang::Decl*>&) (this=0x7f6917ca84a0, 
    DC=0x7f68fdb16850, IsKindWeWant=..., Result=...) at /mnt/root/interpreter/cling/lib/Interpreter/InterpreterCallbacks.cpp:170
#15 0x00007f69083873b4 in clang::MultiplexExternalSemaSource::FindExternalLexicalDecls(clang::DeclContext const*, llvm::function_ref<bool (clang::Decl::Kind)>, llvm::SmallVectorImpl<clang::Decl*>&) (this=0x7f6918df92d0, 
    DC=0x7f68fdb16850, IsKindWeWant=..., Result=...) at /mnt/root/interpreter/llvm/src/tools/clang/lib/Sema/MultiplexExternalSemaSource.cpp:121
#16 0x00007f690b4f0ed4 in clang::ExternalASTSource::FindExternalLexicalDecls (this=0x7f6918df92d0, DC=0x7f68fdb16850, Result=...) at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/AST/ExternalASTSource.h:185
#17 0x00007f690b4ee6e0 in clang::DeclContext::LoadLexicalDeclsFromExternalStorage (this=0x7f68fdb16850) at /mnt/root/interpreter/llvm/src/tools/clang/lib/AST/DeclBase.cpp:1356
#18 0x00007f690b4ee9a0 in clang::DeclContext::decls_begin (this=0x7f68fdb16850) at /mnt/root/interpreter/llvm/src/tools/clang/lib/AST/DeclBase.cpp:1412
#19 0x00007f6906ea3bdc in clang::DeclContext::decls (this=0x7f68fdb16850) at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/AST/DeclBase.h:2109
#20 0x00007f69070727e4 in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseDeclContextHelper(clang::DeclContext *) (this=0x7f6918c66d10, DC=0x7f68fdb16850)
    at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/AST/RecursiveASTVisitor.h:1393
#21 0x00007f6907051440 in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseCXXRecordDecl(clang::CXXRecordDecl *) (this=0x7f6918c66d10, D=0x7f68fdb16810)
    at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/AST/RecursiveASTVisitor.h:1908
#22 0x00007f6907048fe4 in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseDecl(clang::Decl *) (this=0x7f6918c66d10, D=0x7f68fdb16810)
    at /mnt/root-build-runtime_cxxmodules-ON/interpreter/llvm/src/tools/clang/include/clang/AST/DeclNodes.inc:283
#23 0x00007f69070500a8 in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseClassTemplateDecl(clang::ClassTemplateDecl *) (this=0x7f6918c66d10, D=0x7f68fdb166d8)
    at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/AST/RecursiveASTVisitor.h:1785
#24 0x00007f6907048eb0 in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseDecl(clang::Decl *) (this=0x7f6918c66d10, D=0x7f68fdb166d8)
    at /mnt/root-build-runtime_cxxmodules-ON/interpreter/llvm/src/tools/clang/include/clang/AST/DeclNodes.inc:227
#25 0x00007f690707284c in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseDeclContextHelper(clang::DeclContext *) (this=0x7f6918c66d10, DC=0x7f6918df4740)
    at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/AST/RecursiveASTVisitor.h:1395
#26 0x00007f69070583d4 in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseTranslationUnitDecl(clang::TranslationUnitDecl *) (this=0x7f6918c66d10, D=0x7f6918df4718)
    at /mnt/root/interpreter/llvm/src/tools/clang/include/clang/AST/RecursiveASTVisitor.h:1496
#27 0x00007f69070497f8 in clang::RecursiveASTVisitor<loadGlobalModuleIndex(cling::Interpreter&)::DefinitionFinder>::TraverseDecl(clang::Decl *) (this=0x7f6918c66d10, D=0x7f6918df4718)
    at /mnt/root-build-runtime_cxxmodules-ON/interpreter/llvm/src/tools/clang/include/clang/AST/DeclNodes.inc:621
#28 0x00007f690702b4fc in DefinitionFinder::DefinitionFinder (this=0x7f6918c66d10, IDs=..., TU=0x7f6918df4718) at /mnt/root/core/metacling/src/TCling.cxx:1124
#29 0x00007f690702ba84 in loadGlobalModuleIndex (interp=...) at /mnt/root/core/metacling/src/TCling.cxx:1175
#30 0x00007f690702c1e8 in RegisterCxxModules (clingInterp=...) at /mnt/root/core/metacling/src/TCling.cxx:1273
#31 0x00007f690702dce0 in TCling::TCling (this=0x7f6918d2a730, name=0x7f690c9df1c0 "C++", title=0x7f690c9df1a8 "cling C++ Interpreter", argv=0x7f6918c67348, interpLibHandle=0x7f6918d00330)
    at /mnt/root/core/metacling/src/TCling.cxx:1592
#32 0x00007f6907028fe0 in CreateInterpreter (interpLibHandle=0x7f6918d00330, argv=0x7f6918c67348) at /mnt/root/core/metacling/src/TCling.cxx:609
#33 0x00007f691976892c in TROOT::InitInterpreter (this=0x7f6919c40c38 <ROOT::Internal::GetROOT1()::alloc>) at /mnt/root/core/base/src/TROOT.cxx:2072
#34 0x00007f691976271c in ROOT::Internal::GetROOT2 () at /mnt/root/core/base/src/TROOT.cxx:383
#35 0x00007f6919762940 in ROOT::GetROOT () at /mnt/root/core/base/src/TROOT.cxx:465
#36 0x00007f69197762c4 in TApplication::TApplication (this=0x7f6918cf8e70, appClassName=0x7f6918c6b508 "Rint", argc=0x7f6918c6956c, argv=0x7f6918c696f8, numOptions=0) at /mnt/root/core/base/src/TApplication.cxx:153
--Type <RET> for more, q to quit, c to continue without paging--c
#37 0x00007f6925d6e398 in TRint::TRint (this=0x7f6918cf8e70, appClassName=0x7f6918c6b508 "Rint", argc=0x7f6918c6956c, argv=0x7f6918c696f8, options=0x0, numOptions=0, noLogo=false, exitOnUnknownArgs=true)
    at /mnt/root/core/rint/src/TRint.cxx:174
#38 0x00007f6918c6b2c0 in main (argc=4, argv=0x7f6918c696f8) at /mnt/root/main/src/rmain.cxx:81
(gdb)

Maybe I should have let it run longer? After all it's jitting while jitting.

• But even just running through qemu-aarch64-static without gdb hook doesn't actually segfault, only when running directly:

# ROOTIGNOREPREFIX=1 ROOT_HIST=0 qemu-aarch64-static /mnt/root-build-runtime_cxxmodules-ON/bin/root.exe -l -q -b
^C
# ROOTIGNOREPREFIX=1 ROOT_HIST=0 /mnt/root-build-runtime_cxxmodules-ON/bin/root.exe -l -q -b
resolveAArch64Relocation, LocalAddress: 0x55849bf010 FinalAddress: 0x55849bf010 Value: 0x7fdbf6065000 Type: 0x113 Addend: 0x0
root.exe: /mnt/root/interpreter/llvm/src/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp:488: void llvm::RuntimeDyldELF::resolveAArch64Relocation(const llvm::SectionEntry&, uint64_t, uint64_t, uint32_t, int64_t): Assertion `isInt<33>(Result) && "overflow check failed for relocation"' failed.
Aborted (core dumped)

[ferdymercury]

@wdconinc Do you still see the same crash with master?

qemu-s390x VM works, see #10409 (comment)

[wdconinc]

We took arm64 out of our regular builds. I'll see if I can revive it over the weekend and get back to you (it would initially be a test with root@6.32.14, but once I have the general build infrastructure for arm64 revived I can fairly easily change to master 🤞 ).

[wdconinc]

Whatever the issue was, it appears to have been resolved in root@6.32.14, whether inside ROOT itself, or through changes in the compiler we use (gcc-12 to gcc-14). Current builds of ROOT for linux/arm64/v8 under qemu appear to compile fine.

[wdconinc]

From my point of view, this issue report can therefore be closed as resolved.

[wdconinc]

Actually, let me take some of that back for now. I may have spoken too soon.