ocloc crashes with LLVM 13

[zboszor]

I have a (not really usable) backtrace from running ocloc via a qemu wrapper when building under Yocto 3.4 with LLVM 13 from meta-clang.

$ cd /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/git/shared/source/built_ins/kernels
$ PATH=/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot-native/bin:/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot-native/sbin:/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot-native/usr/bin:/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot-native/usr/sbin /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/qemuwrapper /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/ocloc -q -file /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/built_ins/x64/gen12lp/bindful_copy_buffer_to_buffer_stateless_Gen12LPlp.spv -spirv_input -device tgllp -cl-intel-greater-than-4GB-buffer-required -64 -output bindful_copy_buffer_to_buffer_stateless_0 -out_dir /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/built_ins/x64/gen12lp -revision_id 0 -options -cl-kernel-arg-info
[0]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/libocloc.so(+0x7f3f0) [0x40018bb3f0]
[1]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libc.so.6(+0x41930) [0x400193e930]
[2]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libLLVM-13.so(_ZN4llvm11PointerType3getEPNS_4TypeEj+0x20) [0x40048009a0]
[3]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2b20e5) [0x40020b90e5]
[4]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2cf642) [0x40020d6642]
[5]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2d2b11) [0x40020d9b11]
[6]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2d8588) [0x40020df588]
[7]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2ddf9f) [0x40020e4f9f]
[8]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2dea98) [0x40020e5a98]
[9]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x219d78) [0x4002020d78]
[10]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x21cada) [0x4002023ada]
[11]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x21ddaa) [0x4002024daa]
[12]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2ec6ee) [0x40020f36ee]
[13]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libigc.so.1(+0x2ed59f) [0x40020f459f]
[14]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/libocloc.so(_ZN3NEO15OfflineCompiler15buildSourceCodeEv+0x23b) [0x400188e3db]
[15]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/libocloc.so(_ZN3NEO15OfflineCompiler5buildEv+0x45) [0x4001892e15]
[16]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/libocloc.so(+0x7f499) [0x40018bb499]
[17]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/libocloc.so(_Z20buildWithSafetyGuardPN3NEO15OfflineCompilerE+0xba) [0x40018bb34a]
[18]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/libocloc.so(oclocInvoke+0xc3b) [0x400187e5eb]
[19]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/ocloc(main+0x23) [0x4000000753]
[20]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libc.so.6(+0x2d51b) [0x400192a51b]
[21]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib/libc.so.6(__libc_start_main+0x7c) [0x400192a5cc]
[22]: /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin/ocloc(_start+0x25) [0x4000000785]
qemu: uncaught target signal 6 (Aborted) - core dumped
/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/qemuwrapper: line 2: 3885868 Aborted                 (core dumped) PSEUDO_UNLOAD=1 qemu-x86_64 -r 3.2.0 -cpu Nehalem,check=false -L /data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot -E LD_LIBRARY_PATH=/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/build/bin:/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib:/data/dtd-yocto-3.4/tmp-sicom-glibc/work/corei7-64-oe-linux/intel-compute-runtime/21.36.20889-r0/recipe-sysroot/usr/lib "$@"

Currently I can only build the latest IGC+compute-runtime via a forked meta-intel Yocto layer if I use -DNEO_DISABLE_BUILTINS_COMPILATION=1 with LLVM 13.

[JacekDanecki]

This looks like an issue on IGC side, which according to build_ubuntu.md currently supports llvm 11 at production quality and llvm 12 as experimental. As I can see in IGC commit history support for llvm 13 is under development.
Transferring issue to IGC project.

[pszymich]

Hi @zboszor,
I've put your issue on hold in the LLVM 13 support project. When we unclog VectorCompiler build I will look into this.

Please note build_ubuntu.md doc is slightly out-of-date with refactor coming; for current up-to-date LLVM version support information please visit our projects page: https://github.com/intel/intel-graphics-compiler/projects.

[eero-t]

What are the versions of the dependencies other than LLVM?

Why the build is done under (x86_64) Qemu, and what additional limitations (e.g. RAM) it sets on the build environment?

Can you reproduce ocloc crash without Qemu (e.g. in a docker container, if you want build to be isolated)?

[foutrelis]

Traced it to llvm/llvm-project@b00cff5 and verified that reverting that commit on top of LLVM 13.0.0 avoids the ocloc crash.

It seems that IGC should not pass nullptr to "GetElementPtrInst::Create() (and IRBuilder methods based on it)" in order to work with LLVM 13.

Edit: The above allowed me to remove NEO_DISABLE_BUILTINS_COMPILATION=ON but when dropping SKIP_UNIT_TESTS=1 as well, I get a different ocloc crash:

Edit 2: Digging further into the second crash, it appears to crash here because T is null. The TargetRegistry::lookupTarget call sets Err to No available targets are compatible with triple "vISA_64".

[root@build test_modules]# LD_LIBRARY_PATH=/build/intel-compute-runtime/src/build/bin /build/intel-compute-runtime/src/build/bin/ocloc -q -file test_kernel.cl -device glk -out_dir /build/intel-compute-runtime/src/build/bin/level_zero/Gen9lp/test_files/x64/ -options -g
[0]: /build/intel-compute-runtime/src/build/bin/libocloc.so(_ZN16SafetyGuardLinux9sigActionEiP9siginfo_tPv+0x39) [0x7f85fe336259]
[1]: /usr/lib/libc.so.6(+0x3cda0) [0x7f85fe0eada0]
[2]: /usr/lib/libigc.so.1(+0x850d46) [0x7f85efc13d46]
[3]: /usr/lib/libigc.so.1(+0x8488f5) [0x7f85efc0b8f5]
[4]: /usr/lib/libigc.so.1(+0x571ba3) [0x7f85ef934ba3]
[5]: /usr/lib/libLLVM-13.so(_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE+0x430) [0x7f85f804c8d0]
[6]: /usr/lib/libLLVM-13.so(_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE+0x2c) [0x7f85f804ca3c]
[7]: /usr/lib/libLLVM-13.so(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x3fa) [0x7f85f804e25a]
[8]: /usr/lib/libigc.so.1(+0x3a5e6f) [0x7f85ef768e6f]
[9]: /usr/lib/libigc.so.1(+0x3781da) [0x7f85ef73b1da]
[10]: /usr/lib/libigc.so.1(+0x22ad37) [0x7f85ef5edd37]
[11]: /usr/lib/libigc.so.1(+0x2fe887) [0x7f85ef6c1887]
[12]: /usr/lib/libigc.so.1(+0x2ff75f) [0x7f85ef6c275f]
[13]: /build/intel-compute-runtime/src/build/bin/libocloc.so(_ZN3NEO15OfflineCompiler15buildSourceCodeEv+0x678) [0x7f85fe2f7b78]
[14]: /build/intel-compute-runtime/src/build/bin/libocloc.so(_ZN3NEO15OfflineCompiler5buildEv+0x5c) [0x7f85fe2fcd7c]
[15]: /build/intel-compute-runtime/src/build/bin/libocloc.so(_ZN16SafetyGuardLinux4callIiN3NEO15OfflineCompilerEMS2_FivEEET_PT0_T1_S5_+0x50) [0x7f85fe336320]
[16]: /build/intel-compute-runtime/src/build/bin/libocloc.so(_Z20buildWithSafetyGuardPN3NEO15OfflineCompilerE+0xc9) [0x7f85fe3361a9]
[17]: /build/intel-compute-runtime/src/build/bin/libocloc.so(oclocInvoke+0x642) [0x7f85fe2dfc02]
[18]: /build/intel-compute-runtime/src/build/bin/ocloc(main+0x2a) [0x557a3fb0475a]
[19]: /usr/lib/libc.so.6(__libc_start_main+0xd5) [0x7f85fe0d5b25]
[20]: /build/intel-compute-runtime/src/build/bin/ocloc(_start+0x2e) [0x557a3fb0479e]
Aborted (core dumped)

The three top-most symbols from libigc.so.1 appear to be:

(gdb) info symbol 0x850d46
IGC::StreamEmitter::StreamEmitter(llvm::raw_pwrite_stream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, IGC::DebugEmitterOpts const&) + 646 in section .text
(gdb) info symbol 0x8488f5
IGC::DebugEmitter::Initialize(std::unique_ptr<IGC::VISAModule, std::default_delete<IGC::VISAModule> >, IGC::DebugEmitterOpts const&) + 229 in section .text
(gdb) info symbol 0x571ba3
IGC::EmitPass::runOnFunction(llvm::Function&) + 7891 in section .text

[zboszor]

What are the versions of the dependencies other than LLVM?

The versions are in the honister and current master branches of Yocto's openembedded-core, meta-openembedded, meta-clang and meta-intel.

Why the build is done under (x86_64) Qemu, and what additional limitations (e.g. RAM) it sets on the build environment?

Because in a cross-compiling environment (e.g. Yocto) you have two ways to deal with executables built and run during the build: either use a host-native build variant and use the executables from that (which does not work if the executable relies on type sizes and the host and target are different), or use a Qemu wrapper (e.g. with -DCMAKE_CROSSCOMPILING_EMULATOR=... and just do the complete build for the target architecture.

As far as I know, the qemu-native build Yocto uses for this purpose doesn't limit RAM, it simply allocates from the host system as needed. The target binaries are run just like any regular user-space binary only via the CPU emulator.

Can you reproduce ocloc crash without Qemu (e.g. in a docker container, if you want build to be isolated)?

According to the above Arch Linux commits, the answer is yes.

[zboszor]

Traced it to llvm/llvm-project@b00cff5 and verified that reverting that commit on top of LLVM 13.0.0 avoids the ocloc crash.

It seems that IGC should not pass nullptr to "GetElementPtrInst::Create() (and IRBuilder methods based on it)" in order to work with LLVM 13.

And here's the reasoning from LLVM devs: https://reviews.llvm.org/D105653?id=357321
There are 11 locations currently in IGC that use GetElementPtrInst::Create(nullptr, ...)

Edit: The above allowed me to remove NEO_DISABLE_BUILTINS_COMPILATION=ON but when dropping SKIP_UNIT_TESTS=1 as well, I get a different ocloc crash:

Edit 2: Digging further into the second crash, it appears to crash here because T is null. The TargetRegistry::lookupTarget call sets Err to No available targets are compatible with triple "vISA_64".

[zboszor]

Edit 2: Digging further into the second crash, it appears to crash here because T is null. The TargetRegistry::lookupTarget call sets Err to No available targets are compatible with triple "vISA_64".

IGC calls TargetRegistry::RegisterTarget() twice in IGC/VectorCompiler/lib/GenXCodeGen/TargetInfo/GenXTargetInfo.cpp with target/backend names "genx32" and "genx64", respectively, but doesn't provide the function for the 5th argument Target::ArchMatchFnTy ArchMatchFn so VISAModule::m_triple == "vISA_64" can't be matched as a compatible machine triple.

Alternatively, the "machine triple" setting:

./IGC/DebugInfo/VISAModule.hpp:356:        std::string m_triple = "vISA_64";

should be std::string m_triple = "genx64"; instead.