Error on Linux/NixOS (ENOENT on shared libaries)

[lukego]

I tried running the KernelAbstractions.jl Quickstart instructions on my laptop (NixOS Thinkpad T490s with Julia 1.9.3 installed from the NixOS binary.) I'm seeing errors that look superficially like incomplete installation of oneAPI.jl dependencies.

More complete output is on a gist but here's the gist:

oneAPI looks happy on the surface:

julia> oneAPI.versioninfo()
Binary dependencies:
- NEO: 23.17.26241+3
- libigc: 1.0.13822+0
- gmmlib: 22.3.0+0
- SPIRV_LLVM_Translator_unified: 0.3.0+0
- SPIRV_Tools: 2023.2.0+0

Toolchain:
- Julia: 1.9.3
- LLVM: 14.0.6

1 driver:
- 00000000-0000-0000-179f-727101036681 (v1.3.26241, API v1.3.0)

1 device:
- Intel(R) UHD Graphics 620

but running the KernelAbstractions.jl Quickstart example fails:

julia> backend = get_backend(A)
oneAPI.oneAPIKernels.oneAPIBackend()

julia> mul2_kernel(backend, 64)(A, ndrange=size(A))
ERROR: IOError: could not spawn setenv(`/home/luke/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin/llvm-spirv ......]): no such file or directory (ENOENT)

where ENOENT seems to be on executing llvm-spirv:

execve("/home/luke/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin/llvm-spirv", ["/home/luke/.julia/artifacts/612d"..., "--spirv-debug-info-version=ocl-1"..., "--spirv-ext=+SPV_EXT_relaxed_pri"..., "-o", "/tmp/jl_kJidCnm9yz.spv", "/tmp/jl_61c3IJkrmZ.bc"], 0x7ffcb4ae2bc8 /* 75 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory

and it seems to be expecting to find share libraries that aren't there:

	linux-vdso.so.1 (0x00007ffd6b984000)
	libpthread.so.0 => /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib/libpthread.so.0 (0x00007ff59b3df000)
	libLLVMSPIRVLib.so.14jl => /home/luke/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin/../lib/libLLVMSPIRVLib.so.14jl (0x00007ff59ae00000)
	libLLVM-14jl.so => not found
	libstdc++.so.6 => not found
	libm.so.6 => /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib/libm.so.6 (0x00007ff59ad20000)
	libgcc_s.so.1 => /nix/store/rfckdjskd983ylf05jm9mlsw7y618hyr-xgcc-12.3.0-libgcc/lib/libgcc_s.so.1 (0x00007ff59b3bc000)
	libc.so.6 => /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib/libc.so.6 (0x00007ff59ab3a000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/ld03l52xq2ssn4x0g5asypsxqls40497-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007ff59b3e6000)
	libLLVM-14jl.so => not found
	libstdc++.so.6 => not found

I'm wondering if this is a bug in the Julia libraries, or a bug in the NixOS packaging of Julia, or user error?

[maleadt]

JLL-provided libraries and binaries are not indented to be used in isolation:

julia> oneAPI.SPIRV_LLVM_Translator_unified_jll.llvm_spirv()
setenv(`/home/tim/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin/llvm-spirv`,["PATH=/home/tim/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin:/home/tim/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games", "EDITOR=nano", "GIT_COMMITTER_EMAIL=tim.besard@gmail.com", "GIT_AUTHOR_NAME=Tim Besard", "LC_ALL=en_US.UTF-8", "LD_LIBRARY_PATH=/home/tim/Julia/depot/juliaup/julia-1.9.4+0.x64.linux.gnu/bin/../lib/julia:/home/tim/Julia/depot/juliaup/julia-1.9.4+0.x64.linux.gnu/bin/../lib", "LC_CTYPE=en_US.UTF-8", "DIRENV_WATCHES=eJxszs1KxDAQAOB3mXPZmc3PJundiyB4Fw-TZJYG0gbStAriu3uXvsDH9_ED7zwWmAGXtgqOsuLrUQvjTbazJ5jgreVRVoH5_ggPGxyRnuDlu-xjh3n0Q36nC-RWW-KK-8JdMJcu24lca_tCb6JXip1yTx-8cEiBUybndPaWOHIiJhODJS2avEkuPUnHkLW9a6_ksmT-lT7_AgAA__8eCkLH", "DIRENV_DIR=-/home/tim/Julia", "STARSHIP_SESSION_KEY=1380513656307742"  …  "XDG_SESSION_ID=26694",  "LC_TERMINAL=iTerm2", "OCL_ICD_VENDORS=/home/tim/.julia/artifacts/44fab11d374e3be02bcbd8229c1f5f489470c6bf/lib/intel-opencl/libigdrcl.so", "SSH_CLIENT=100.82.95.56 52916 22", "STARSHIP_SHELL=zsh", "SSH_TTY=/dev/pts/0", "HOME=/home/tim", "TERM=xterm-256color", "OPENBLAS_MAIN_FREE=1"])

shell> /home/tim/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin/llvm-spirv --version
/home/tim/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin/llvm-spirv: error while loading shared libraries: libLLVM-14jl.so: cannot open shared object file: No such file or directory

julia> run(`$(oneAPI.SPIRV_LLVM_Translator_unified_jll.llvm_spirv()) --version`)
LLVM (http://llvm.org/):
  LLVM version 14.0.6jl
  Optimized build.
  Default target: x86_64-linux-gnu
  Host CPU: tigerlake
Process(setenv(`/home/tim/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin/llvm-spirv --version`,["PATH=/home/tim/.julia/artifacts/612d93515fc7555212b03aabd642b1014f315794/bin:/home/tim/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games", "EDITOR=nano", "GIT_COMMITTER_EMAIL=tim.besard@gmail.com", "GIT_AUTHOR_NAME=Tim Besard", "LC_ALL=en_US.UTF-8", "LD_LIBRARY_PATH=/home/tim/Julia/depot/juliaup/julia-1.9.4+0.x64.linux.gnu/bin/../lib/julia:/home/tim/Julia/depot/juliaup/julia-1.9.4+0.x64.linux.gnu/bin/../lib", "LC_CTYPE=en_US.UTF-8", "DIRENV_WATCHES=eJxszs1KxDAQAOB3mXPZmc3PJundiyB4Fw-TZJYG0gbStAriu3uXvsDH9_ED7zwWmAGXtgqOsuLrUQvjTbazJ5jgreVRVoH5_ggPGxyRnuDlu-xjh3n0Q36nC-RWW-KK-8JdMJcu24lca_tCb6JXip1yTx-8cEiBUybndPaWOHIiJhODJS2avEkuPUnHkLW9a6_ksmT-lT7_AgAA__8eCkLH", "DIRENV_DIR=-/home/tim/Julia", "STARSHIP_SESSION_KEY=1380513656307742"  …  "XDG_SESSION_ID=26694", "LC_TERMINAL=iTerm2", "OCL_ICD_VENDORS=/home/tim/.julia/artifacts/44fab11d374e3be02bcbd8229c1f5f489470c6bf/lib/intel-opencl/libigdrcl.so", "SSH_CLIENT=100.82.95.56 52916 22", "STARSHIP_SHELL=zsh", "SSH_TTY=/dev/pts/0", "HOME=/home/tim", "TERM=xterm-256color", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(0))

This is likely Nix messing with the runtime environment set-up by JLLWrappers, including LD_LIBRARY_PATH.

[lukego]

I close this because it sounds like an issue with the Nix packaging.

(I don't want to fight with Nix-vs-Julia at the moment so I'll switch to CUDA which works out of the box.)

[maleadt]

Happy to improve the situation for Nix users, but this may require some changes to the underlying infrastructure (JLLWrappers) instead. You could try running with LD_DEBUG=libs to see what configuration is being used by the call to llvm-spirv.