res/local.jl Inconsistency in commandline options with multiple vendor libraries installed

[michel2323]

Thanks for this script.

When running it on a system with both Intel CPU and GPUs, I get the following output:

: CommandLine Error: Option 'enable-nonnull-arg-prop' registered more than once!                                                                                                                                                                                                        LLVM ERROR: inconsistency in registered CommandLine options                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     signal (6): Aborted

This seems related to https://bugs.llvm.org/show_bug.cgi?id=30587

On such a system you can trigger by loading any runtime library:

julia -e 'using Libdl; Libdl.find_library("libigc")'   

Output:

: CommandLine Error: Option 'enable-nonnull-arg-prop' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

signal (6): Aborted
in expression starting at none:1
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
_ZN4llvm18report_fatal_errorERKNS_5TwineEb at /home/mschanen/julia/julia-1.6.5/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm18report_fatal_errorEPKcb at /home/mschanen/julia/julia-1.6.5/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN12_GLOBAL__N_117CommandLineParser9addOptionEPN4llvm2cl6OptionEPNS2_10SubCommandE at /home/mschanen/julia/julia-1.6.5/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm2cl6Option11addArgumentEv at /home/mschanen/julia/julia-1.6.5/bin/../lib/julia/libLLVM-11jl.so (unknown line)
unknown function (ip: 0x7feccab0e567)
call_init.part.0 at /lib64/ld-linux-x86-64.so.2 (unknown line)
_dl_init at /lib64/ld-linux-x86-64.so.2 (unknown line)
_dl_catch_exception at /lib64/libc.so.6 (unknown line)
dl_open_worker at /lib64/ld-linux-x86-64.so.2 (unknown line)
_dl_catch_exception at /lib64/libc.so.6 (unknown line)
_dl_open at /lib64/ld-linux-x86-64.so.2 (unknown line)
dlopen_doit at /lib64/libdl.so.2 (unknown line)
_dl_catch_exception at /lib64/libc.so.6 (unknown line)
_dl_catch_error at /lib64/libc.so.6 (unknown line)
_dlerror_run at /lib64/libdl.so.2 (unknown line)
dlopen at /lib64/libdl.so.2 (unknown line)
jl_load_dynamic_library at /buildworker/worker/package_linux64/build/src/dlload.c:264
#dlopen#3 at ./libdl.jl:114
dlopen##kw at ./libdl.jl:114 [inlined]
find_library at ./libdl.jl:203
find_library at ./libdl.jl:211 [inlined]
find_library at ./libdl.jl:211
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
exec_options at ./client.jl:261
_start at ./client.jl:485
jfptr__start_38548.clone_1 at /home/mschanen/julia/julia-1.6.5/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:560
repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42
Allocations: 2649 (Pool: 2639; Big: 10); GC: 0
[1]    50921 abort      julia -e 'using Libdl; Libdl.find_library("libigc")'

[maleadt]

This happens when loading multiple copies of LLVM. Julia's copy should be protected by this because of symbol versioning (@vchuravy can you confirm?), so is it possible you're either using a non-official Julia build (with SYSTEM_LLVM) or LD_PRELOADing a library that dynamically links against LLVM, or something similar?

[vchuravy]

Yeah we should no longer get these from Julia normally, but of course most other LLVM are not hardened and I don't know how weak symbols will play with it

[vchuravy]

But I don't recall if we got all this in place for 1.6 and LLVM 11

[michel2323]

This error appeared with Julia 1.6 and 1.8. Peng then pointed me to not load the oneAPI module on the target system, but only load the Intel Compute Runtime. We have to use the system libraries. That resolved the issue.

[maleadt]

So I'm guessing libze's plugin loader (from the system) then loaded some other back-end that dynamically links against LLVM?

[maleadt]

I don't think there's anything actionable here, as it requires changes to how the libraries we load are compiled.