Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intel Data Center GPU Max series is not supported? #365

Closed
thoth291 opened this issue Sep 24, 2023 · 11 comments
Closed

Intel Data Center GPU Max series is not supported? #365

thoth291 opened this issue Sep 24, 2023 · 11 comments

Comments

@thoth291
Copy link

Here is what I'm trying to do:

#add oneAPI
using oneAPI
Here is what I'm getting (it also kills Julia): 
(julia-1.9.3) [intel-gpu]$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.3 (2023-08-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using oneAPI
Abort was called at 181 line in file:
/workspace/srcdir/compute-runtime/shared/source/os_interface/linux/engine_info.cpp

[1334102] signal (6.-6): Aborted
in expression starting at REPL[1]:1
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
_ZN3NEO14abortExecutionEv at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO18abortUnrecoverableEiPKc at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO10EngineInfo16assignCopyEngineEN10aub_stream10EngineTypeEjRKNS_19EngineClassInstanceERSt6bitsetILm9EERjS9_ at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO10EngineInfoC1EPNS_3DrmERKSt6vectorINS_18EngineCapabilitiesESaIS4_EE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO3Drm15queryEngineInfoEb at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO3Drm6createEOSt10unique_ptrINS_13HwDeviceIdDrmESt14default_deleteIS2_EERNS_21RootDeviceEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO18initDrmOsInterfaceEOSt10unique_ptrINS_10HwDeviceIdESt14default_deleteIS1_EEjPNS_21RootDeviceEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO13DeviceFactory25prepareDeviceEnvironmentsERNS_20ExecutionEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO25prepareDeviceEnvironmentsERNS_20ExecutionEnvironmentERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEj at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO25prepareDeviceEnvironmentsERNS_20ExecutionEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO13DeviceFactory13createDevicesERNS_20ExecutionEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN2L09DriverImp10initializeEP12_ze_result_t at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZZSt9call_onceIZN2L09DriverImp10driverInitEjEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
__pthread_once_slow at /lib64/libpthread.so.0 (unknown line)
_ZN2L04initEj at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN6loader9context_t11init_driverENS_8driver_tEj at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
_ZN6loader9context_t13check_driversEj at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
_ZN6ze_lib9context_t4InitEjb at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
_ZZSt9call_onceIZ6zeInitEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
__pthread_once_slow at /lib64/libpthread.so.0 (unknown line)
zeInit at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
macro expansion at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:1288 [inlined]
#5 at /home/user/julia/packages/oneAPI/g8ZUm/lib/utils/call.jl:25 [inlined]
#1 at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:15 [inlined]
retry_reclaim at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/utils.jl:4
check at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:13 [inlined]
zeInit at /home/user/julia/packages/oneAPI/g8ZUm/lib/utils/call.jl:24 [inlined]
__init__ at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/oneL0.jl:100
jfptr___init___955 at /home/user/julia/compiled/v1.9/oneAPI/X4ZbC_sUjTk.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_module_run_initializer at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:75
ijl_init_restored_modules at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/module.c:982
register_restored_modules at ./loading.jl:1115
_include_from_serialized at ./loading.jl:1061
_require_search_from_serialized at ./loading.jl:1506
_require at ./loading.jl:1783
_require_prelocked at ./loading.jl:1660
macro expansion at ./loading.jl:1648 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1611
jfptr_require_45889.clone_1 at /home/user/_envs/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
call_require at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:466 [inlined]
eval_import_path at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:503
jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:731
jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
eval_user_input at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:153
repl_backend_loop at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:249
#start_repl_backend#46 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:234
start_repl_backend at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:231
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#run_repl#59 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:379
run_repl at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:365
jfptr_run_repl_60908.clone_1 at /home/user/_envs/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#1017 at ./client.jl:421
jfptr_YY.1017_36106.clone_1 at /home/user/_envs/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:819 [inlined]
invokelatest at ./essentials.jl:816 [inlined]
run_main_repl at ./client.jl:405
exec_options at ./client.jl:322
_start at ./client.jl:522
jfptr__start_40034.clone_1 at /home/user/_envs/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
true_main at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:573
jl_repl_entrypoint at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:717
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 14047988 (Pool: 14047174; Big: 814); GC: 28
Aborted (core dumped)

Here is the L0 version:

$ xpu-smi -v
CLI:
    Version: 1.2.13.20230704
    Build ID: 100e4c46

Service:
    Version: 1.2.13.20230704
    Build ID: 100e4c46
    Level Zero Version: 1.11.0

What it takes to make it work?

P.S. I have 2 PV on one SR machine.
@maleadt
Copy link
Member

maleadt commented Sep 24, 2023

This looks like a bug in the Intel driver. Try using the debug artifacts. Normally you do that by calling set_debug!, but since you can't import oneAPI.jl, here's what that does:

oneAPI.jl/src/oneAPI.jl

Lines 85 to 90 in 6623a74

function set_debug!(debug::Bool)
for jll in [oneL0.NEO_jll, oneL0.NEO_jll.libigc_jll]
Preferences.set_preferences!(jll, "debug" => string(debug); force=true)
end
@info "oneAPI debug mode $(debug ? "enabled" : "disabled"); please re-start Julia."
end

@thoth291
Copy link
Author

thoth291 commented Sep 25, 2023 •
edited
Loading

@maleadt ,

It fails on loading oneL0 with the same error - so I can't set those preferences...
Any other options?

Thank you for helping with this!

@maleadt
Copy link
Member

maleadt commented Sep 25, 2023

Use the following LocalPreferences.jl in the directory of an environment (i.e. next to a Project.toml and Manifest.toml) that has these JLLs as dependency:

[NEO_jll]
debug = "true"

[libigc_jll]
debug = "true"

The easiest is probably to check-out oneAPI.jl, put this in a LocalPreferences.jl in the check-out, and start Julia with --project.

@thoth291
Copy link
Author

thoth291 commented Sep 28, 2023 •
edited
Loading

Here is the result of running using oneAPI after adding LocalPreferences.toml.

New messages 
Abort was called at 181 line in file:
/workspace/srcdir/compute-runtime/shared/source/os_interface/linux/engine_info.cpp

[1504513] signal (6.-6): Aborted
in expression starting at REPL[1]:1
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
_ZN3NEO14abortExecutionEv at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO18abortUnrecoverableEiPKc at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO10EngineInfo16assignCopyEngineEN10aub_stream10EngineTypeEjRKNS_19EngineClassInstanceERSt6bitsetILm9EERjS9_ at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO10EngineInfoC1EPNS_3DrmERKSt6vectorINS_18EngineCapabilitiesESaIS4_EE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO3Drm15queryEngineInfoEb at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO3Drm6createEOSt10unique_ptrINS_13HwDeviceIdDrmESt14default_deleteIS2_EERNS_21RootDeviceEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO18initDrmOsInterfaceEOSt10unique_ptrINS_10HwDeviceIdESt14default_deleteIS1_EEjPNS_21RootDeviceEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO13DeviceFactory25prepareDeviceEnvironmentsERNS_20ExecutionEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO25prepareDeviceEnvironmentsERNS_20ExecutionEnvironmentERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEj at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO25prepareDeviceEnvironmentsERNS_20ExecutionEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN3NEO13DeviceFactory13createDevicesERNS_20ExecutionEnvironmentE at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN2L09DriverImp10initializeEP12_ze_result_t at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZZSt9call_onceIZN2L09DriverImp10driverInitEjEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
__pthread_once_slow at /lib64/libpthread.so.0 (unknown line)
_ZN2L04initEj at /home/user/julia/artifacts/15f368fd04234dfd3749affe9f9ea6318d7ab7b6/lib/libze_intel_gpu.so.1 (unknown line)
_ZN6loader9context_t11init_driverENS_8driver_tEj at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
_ZN6loader9context_t13check_driversEj at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
_ZN6ze_lib9context_t4InitEjb at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
_ZZSt9call_onceIZ6zeInitEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
__pthread_once_slow at /lib64/libpthread.so.0 (unknown line)
zeInit at /home/user/julia/artifacts/f97d53a3b7456f3d29c093c5c29431b1b3a482e8/lib/libze_loader.so (unknown line)
macro expansion at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:1288 [inlined]
#5 at /home/user/julia/packages/oneAPI/g8ZUm/lib/utils/call.jl:25 [inlined]
#1 at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:15 [inlined]
retry_reclaim at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/utils.jl:4
check at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:13 [inlined]
zeInit at /home/user/julia/packages/oneAPI/g8ZUm/lib/utils/call.jl:24 [inlined]
__init__ at /home/user/julia/packages/oneAPI/g8ZUm/lib/level-zero/oneL0.jl:100
jfptr___init___955 at /home/user/julia/compiled/v1.9/oneAPI/X4ZbC_sUjTk.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_module_run_initializer at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:75
ijl_init_restored_modules at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/module.c:982
register_restored_modules at ./loading.jl:1115
_include_from_serialized at ./loading.jl:1061
_require_search_from_serialized at ./loading.jl:1506
_require at ./loading.jl:1783
_require_prelocked at ./loading.jl:1660
macro expansion at ./loading.jl:1648 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1611
jfptr_require_45889.clone_1 at /home/user/julia/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
call_require at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:466 [inlined]
eval_import_path at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:503
jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:731
jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
eval_user_input at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:153
repl_backend_loop at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:249
#start_repl_backend#46 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:234
start_repl_backend at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:231
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#run_repl#59 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:379
run_repl at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:365
jfptr_run_repl_60908.clone_1 at /home/user/julia/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#1017 at ./client.jl:421
jfptr_YY.1017_36106.clone_1 at /home/user/julia/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:819 [inlined]
invokelatest at ./essentials.jl:816 [inlined]
run_main_repl at ./client.jl:405
exec_options at ./client.jl:322
_start at ./client.jl:522
jfptr__start_40034.clone_1 at /home/user/julia/julia-1.9.3/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
true_main at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:573
jl_repl_entrypoint at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:717
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 14082583 (Pool: 14081587; Big: 996); GC: 28
Aborted (core dumped)

cat Project.toml

[deps]
Preferences = "21216c6a-2e73-6563-6e65-726566657250"
oneAPI = "8f75cd03-7ff8-4ecb-9b8f-daf728133b1b"

cat LocalPreferences.toml

[NEO_jll]
debug = "true"

[libigc_jll]
debug = "true"

@pengtu
Copy link
Contributor

pengtu commented Sep 28, 2023

Here is a full list of steps to use Julia oneAPI.jl with the local driver. Please follow the steps and report back any errors.

download and extract Julia 1.9

wget https://julialangnightlies-s3.julialang.org/bin/linux/x64/1.9/julia-latest-linux64.tar.gz
tar -xvf julia-latest-linux64.tar.gz
export JULIA=$(echo $(pwd)/julia-*/bin/julia)

clone oneAPI.jl

git clone https://github.com/JuliaGPU/oneAPI.jl
cd oneAPI.jl

install dependencies (this will take a minute)

$JULIA --project -e 'using Pkg; Pkg.instantiate()'

#build with local changes
$JULIA --project -e 'using Pkg; Pkg.build()'

have oneAPI.jl use system dependencies of the oneAPI toolchain

$JULIA res/local.jl

that's it. you can now run julia with the --project flag in the oneAPI dir

$JULIA --project -e 'using oneAPI; oneAPI.versioninfo()'

or for something more useful: run the oneAPI.jl test suite

$JULIA --project --threads=auto -e 'using Pkg; Pkg.test()'

@maleadt
Copy link
Member

maleadt commented Sep 28, 2023 •
edited
Loading

have oneAPI.jl use system dependencies of the oneAPI toolchain

But that shouldn't be required; we should figure out what's up here so that we can update the drivers (or file an issue on the compute-runtime repository).

@thoth291 You don't seem to be using the debug artifacts, matching those hashes to https://github.com/JuliaBinaryWrappers/NEO_jll.jl/blob/d0f3719b17e942e59d3e6c738766f3e5b64c4677/Artifacts.toml#L25-L32. Did you start Julia with --project from the oneAPI.jl check-out where you put that LocalPreferences.jl?

@pengtu
Copy link
Contributor

pengtu commented Sep 28, 2023 •
edited
Loading

Many customers are getting prerelease drivers that work with their specific cards. It will take time for the public released driver to catch up. Hence, I suggest to try locally installed drivers first for PVC issues.

@thoth291
Copy link
Author

that's it. you can now run julia with the --project flag in the oneAPI dir

$JULIA --project -e 'using oneAPI; oneAPI.versioninfo()'

This is what I've got:

Binary dependencies:
- NEO: 23.17.26241+0
- libigc: 1.0.13822+0
- gmmlib: 22.3.0+0
- SPIRV_LLVM_Translator_unified: 0.3.0+0
- SPIRV_Tools: 2023.2.0+0

Toolchain:
- Julia: 1.9.3
- LLVM: 14.0.6

1 driver:
- 00000000-0000-0000-1789-421801036794 (v1.3.26516, API v1.3.0)

2 devices:
- Intel(R) Data Center GPU Max 1100
- Intel(R) Data Center GPU Max 1100

or for something more useful: run the oneAPI.jl test suite

$JULIA --project --threads=auto -e 'using Pkg; Pkg.test()'

This didn't work for some reason :-(.
I've got this:

ERROR: trying to test unnamed project
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
   @ Pkg.API /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/API.jl:433
 [3] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [4] test(pkgs::Vector{Pkg.Types.PackageSpec})
   @ Pkg.API /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [5] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/API.jl:171
 [6] test()
   @ Pkg.API /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/API.jl:162
 [7] top-level scope
   @ none:1

But this way it partly worked:

output of the test 
pkg> test oneAPI
     Testing oneAPI
      Status `/tmp/jl_WjrFEl/Project.toml`
  [79e6a3ab] Adapt v3.6.2
  [7a1cc6ca] FFTW v1.7.1
  [0c68f7d7] GPUArrays v8.8.1
  [63c18a36] KernelAbstractions v0.9.8
  [276daf66] SpecialFunctions v2.3.1
  [90137ffa] StaticArrays v1.6.2
  [8f75cd03] oneAPI v1.3.0
  [700fe977] NEO_jll v23.17.26241+0
  [94295238] libigc_jll v1.0.13822+0
  [b049733a] oneAPI_Support_jll v0.2.2+0
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics v1.9.0 `@stdlib/Statistics`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_WjrFEl/Manifest.toml`
  [621f4979] AbstractFFTs v1.5.0
  [79e6a3ab] Adapt v3.6.2
  [a9b6321e] Atomix v0.1.0
  [fa961155] CEnum v0.4.2
  [ffbed154] DocStringExtensions v0.9.3
  [e2ba6199] ExprTools v0.1.10
  [7a1cc6ca] FFTW v1.7.1
  [0c68f7d7] GPUArrays v8.8.1
  [46192b85] GPUArraysCore v0.1.5
� [61eb1bfa] GPUCompiler v0.21.4
  [92d709cd] IrrationalConstants v0.2.2
  [692b3bcd] JLLWrappers v1.5.0
  [63c18a36] KernelAbstractions v0.9.8
  [929cbde3] LLVM v6.1.0
  [2ab3a3ac] LogExpFunctions v0.3.26
  [1914dd2f] MacroTools v0.5.11
  [aea7be01] PrecompileTools v1.2.0
  [21216c6a] Preferences v1.4.0
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [6c6a2e73] Scratch v1.2.0
  [276daf66] SpecialFunctions v2.3.1
  [90137ffa] StaticArrays v1.6.2
  [1e83bf80] StaticArraysCore v1.4.2
  [a759f4b9] TimerOutputs v0.5.23
  [013be700] UnsafeAtomics v0.2.1
  [d80eeb9a] UnsafeAtomicsLLVM v0.1.3
  [8f75cd03] oneAPI v1.3.0
  [f5851436] FFTW_jll v3.3.10+0
  [1d5cc7b8] IntelOpenMP_jll v2023.2.0+0
  [dad2f222] LLVMExtra_jll v0.0.23+0
  [856f044c] MKL_jll v2023.2.0+0
  [700fe977] NEO_jll v23.17.26241+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [85f0d8ed] SPIRV_LLVM_Translator_unified_jll v0.3.0+0
  [6ac6d60f] SPIRV_Tools_jll v2023.2.0+0
  [09858cae] gmmlib_jll v22.3.0+0
  [94295238] libigc_jll v1.0.13822+0
  [f4bc562b] oneAPI_Level_Zero_Headers_jll v1.6.3+0
  [13eca655] oneAPI_Level_Zero_Loader_jll v1.11.0+0
  [b049733a] oneAPI_Support_jll v0.2.2+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.9.2 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics v1.9.0 `@stdlib/Statistics`
  [fa267f1f] TOML v1.0.3 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v1.0.5+0 `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll v7.84.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.2+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.10.11 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.21+4 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [bea87d4a] SuiteSparse_jll v5.10.1+6 `@stdlib/SuiteSparse_jll`
  [83775a58] Zlib_jll v1.2.13+0 `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll v5.8.0+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.48.0+0 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with � have new versions available but compatibility constraints restrict them from upgrading.
malloc(): corrupted top size
ERROR: failed process: Process(setenv(`/home/user/julia/julia-1.9.3/libexec/julia/7z x /home/user/dist/julia/julia-1.9.3/depot/registries/General.tar.gz -so`,...), ProcessSignaled(6)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] open(::Pkg.Registry.var"#19#22"{IOBuffer, Vector{UInt8}, Dict{String, String}}, ::Cmd; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./process.jl:437
  [3] open
    @ ./process.jl:414 [inlined]
  [4] uncompress_registry(tar_gz::String)
    @ Pkg.Registry /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/Registry/registry_instance.jl:256
  [5] Pkg.Registry.RegistryInstance(path::String)
    @ Pkg.Registry /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/Registry/registry_instance.jl:323
  [6] reachable_registries(; depots::Vector{String})
    @ Pkg.Registry /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/Registry/registry_instance.jl:429
  [7] reachable_registries
    @ /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/Registry/registry_instance.jl:399 [inlined]
  [8] Context
    @ ./util.jl:567 [inlined]
  [9] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:warn_loaded,), Tuple{Bool}}})
    @ Pkg.API /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/API.jl:147
 [10] precompile(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:warn_loaded,), Tuple{Bool}}})
    @ Pkg.API /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Pkg/src/API.jl:171
 [11] top-level scope
    @ none:6
     Testing Precompilation of test environment failed. Continuing to tests
     Testing Running tests...
� Info: System information:
� Binary dependencies:
� - NEO: 23.17.26241+0 (debug)
� - libigc: 1.0.13822+0 (debug)
� - gmmlib: 22.3.0+0
� - SPIRV_LLVM_Translator_unified: 0.3.0+0
� - SPIRV_Tools: 2023.2.0+0
� 
� Toolchain:
� - Julia: 1.9.3
� - LLVM: 14.0.6
� 
� 1 driver:
� - 00000000-0000-0000-1789-42bb01036794 (v1.3.26516, API v1.3.0)
� 
� 2 devices:
� - Intel(R) Data Center GPU Max 1100
� - Intel(R) Data Center GPU Max 1100
[ Info: Using oneAPI support library at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/liboneapi_support.so
[ Info: Running 224 tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the JULIA_CPU_THREADS environment variable.
                                                   |          | ---------------- CPU ---------------- |
Test                                      (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
sycl                                          (10) |     0.48 |   0.00 |  0.0 |      40.30 |   988.20 |
pointer                                        (8) |     0.73 |   0.01 |  1.4 |      53.34 |   988.20 |
level-zero                                     (6) |     3.60 |   0.09 |  2.4 |     309.21 |   988.20 |
      From worker 3:    terminate called after throwing an instance of 'sycl::_V1::exception'
      From worker 3:      what():  Level-Zero error:700000041879048196
      From worker 3:    On device: 'Intel(R) Data Center GPU Max 1100'
      From worker 3:    in kernel: oneapi::mkl::blas::sgemm_itcopy
      From worker 3:
      From worker 3:    [1531633] signal (6.-6): Aborted
      From worker 3:    in expression starting at /home/user/conda/julia/packages/oneAPI/g8ZUm/examples/gemm.jl:7
      From worker 3:    gsignal at /lib64/libc.so.6 (unknown line)
      From worker 3:    abort at /lib64/libc.so.6 (unknown line)
      From worker 3:    __verbose_terminate_handler at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/vterminate.cc:95
      From worker 3:    __terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      From worker 3:    terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      From worker 3:    __cxa_throw at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_throw.cc:98
      From worker 3:    _ZN6oneapi3mkl3gpu13build_programEPiPN4sycl3_V15queueEPvS7_iPKcS9_mcS9_Pb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl3gpuL22mkl_gpu_get_kernel_extEPiPN4sycl3_V15queueEiPKcS8_mcS8_S8_S8_mPKvmbb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl3gpu24mkl_gpu_get_spirv_kernelEPiPN4sycl3_V15queueEiPK22mkl_gpu_spirv_kernel_tPKcSB_ at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl3gpu40mkl_blas_gpu_sgemm_copybased_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl3gpu30mkl_blas_gpu_sgemm_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl3gpu19sgemm_sycl_internalEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl3gpu10sgemm_syclEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl4blas5sgemmERN4sycl3_V15queueE10MKL_LAYOUTNS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS3_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    _ZN6oneapi3mkl4blas12column_major4gemmERN4sycl3_V15queueENS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS4_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 3:    gemm at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/include/oneapi/mkl/blas/usm_decls.hpp:38 [inlined]
      From worker 3:    onemklSgemm at /workspace/srcdir/oneAPI.jl/deps/src/onemkl.cpp:181
      From worker 3:    onemklSgemm at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/support/liboneapi_support.jl:98
      From worker 3:    unknown function (ip: 0x14798ee5a8f9)
      From worker 3:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 3:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 3:    gemm! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/wrappers.jl:1185
      From worker 3:    generic_matmatmul! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:215
      From worker 3:    gemm_wrapper! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:238 [inlined]
      From worker 3:    mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:161 [inlined]
      From worker 3:    mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:276 [inlined]
      From worker 3:    * at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:148
      From worker 3:    unknown function (ip: 0x14798ee58926)
      From worker 3:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 3:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 3:    jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 3:    do_call at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:126
      From worker 3:    eval_value at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:226
      From worker 3:    eval_stmt_value at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
      From worker 3:    eval_body at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:624
      From worker 3:    jl_interpret_toplevel_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:762
      From worker 3:    jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:912
      From worker 3:    jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:856
      From worker 3:    ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 3:    eval at ./boot.jl:370 [inlined]
      From worker 3:    include_string at ./loading.jl:1903
      From worker 3:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 3:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 3:    _include at ./loading.jl:1963
      From worker 3:    include at ./Base.jl:457
      From worker 3:    jfptr_include_35036.clone_1 at /home/user/julia/julia-1.9.3/lib/julia/sys.so (unknown line)
      From worker 3:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 3:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 3:    exec_options at ./client.jl:307
      From worker 3:    _start at ./client.jl:522
      From worker 3:    jfptr__start_40034.clone_1 at /home/user/julia/julia-1.9.3/lib/julia/sys.so (unknown line)
      From worker 3:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 3:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 3:    jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 3:    true_main at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:573
      From worker 3:    jl_repl_entrypoint at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jlapi.c:717
      From worker 3:    main at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/cli/loader_exe.c:59
      From worker 3:    __libc_start_main at /lib64/libc.so.6 (unknown line)
      From worker 3:    unknown function (ip: 0x4010b8)
      From worker 3:    Allocations: 31508343 (Pool: 31497157; Big: 11186); GC: 55
      From worker 4:    � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 4:    �   caller = ip:0x0
      From worker 4:    � @ Core :-1
      From worker 11:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 11:   �   caller = ip:0x0
      From worker 11:   � @ Core :-1
      From worker 36:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 36:   �   caller = ip:0x0
      From worker 36:   � @ Core :-1
      From worker 25:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 25:   �   caller = ip:0x0
      From worker 25:   � @ Core :-1
      From worker 23:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 23:   �   caller = ip:0x0
      From worker 23:   � @ Core :-1
      From worker 19:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 19:   �   caller = ip:0x0
      From worker 19:   � @ Core :-1
      From worker 21:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 21:   �   caller = ip:0x0
      From worker 21:   � @ Core :-1
      From worker 13:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 13:   �   caller = ip:0x0
      From worker 13:   � @ Core :-1
      From worker 22:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 22:   �   caller = ip:0x0
      From worker 22:   � @ Core :-1
      From worker 9:    � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 9:    �   caller = ip:0x0
      From worker 9:    � @ Core :-1
      From worker 28:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 28:   �   caller = ip:0x0
      From worker 28:   � @ Core :-1
      From worker 34:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 34:   �   caller = ip:0x0
      From worker 34:   � @ Core :-1
      From worker 16:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 16:   �   caller = ip:0x0
      From worker 16:   � @ Core :-1
      From worker 30:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 30:   �   caller = ip:0x0
      From worker 30:   � @ Core :-1
      From worker 35:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 35:   �   caller = ip:0x0
      From worker 35:   � @ Core :-1
      From worker 20:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 20:   �   caller = ip:0x0
      From worker 20:   � @ Core :-1
      From worker 32:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 32:   �   caller = ip:0x0
      From worker 32:   � @ Core :-1
      From worker 12:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 12:   �   caller = ip:0x0
      From worker 12:   � @ Core :-1
      From worker 2:    � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 2:    �   caller = ip:0x0
      From worker 2:    � @ Core :-1
      From worker 15:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 15:   �   caller = ip:0x0
      From worker 15:   � @ Core :-1
      From worker 26:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 26:   �   caller = ip:0x0
      From worker 26:   � @ Core :-1
      From worker 18:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 18:   �   caller = ip:0x0
      From worker 18:   � @ Core :-1
      From worker 24:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 24:   �   caller = ip:0x0
      From worker 24:   � @ Core :-1
gpuarrays/interface                           (19) |    13.41 |   0.42 |  3.2 |    1029.75 |  1063.88 |
      From worker 27:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 27:   �   caller = ip:0x0
      From worker 27:   � @ Core :-1
array                                          (2) |    13.60 |   0.45 |  3.3 |    1089.44 |  1067.74 |
      From worker 33:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 33:   �   caller = ip:0x0
      From worker 33:   � @ Core :-1
      From worker 29:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 29:   �   caller = ip:0x0
      From worker 29:   � @ Core :-1
      From worker 4:    � Warning: spirv_dis() is deprecated, use the non-do-block form
      From worker 4:    �   caller = ip:0x0
      From worker 4:    � @ Core :-1
gpuarrays/math/intrinsics                     (25) |    16.73 |   0.56 |  3.3 |    1360.07 |  1075.95 |
random                                         (9) |    17.56 |   0.57 |  3.3 |    1391.70 |  1095.91 |
      From worker 17:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 17:   �   caller = ip:0x0
      From worker 17:   � @ Core :-1
      From worker 11:
      From worker 11:   [1531474] signal (11.1): Segmentation fault
      From worker 11:   in expression starting at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/device/intrinsics.jl:47
      From worker 11:   unknown function (ip: 0x148e71bcf000)
      From worker 11:   unknown function (ip: 0x148e71bcf36a)
      From worker 11:   unknown function (ip: 0x148e71894340)
      From worker 11:   unknown function (ip: 0x148e7187e91c)
      From worker 11:   unknown function (ip: 0x148e717ff137)
      From worker 11:   unknown function (ip: 0x148e717ff240)
      From worker 11:   unknown function (ip: 0x148e717ff31b)
      From worker 11:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:1556 [inlined]
      From worker 11:   #63 at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/utils/call.jl:25 [inlined]
      From worker 11:   #1 at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:15 [inlined]
      From worker 11:   retry_reclaim at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/level-zero/utils.jl:4
      From worker 11:   check at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/level-zero/libze.jl:13 [inlined]
      From worker 11:   zeCommandQueueSynchronize at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/utils/call.jl:24 [inlined]
      From worker 11:   synchronize at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/level-zero/cmdqueue.jl:34 [inlined]
      From worker 11:   synchronize at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/level-zero/cmdqueue.jl:34 [inlined]
      From worker 11:   synchronize at /home/user/conda/julia/packages/oneAPI/g8ZUm/src/context.jl:59
      From worker 11:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/src/utils.jl:78 [inlined]
      From worker 11:   #25 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:127 [inlined]
      From worker 11:   RedirectStdStream at ./stream.jl:1429
      From worker 11:   #24 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:87 [inlined]
      From worker 11:   #open#409 at ./io.jl:395
      From worker 11:   open at ./io.jl:392 [inlined]
      From worker 11:   #23 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:86 [inlined]
      From worker 11:   mktemp at ./file.jl:732
      From worker 11:   unknown function (ip: 0x148e3932f983)
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   mktemp at ./file.jl:730
      From worker 11:   unknown function (ip: 0x148e3932db22)
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 11:   do_call at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:126
      From worker 11:   eval_value at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:226
      From worker 11:   eval_stmt_value at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
      From worker 11:   eval_body at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:624
      From worker 11:   eval_body at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:533
      From worker 11:   eval_body at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:533
      From worker 11:   jl_interpret_toplevel_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/interpreter.c:762
      From worker 11:   jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:912
      From worker 11:   jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:856
      From worker 11:   ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 11:   eval at ./boot.jl:370 [inlined]
      From worker 11:   include_string at ./loading.jl:1903
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   _include at ./loading.jl:1963
      From worker 11:   include at ./client.jl:478 [inlined]
      From worker 11:   #13 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:86 [inlined]
      From worker 11:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 11:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 11:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 11:   macro expansion at ./timing.jl:501 [inlined]
      From worker 11:   top-level scope at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:51
      From worker 11:   jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:903
      From worker 11:   ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 11:   eval at ./boot.jl:370 [inlined]
      From worker 11:   runtests at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:55
      From worker 11:   unknown function (ip: 0x148e9a713fa6)
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 11:   jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 11:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 11:   #invokelatest#2 at ./essentials.jl:819
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 11:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 11:   invokelatest at ./essentials.jl:816
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 11:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 11:   #110 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
      From worker 11:   run_work_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
      From worker 11:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
      From worker 11:   #109 at ./task.jl:514
      From worker 11:   unknown function (ip: 0x148e9a7127cf)
      From worker 11:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 11:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 11:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 11:   start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
      From worker 11:   Allocations: 40555389 (Pool: 40531290; Big: 24099); GC: 63
device/intrinsics                             (11) |         failed at 2023-09-28T23:02:25.465
Worker 11 terminated.
      From worker 31:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 31:   �   caller = ip:0x0
      From worker 31:   � @ Core :-1
gpuarrays/uniformscaling                      (22) |    19.06 |   0.59 |  3.1 |    1408.96 |  1110.77 |
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:947
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:955
 [3] unsafe_read
   @ ./io.jl:761 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:760
 [5] read!
   @ ./io.jl:762 [inlined]
 [6] deserialize_hdr_raw
   @ /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:514
      From worker 7:    � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 7:    �   caller = ip:0x0
      From worker 7:    � @ Core :-1
      From worker 5:    � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 5:    �   caller = ip:0x0
      From worker 5:    � @ Core :-1
gpuarrays/indexing scalar                     (12) |    22.88 |   0.85 |  3.7 |    1725.55 |  1147.11 |
gpuarrays/reductions/any all count            (20) |    24.19 |   0.85 |  3.5 |    2219.33 |  1168.38 |
      From worker 3:    � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 3:    �   caller = ip:0x0
      From worker 3:    � @ Core :-1
examples                                       (3) |         failed at 2023-09-28T23:02:33.041
      From worker 4:    WARNING: Method definition #3054#kernel(Any) in module Main at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/execution.jl:298 overwritten at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/execution.jl:306.
gpuarrays/random                              (30) |    29.26 |   1.04 |  3.6 |    2482.84 |  1135.29 |
      From worker 14:   � Warning: llvm_spirv() is deprecated, use the non-do-block form
      From worker 14:   �   caller = ip:0x0
      From worker 14:   � @ Core :-1
gpuarrays/indexing find                       (17) |    30.01 |   1.36 |  4.5 |    3068.52 |  1250.95 |
execution                                      (4) |    32.70 |   0.83 |  2.5 |    2177.05 |  1111.62 |
gpuarrays/indexing multidimensional           (18) |    36.66 |   1.26 |  3.4 |    3006.56 |  1190.53 |
      From worker 24:   terminate called after throwing an instance of 'sycl::_V1::runtime_error'
      From worker 24:     what():  Native API failed. Native API returns: -30 (PI_ERROR_INVALID_VALUE) -30 (PI_ERROR_INVALID_VALUE)
      From worker 24:
      From worker 24:   [1531517] signal (6.-6): Aborted
      From worker 24:   in expression starting at none:1
      From worker 24:   gsignal at /lib64/libc.so.6 (unknown line)
      From worker 24:   abort at /lib64/libc.so.6 (unknown line)
      From worker 24:   __verbose_terminate_handler at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/vterminate.cc:95
      From worker 24:   __terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      From worker 24:   terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      From worker 24:   __cxa_throw at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_throw.cc:98
      From worker 24:   _ZNK4sycl3_V16detail6plugin13checkPiResultINS0_13runtime_errorEEEv10_pi_result at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libsycl.so.6 (unknown line)
      From worker 24:   _ZN4sycl3_V16detail18make_kernel_bundleEmRKNS0_7contextEbNS0_12bundle_stateENS0_7backendE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libsycl.so.6 (unknown line)
      From worker 24:   _ZN6oneapi3mkl3gpu13build_programEPiPN4sycl3_V15queueEPvS7_iPKcS9_mcS9_Pb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl3gpuL22mkl_gpu_get_kernel_extEPiPN4sycl3_V15queueEiPKcS8_mcS8_S8_S8_mPKvmbb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl3gpu24mkl_gpu_get_spirv_kernelEPiPN4sycl3_V15queueEiPK22mkl_gpu_spirv_kernel_tPKcSB_ at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl3gpu40mkl_blas_gpu_sgemm_copybased_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl3gpu30mkl_blas_gpu_sgemm_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl3gpu19sgemm_sycl_internalEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl3gpu10sgemm_syclEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl4blas5sgemmERN4sycl3_V15queueE10MKL_LAYOUTNS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS3_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   _ZN6oneapi3mkl4blas12column_major4gemmERN4sycl3_V15queueENS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS4_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 24:   gemm at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/include/oneapi/mkl/blas/usm_decls.hpp:38 [inlined]
      From worker 24:   onemklSgemm at /workspace/srcdir/oneAPI.jl/deps/src/onemkl.cpp:181
      From worker 24:   onemklSgemm at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/support/liboneapi_support.jl:98
      From worker 24:   unknown function (ip: 0x1466e60f38a9)
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   gemm! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/wrappers.jl:1185
      From worker 24:   generic_matmatmul! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:215
      From worker 24:   gemm_wrapper! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:238 [inlined]
      From worker 24:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:161 [inlined]
      From worker 24:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:276 [inlined]
      From worker 24:   * at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:148
      From worker 24:   unknown function (ip: 0x1466e60f1b36)
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 24:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 24:   #compare#10 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:44
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 24:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 24:   compare at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:38
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:478 [inlined]
      From worker 24:   macro expansion at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite/linalg.jl:284 [inlined]
      From worker 24:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1586 [inlined]
      From worker 24:   #402 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite/linalg.jl:278
      From worker 24:   unknown function (ip: 0x1467656dafa6)
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   #test_linalg_mul!_matrix-matrix#401 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80
      From worker 24:   test_linalg_mul!_matrix-matrix at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80
      From worker 24:   unknown function (ip: 0x1467656d70c2)
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   #18 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:92
      From worker 24:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 24:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 24:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 24:   macro expansion at ./timing.jl:501 [inlined]
      From worker 24:   top-level scope at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:51
      From worker 24:   jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:903
      From worker 24:   ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 24:   eval at ./boot.jl:370 [inlined]
      From worker 24:   runtests at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:55
      From worker 24:   unknown function (ip: 0x1467656cafb6)
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 24:   jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 24:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 24:   #invokelatest#2 at ./essentials.jl:819
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 24:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 24:   invokelatest at ./essentials.jl:816
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 24:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 24:   #110 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
      From worker 24:   run_work_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
      From worker 24:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
      From worker 24:   #109 at ./task.jl:514
      From worker 24:   unknown function (ip: 0x1467656c97df)
      From worker 24:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 24:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 24:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 24:   start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
      From worker 24:   Allocations: 81399771 (Pool: 81335504; Big: 64267); GC: 122
gpuarrays/base                                (31) |    37.33 |   1.47 |  3.9 |    3964.49 |  1348.53 |
Worker 24 terminated.
gpuarrays/linalg/mul!/matrix-matrix           (24) |         failed at 2023-09-28T23:02:44.438
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514
      From worker 15:   terminate called after throwing an instance of 'sycl::_V1::exception'
      From worker 15:     what():  Level-Zero error:700000041879048196
      From worker 15:   On device: 'Intel(R) Data Center GPU Max 1100'
      From worker 15:   in kernel: oneapi::mkl::blas::sgemm_itcopy
      From worker 15:
      From worker 15:   [1531482] signal (6.-6): Aborted
      From worker 15:   in expression starting at none:1
      From worker 15:   gsignal at /lib64/libc.so.6 (unknown line)
      From worker 15:   abort at /lib64/libc.so.6 (unknown line)
      From worker 15:   __verbose_terminate_handler at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/vterminate.cc:95
      From worker 15:   __terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      From worker 15:   terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      From worker 15:   __cxa_throw at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_throw.cc:98
      From worker 15:   _ZN6oneapi3mkl3gpu13build_programEPiPN4sycl3_V15queueEPvS7_iPKcS9_mcS9_Pb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl3gpuL22mkl_gpu_get_kernel_extEPiPN4sycl3_V15queueEiPKcS8_mcS8_S8_S8_mPKvmbb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl3gpu24mkl_gpu_get_spirv_kernelEPiPN4sycl3_V15queueEiPK22mkl_gpu_spirv_kernel_tPKcSB_ at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl3gpu40mkl_blas_gpu_sgemm_copybased_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl3gpu30mkl_blas_gpu_sgemm_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl3gpu19sgemm_sycl_internalEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl3gpu10sgemm_syclEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl4blas5sgemmERN4sycl3_V15queueE10MKL_LAYOUTNS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS3_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   _ZN6oneapi3mkl4blas12column_major4gemmERN4sycl3_V15queueENS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS4_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 15:   gemm at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/include/oneapi/mkl/blas/usm_decls.hpp:38 [inlined]
      From worker 15:   onemklSgemm at /workspace/srcdir/oneAPI.jl/deps/src/onemkl.cpp:181
      From worker 15:   onemklSgemm at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/support/liboneapi_support.jl:98
      From worker 15:   unknown function (ip: 0x149cb4d646a9)
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   gemm! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/wrappers.jl:1185
      From worker 15:   generic_matmatmul! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:215
      From worker 15:   gemm_wrapper! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:238 [inlined]
      From worker 15:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:161 [inlined]
      From worker 15:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:276 [inlined]
      From worker 15:   * at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:148 [inlined]
      From worker 15:   power_by_squaring at ./intfuncs.jl:269
      From worker 15:   ^ at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/dense.jl:479 [inlined]
      From worker 15:   #414 at ./none:0
      From worker 15:   unknown function (ip: 0x149cb4d5b392)
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   #compare#10 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:44
      From worker 15:   compare at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:38
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:478 [inlined]
      From worker 15:   #413 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite/math.jl:20
      From worker 15:   #test_math_power#412 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80 [inlined]
      From worker 15:   test_math_power at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80
      From worker 15:   unknown function (ip: 0x149d1eb69ef2)
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   #18 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:92
      From worker 15:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 15:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 15:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 15:   macro expansion at ./timing.jl:501 [inlined]
      From worker 15:   top-level scope at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:51
      From worker 15:   jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:903
      From worker 15:   ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 15:   eval at ./boot.jl:370 [inlined]
      From worker 15:   runtests at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:55
      From worker 15:   unknown function (ip: 0x149d1eb5d276)
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 15:   jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 15:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 15:   #invokelatest#2 at ./essentials.jl:819
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 15:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 15:   invokelatest at ./essentials.jl:816
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 15:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 15:   #110 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
      From worker 15:   run_work_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
      From worker 15:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
      From worker 15:   #109 at ./task.jl:514
      From worker 15:   unknown function (ip: 0x149d1eb5ba9f)
      From worker 15:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 15:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 15:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 15:   start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
      From worker 15:   Allocations: 84419297 (Pool: 84357579; Big: 61718); GC: 134
Worker 15 terminated.
gpuarrays/math/power                          (15) |         failed at 2023-09-28T23:02:45.364
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514
gpuarrays/constructors                        (29) |    39.02 |   1.44 |  3.7 |    3463.04 |  1219.89 |
      From worker 7:    terminate called after throwing an instance of 'sycl::_V1::exception'
      From worker 7:      what():  Level-Zero error:700000041879048196
      From worker 7:    On device: 'Intel(R) Data Center GPU Max 1100'
      From worker 7:    in kernel: oneapi::mkl::blas::ssymm_iucopy
      From worker 7:
      From worker 7:    [1531465] signal (6.-6): Aborted
      From worker 7:    in expression starting at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/onemkl.jl:564
      From worker 7:    gsignal at /lib64/libc.so.6 (unknown line)
      From worker 7:    abort at /lib64/libc.so.6 (unknown line)
      From worker 7:    __verbose_terminate_handler at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/vterminate.cc:95
      From worker 7:    __terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      From worker 7:    terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      From worker 7:    __cxa_throw at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_throw.cc:98
      From worker 7:    _ZN6oneapi3mkl3gpu13build_programEPiPN4sycl3_V15queueEPvS7_iPKcS9_mcS9_Pb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    _ZN6oneapi3mkl3gpuL22mkl_gpu_get_kernel_extEPiPN4sycl3_V15queueEiPKcS8_mcS8_S8_S8_mPKvmbb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    _ZN6oneapi3mkl3gpu24mkl_gpu_get_spirv_kernelEPiPN4sycl3_V15queueEiPK22mkl_gpu_spirv_kernel_tPKcSB_ at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    _ZN6oneapi3mkl3gpu30mkl_blas_gpu_ssymm_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    _ZN6oneapi3mkl3gpu19ssymm_sycl_internalEPN4sycl3_V15queueE10MKL_LAYOUT8MKL_SIDE8MKL_UPLOllfPKflSA_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISF_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    _ZN6oneapi3mkl3gpu10ssymm_syclEPN4sycl3_V15queueE10MKL_LAYOUT8MKL_SIDE8MKL_UPLOllfPKflSA_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISF_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    _ZN6oneapi3mkl4blas5ssymmERN4sycl3_V15queueE10MKL_LAYOUTNS0_4sideENS0_4uploEllfPKflSA_lfPflNS1_12compute_modeERKSt6vectorINS3_5eventESaISE_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    _ZN6oneapi3mkl4blas12column_major4symmERN4sycl3_V15queueENS0_4sideENS0_4uploEllfPKflSA_lfPflNS1_12compute_modeERKSt6vectorINS4_5eventESaISE_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 7:    symm at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/include/oneapi/mkl/blas/usm_decls.hpp:64 [inlined]
      From worker 7:    onemklSsymm at /workspace/srcdir/oneAPI.jl/deps/src/onemkl.cpp:400
      From worker 7:    onemklSsymm at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/support/liboneapi_support.jl:296
      From worker 7:    unknown function (ip: 0x14e041fb6836)
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    symm! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/wrappers.jl:209
      From worker 7:    unknown function (ip: 0x14e041fb64b1)
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/onemkl.jl:579 [inlined]
      From worker 7:    macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 7:    macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/onemkl.jl:579 [inlined]
      From worker 7:    macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1586 [inlined]
      From worker 7:    macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/onemkl.jl:565 [inlined]
      From worker 7:    macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 7:    top-level scope at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/onemkl.jl:565
      From worker 7:    jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:903
      From worker 7:    jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:856
      From worker 7:    ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 7:    eval at ./boot.jl:370 [inlined]
      From worker 7:    include_string at ./loading.jl:1903
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    _include at ./loading.jl:1963
      From worker 7:    include at ./client.jl:478 [inlined]
      From worker 7:    #13 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:86 [inlined]
      From worker 7:    macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 7:    macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 7:    macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 7:    macro expansion at ./timing.jl:501 [inlined]
      From worker 7:    top-level scope at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:51
      From worker 7:    jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:903
      From worker 7:    ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 7:    eval at ./boot.jl:370 [inlined]
      From worker 7:    runtests at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:55
      From worker 7:    unknown function (ip: 0x14e0aae68f86)
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 7:    jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 7:    do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 7:    #invokelatest#2 at ./essentials.jl:819
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 7:    do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 7:    invokelatest at ./essentials.jl:816
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 7:    do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 7:    #110 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
      From worker 7:    run_work_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
      From worker 7:    macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
      From worker 7:    #109 at ./task.jl:514
      From worker 7:    unknown function (ip: 0x14e0aae677bf)
      From worker 7:    _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 7:    ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 7:    jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 7:    start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
      From worker 7:    Allocations: 63556319 (Pool: 63510779; Big: 45540); GC: 99
Worker 7 terminated.
onemkl                                         (7) |         failed at 2023-09-28T23:02:49.788
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514
kernelabstractions                             (5) |    46.00 |   2.06 |  4.5 |    3885.99 |  1286.61 |
      From worker 16:   terminate called after throwing an instance of 'sycl::_V1::runtime_error'
      From worker 16:     what():  Native API failed. Native API returns: -30 (PI_ERROR_INVALID_VALUE) -30 (PI_ERROR_INVALID_VALUE)
      From worker 16:
      From worker 16:   [1531484] signal (6.-6): Aborted
      From worker 16:   in expression starting at none:1
      From worker 16:   gsignal at /lib64/libc.so.6 (unknown line)
      From worker 16:   abort at /lib64/libc.so.6 (unknown line)
      From worker 16:   __verbose_terminate_handler at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/vterminate.cc:95
      From worker 16:   __terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      From worker 16:   terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      From worker 16:   __cxa_throw at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_throw.cc:98
      From worker 16:   _ZNK4sycl3_V16detail6plugin13checkPiResultINS0_13runtime_errorEEEv10_pi_result at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libsycl.so.6 (unknown line)
      From worker 16:   _ZN4sycl3_V16detail18make_kernel_bundleEmRKNS0_7contextEbNS0_12bundle_stateENS0_7backendE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libsycl.so.6 (unknown line)
      From worker 16:   _ZN6oneapi3mkl3gpu13build_programEPiPN4sycl3_V15queueEPvS7_iPKcS9_mcS9_Pb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl3gpuL22mkl_gpu_get_kernel_extEPiPN4sycl3_V15queueEiPKcS8_mcS8_S8_S8_mPKvmbb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl3gpu24mkl_gpu_get_spirv_kernelEPiPN4sycl3_V15queueEiPK22mkl_gpu_spirv_kernel_tPKcSB_ at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl3gpu40mkl_blas_gpu_sgemm_copybased_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl3gpu30mkl_blas_gpu_sgemm_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl3gpu19sgemm_sycl_internalEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl3gpu10sgemm_syclEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl4blas5sgemmERN4sycl3_V15queueE10MKL_LAYOUTNS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS3_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   _ZN6oneapi3mkl4blas12column_major4gemmERN4sycl3_V15queueENS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS4_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 16:   gemm at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/include/oneapi/mkl/blas/usm_decls.hpp:38 [inlined]
      From worker 16:   onemklSgemm at /workspace/srcdir/oneAPI.jl/deps/src/onemkl.cpp:181
      From worker 16:   onemklSgemm at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/support/liboneapi_support.jl:98
      From worker 16:   unknown function (ip: 0x14b48313b349)
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   gemm! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/wrappers.jl:1185
      From worker 16:   generic_matmatmul! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:215
      From worker 16:   gemm_wrapper! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:238 [inlined]
      From worker 16:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:391 [inlined]
      From worker 16:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:276
      From worker 16:   unknown function (ip: 0x14b48313af9d)
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 16:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 16:   #compare#10 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:44
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 16:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 16:   compare at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:38
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:478 [inlined]
      From worker 16:   macro expansion at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite/linalg.jl:272 [inlined]
      From worker 16:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1586 [inlined]
      From worker 16:   #399 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite/linalg.jl:260
      From worker 16:   unknown function (ip: 0x14b508dcc176)
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   #test_linalg_mul!_vector-matrix#398 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80
      From worker 16:   test_linalg_mul!_vector-matrix at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80
      From worker 16:   unknown function (ip: 0x14b508dc80b2)
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   #18 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:92
      From worker 16:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 16:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 16:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 16:   macro expansion at ./timing.jl:501 [inlined]
      From worker 16:   top-level scope at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:51
      From worker 16:   jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:903
      From worker 16:   ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 16:   eval at ./boot.jl:370 [inlined]
      From worker 16:   runtests at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:55
      From worker 16:   unknown function (ip: 0x14b508dbc266)
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 16:   jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 16:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 16:   #invokelatest#2 at ./essentials.jl:819
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 16:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 16:   invokelatest at ./essentials.jl:816
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 16:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 16:   #110 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
      From worker 16:   run_work_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
      From worker 16:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
      From worker 16:   #109 at ./task.jl:514
      From worker 16:   unknown function (ip: 0x14b508dbaa9f)
      From worker 16:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 16:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 16:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 16:   start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
      From worker 16:   Allocations: 98513399 (Pool: 98448292; Big: 65107); GC: 151
Worker 16 terminated.
gpuarrays/linalg/mul!/vector-matrix           (16) |         failed at 2023-09-28T23:02:55.590
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514
      From worker 27:   terminate called after throwing an instance of 'sycl::_V1::runtime_error'
      From worker 27:     what():  Native API failed. Native API returns: -30 (PI_ERROR_INVALID_VALUE) -30 (PI_ERROR_INVALID_VALUE)
      From worker 27:
      From worker 27:   [1531524] signal (6.-6): Aborted
      From worker 27:   in expression starting at none:1
      From worker 27:   gsignal at /lib64/libc.so.6 (unknown line)
      From worker 27:   abort at /lib64/libc.so.6 (unknown line)
      From worker 27:   __verbose_terminate_handler at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/vterminate.cc:95
      From worker 27:   __terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      From worker 27:   terminate at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      From worker 27:   __cxa_throw at /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/eh_throw.cc:98
      From worker 27:   _ZNK4sycl3_V16detail6plugin13checkPiResultINS0_13runtime_errorEEEv10_pi_result at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libsycl.so.6 (unknown line)
      From worker 27:   _ZN4sycl3_V16detail18make_kernel_bundleEmRKNS0_7contextEbNS0_12bundle_stateENS0_7backendE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libsycl.so.6 (unknown line)
      From worker 27:   _ZN6oneapi3mkl3gpu13build_programEPiPN4sycl3_V15queueEPvS7_iPKcS9_mcS9_Pb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl3gpuL22mkl_gpu_get_kernel_extEPiPN4sycl3_V15queueEiPKcS8_mcS8_S8_S8_mPKvmbb at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl3gpu24mkl_gpu_get_spirv_kernelEPiPN4sycl3_V15queueEiPK22mkl_gpu_spirv_kernel_tPKcSB_ at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl3gpu40mkl_blas_gpu_sgemm_copybased_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl3gpu30mkl_blas_gpu_sgemm_driver_syclEPiPN4sycl3_V15queueEP14blas_arg_usm_tP20mkl_gpu_event_list_t at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl3gpu19sgemm_sycl_internalEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl3gpu10sgemm_syclEPN4sycl3_V15queueE10MKL_LAYOUT13MKL_TRANSPOSES7_lllfPKflS9_lfPflNS0_4blas12compute_modeERKSt6vectorINS3_5eventESaISE_EElll at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl4blas5sgemmERN4sycl3_V15queueE10MKL_LAYOUTNS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS3_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   _ZN6oneapi3mkl4blas12column_major4gemmERN4sycl3_V15queueENS0_9transposeES7_lllfPKflS9_lfPflNS1_12compute_modeERKSt6vectorINS4_5eventESaISD_EE at /home/user/conda/julia/artifacts/5eb72274d03debeeb85d66cfc81455a55135de46/lib/libmkl_sycl.so.3 (unknown line)
      From worker 27:   gemm at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/include/oneapi/mkl/blas/usm_decls.hpp:38 [inlined]
      From worker 27:   onemklSgemm at /workspace/srcdir/oneAPI.jl/deps/src/onemkl.cpp:181
      From worker 27:   onemklSgemm at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/support/liboneapi_support.jl:98
      From worker 27:   unknown function (ip: 0x145879ca5b39)
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   gemm! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/wrappers.jl:1185
      From worker 27:   generic_matmatmul! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:215
      From worker 27:   syrk_wrapper! at /home/user/conda/julia/packages/oneAPI/g8ZUm/lib/mkl/linalg.jl:242 [inlined]
      From worker 27:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:376 [inlined]
      From worker 27:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:425 [inlined]
      From worker 27:   mul! at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:276 [inlined]
      From worker 27:   * at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LinearAlgebra/src/matmul.jl:148 [inlined]
      From worker 27:   unscaled_covzm at /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl:520
      From worker 27:   #covzm#106 at /home/user/conda/julia/packages/GPUArrays/5XhED/src/host/statistics.jl:41
      From worker 27:   covzm at /home/user/conda/julia/packages/GPUArrays/5XhED/src/host/statistics.jl:40 [inlined]
      From worker 27:   #covm#30 at /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl:561
      From worker 27:   covm at /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl:561 [inlined]
      From worker 27:   #cov#38 at /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl:584
      From worker 27:   cov at /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl:584
      From worker 27:   unknown function (ip: 0x145879c9f542)
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   #compare#10 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:44
      From worker 27:   compare at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:38
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:478 [inlined]
      From worker 27:   macro expansion at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite/statistics.jl:44 [inlined]
      From worker 27:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 27:   #424 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite/statistics.jl:42
      From worker 27:   #test_statistics#423 at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80 [inlined]
      From worker 27:   test_statistics at /home/user/conda/julia/packages/GPUArrays/5XhED/test/testsuite.jl:80
      From worker 27:   unknown function (ip: 0x1458d9afb692)
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   #18 at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:92
      From worker 27:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 27:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
      From worker 27:   macro expansion at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:52 [inlined]
      From worker 27:   macro expansion at ./timing.jl:501 [inlined]
      From worker 27:   top-level scope at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:51
      From worker 27:   jl_toplevel_eval_flex at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:903
      From worker 27:   ijl_toplevel_eval_in at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/toplevel.c:971
      From worker 27:   eval at ./boot.jl:370 [inlined]
      From worker 27:   runtests at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/setup.jl:55
      From worker 27:   unknown function (ip: 0x1458d9acffb6)
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 27:   jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 27:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 27:   #invokelatest#2 at ./essentials.jl:819
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 27:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 27:   invokelatest at ./essentials.jl:816
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 27:   do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
      From worker 27:   #110 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
      From worker 27:   run_work_thunk at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
      From worker 27:   macro expansion at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
      From worker 27:   #109 at ./task.jl:514
      From worker 27:   unknown function (ip: 0x1458d9ace7df)
      From worker 27:   _jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
      From worker 27:   ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
      From worker 27:   jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
      From worker 27:   start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
      From worker 27:   Allocations: 117563920 (Pool: 117494873; Big: 69047); GC: 173
gpuarrays/statistics                          (27) |         failed at 2023-09-28T23:03:05.655
Worker 27 terminated.
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:947
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:955
 [3] unsafe_read
   @ ./io.jl:761 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:760
 [5] read!
   @ ./io.jl:762 [inlined]
 [6] deserialize_hdr_raw
   @ /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:514
gpuarrays/reductions/== isequal               (32) |    65.95 |   3.04 |  4.6 |    6685.62 |  1323.56 |
gpuarrays/reductions/mapreducedim!_large      (23) |    80.76 |   2.43 |  3.0 |    9056.49 |  2060.47 |
gpuarrays/reductions/reducedim!               (13) |    96.28 |   4.48 |  4.7 |    8674.33 |  1440.02 |
gpuarrays/linalg                              (14) |    98.73 |   4.15 |  4.2 |   10517.06 |  1600.50 |
gpuarrays/reductions/mapreducedim!            (34) |   131.76 |   5.17 |  3.9 |   11277.18 |  1592.02 |
gpuarrays/reductions/reduce                   (35) |   132.63 |   8.38 |  6.3 |   15831.72 |  1591.21 |
gpuarrays/reductions/sum prod                 (36) |   157.49 |   5.81 |  3.7 |   12516.11 |  1683.36 |
gpuarrays/linalg/norm                         (26) |   164.21 |   7.28 |  4.4 |   12328.88 |  1863.63 |
gpuarrays/reductions/mapreduce                (28) |   184.80 |  11.99 |  6.5 |   25354.22 |  1604.10 |
gpuarrays/broadcasting                        (33) |   188.83 |   9.10 |  4.8 |   15916.82 |  1877.80 |
gpuarrays/reductions/minimum maximum extrema  (21) |   194.82 |   9.56 |  4.9 |   16913.41 |  1966.68 |
Testing finished in 3 minutes, 17 seconds, 994 milliseconds
device/intrinsics: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(11)
Worker 3 failed running test examples:
Some tests did not pass: 1 passed, 1 failed, 0 errored, 0 broken.
examples: Test Failed at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/examples.jl:21
  Expression: success(pipeline(`$cmd $(joinpath(examples_dir, example))`, stderr = stderr))

Stacktrace:
 [1] record(ts::Test.DefaultTestSet, t::Union{Test.Error, Test.Fail})
   @ Test /home/user/julia/julia-1.9.3/share/julia/stdlib/v1.9/Test/src/Test.jl:1041
 [2] top-level scope
   @ /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:360
 [3] include(fname::String)
   @ Base.MainInclude ./client.jl:478
 [4] top-level scope
   @ none:6
 [5] eval
   @ ./boot.jl:370 [inlined]
 [6] exec_options(opts::Base.JLOptions)
   @ Base ./client.jl:280
 [7] _start()
   @ Base ./client.jl:522
gpuarrays/linalg/mul!/matrix-matrix: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(24)
gpuarrays/math/power: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(15)
onemkl: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(7)
gpuarrays/linalg/mul!/vector-matrix: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(16)
gpuarrays/statistics: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(27)

Test Summary:                                  | Pass  Fail  Error  Broken  Total  Time
  Overall                                      | 8597     1      6       7   8611      
    sycl                                       |    4                           4      
    pointer                                    |   35                          35      
    level-zero                                 |   35                          35      
    gpuarrays/interface                        |    7                           7      
    array                                      |   18                          18      
    gpuarrays/math/intrinsics                  |   12                          12      
    random                                     |   57                          57      
    device/intrinsics                          |                 1              1      
    gpuarrays/uniformscaling                   |   56                          56      
    gpuarrays/indexing scalar                  |  477                         477      
    gpuarrays/reductions/any all count         |  101                         101      
    examples                                   |    1     1                     2      
    gpuarrays/random                           |   62                          62      
    gpuarrays/indexing find                    |   45                          45      
    execution                                  |   47                          47      
    gpuarrays/indexing multidimensional        |   70                          70      
    gpuarrays/base                             |   75                          75      
    gpuarrays/linalg/mul!/matrix-matrix        |                 1              1      
    gpuarrays/math/power                       |                 1              1      
    gpuarrays/constructors                     |  900                         900      
    onemkl                                     |                 1              1      
    kernelabstractions                         | 2168                    7   2175      
    gpuarrays/linalg/mul!/vector-matrix        |                 1              1      
    gpuarrays/statistics                       |                 1              1      
    gpuarrays/reductions/== isequal            |  312                         312      
    gpuarrays/reductions/mapreducedim!_large   |   50                          50      
    gpuarrays/reductions/reducedim!            |  192                         192      
    gpuarrays/linalg                           |  275                         275      
    gpuarrays/reductions/mapreducedim!         |  312                         312      
    gpuarrays/reductions/reduce                |  264                         264      
    gpuarrays/reductions/sum prod              |  862                         862      
    gpuarrays/linalg/norm                      |  696                         696      
    gpuarrays/reductions/mapreduce             |  396                         396      
    gpuarrays/broadcasting                     |  402                         402      
    gpuarrays/reductions/minimum maximum extrema |  666                         666      
    FAILURE

Error in testset device/intrinsics:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(11)
Error in testset examples:
Test Failed at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/examples.jl:21
  Expression: success(pipeline(`$cmd $(joinpath(examples_dir, example))`, stderr = stderr))

Error in testset gpuarrays/linalg/mul!/matrix-matrix:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(24)
Error in testset gpuarrays/math/power:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(15)
Error in testset onemkl:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(7)
Error in testset gpuarrays/linalg/mul!/vector-matrix:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(16)
Error in testset gpuarrays/statistics:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(27)
ERROR: LoadError: Test run finished with errors
in expression starting at /home/user/conda/julia/packages/oneAPI/g8ZUm/test/runtests.jl:391
pure virtual method called
terminate called without an active exception

[1522051] signal (6.-6): Aborted
in expression starting at none:0
ERROR: Package oneAPI errored during testing

I also run some example codes and they worked fine.

So I call it ok for PoC.
But I agree with @maleadt that it would be lovely if this would happen automatically of these instructions would be simplified to something like directly install from the repo...

Thank you @pengtu for your help!

Also, I hope that you all will find errors during the testing to be informative to avoid those in the future.

@maleadt
Copy link
Member

maleadt commented Sep 29, 2023

You are mixing things up in your reply, so I'm not sure now what does and doesn't work. Did you use @pengtu's instructions to use a local toolchain? Or did you successfully switch to the debug build of our artifacts? Both approaches cannot be combined, and FWIW using a local toolchain doesn't help us unless you specify exactly what version it is and how it was build (is why I didn't recommend doing so).
The errors you are reporting now are MKL errors, which are different from what you originally reported.

@thoth291
Copy link
Author

Yes, @maleadt , please excuse me - I was not clear in my previous reply.

I followed @pengtu instructions above and build by cloning and building personal copy of oneAPI.
That's all what I did.

Since it worked - that's enough for the PoC I'm working on.
But it would be nice to fix this issue in the future.

My additional report of test failures might indeed be unrelated to the topic - but since I don't really care about them at the moment - (simple tests are passing and everything seems to be working) - I decided not to share it as separate request - but still keep it here for the record.

Thank you both for your help, again!
I hope it clarified things a bit.

I'll keep this thread open until I conclude my PoC (next week) and if everything goes well - I would consider that this ticket could be closed.

Thanks.

@maleadt
Copy link
Member

maleadt commented Feb 26, 2024

I'm updating the toolchain in #388, hopefully that helps.

@maleadt maleadt closed this as completed Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maleadt @pengtu @thoth291