PackageCompiler CUDA app hangs at runtime

[paulxshen]

Dummy app made by PackageCompiler.jl with CUDA.jl as sole dep hangs at runtime on Ubuntu in Google Colab. Works when CUDA removed as dep. Works with CUDA on AWS Ubuntu VM and Windows. I need it on Colab because we're using it as a free GPU ready platform for product demo. I suspect it's something like a dependency loading issue... Julia 11.4 CUDA 5.7 . Full MWE at https://colab.research.google.com/drive/1TVpIO7E_GipQ0wXSQYDE_1F6OaTkAQhz?usp=sharing Thanks!

[dimer116]

I have the same issue, see JuliaLang/PackageCompiler.jl#1026 . I used git bisect to determine which commit causes this between versions 5.5.0 and 5.4.3, it was 763164d. I then used gdb with the following results ("CUDATest" is the name of my dummy project), first I ran the program inside gdb, pressed ctrl+c after a short delay. Then,

(gdb) thread apply all bt

Thread 3 (Thread 0x7fffcdbff640 (LWP 328466) "CUDATest"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fffd00a68e0 <thread_status+96>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fffd00a68e0 <thread_status+96>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fffd00a68e0 <thread_status+96>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7493a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fffd00a6890 <thread_status+16>, cond=0x7fffd00a68b8 <thread_status+56>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7fffd00a68b8 <thread_status+56>, mutex=0x7fffd00a6890 <thread_status+16>) at ./nptl/pthread_cond_wait.c:627
#5  0x00007fffce39532b in blas_thread_server () from /home/holtra/Desktop/misc/CUDA test/test_app/bin/../lib/julia/libopenblas64_.so
#6  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007ffff7526850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7ffff03ff640 (LWP 328465) "CUDATest"):
#0  __pthread_kill_implementation (no_tid=3, signo=2, threadid=140737224111680) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_esrch (threadid=140737224111680, signo=signo@entry=2) at ./nptl/pthread_kill.c:108
#2  0x00007ffff78c67be in jl_ignore_sigint () at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/signal-handling.c:237
#3  signal_listener (arg=<optimized out>) at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/signals-unix.c:843
#4  0x00007ffff7494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5  0x00007ffff7526850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7ffff7e53780 (LWP 328460) "CUDATest"):
#0  0x00007ffff7525b68 in __GI_epoll_pwait (epfd=5, events=0x7fff4a7fc3e0, maxevents=1024, timeout=-1, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1  0x00007ffff794f914 in uv__io_poll (loop=0x7ffff7e0f900 <default_loop_struct>, timeout=-1) at src/unix/linux.c:1404
#2  0x00007ffff7935f59 in uv_run (loop=0x7ffff7e0f900 <default_loop_struct>, mode=UV_RUN_ONCE) at src/unix/core.c:430
#3  0x00007ffff78aa6d9 in ijl_task_get_next (trypoptask=<optimized out>, q=<optimized out>, checkempty=<optimized out>) at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/scheduler.c:522
#4  0x00007fffd4a7fc53 in julia_poptask_91221 () at task.jl:1012
#5  0x00007fffd3e7b763 in julia_wait_90763 () at task.jl:1021
#6  0x00007fffd47c69c0 in julia_#wait#731_90774 () at condition.jl:130
#7  0x00007fffd4c48d87 in wait () at condition.jl:125
#8  julia__trywait_91698 () at asyncevent.jl:145
#9  0x00007fffd34ab08b in julia_profile_printing_listener_100570 () at Base.jl:577
#10 0x00007fffd292ef06 in julia_#1190_100643 () at Base.jl:617
#11 0x00007fffd4c44aff in jfptr_YY.1190_100644.1 () from /home/holtra/Desktop/misc/CUDA test/test_app/lib/julia/sys.so
#12 0x00007ffff7875d03 in jl_apply (nargs=1, args=0x7fffeb5fc1d0) at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157
#13 start_task () at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/task.c:1202

This is just the last three of the all threads, all threads up to thread 17 had similar message as thread 3. I also ran

(gdb) bt
#0  __pthread_kill_implementation (no_tid=3, signo=2, threadid=140737224111680)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_esrch (threadid=140737224111680, signo=signo@entry=2)
    at ./nptl/pthread_kill.c:108
#2  0x00007ffff78c67be in jl_ignore_sigint ()
    at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/signal-handling.c:237
#3  signal_listener (arg=<optimized out>)
    at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/signals-unix.c:843
#4  0x00007ffff7494ac3 in start_thread (arg=<optimized out>)
    at ./nptl/pthread_create.c:442
#5  0x00007ffff7526850 in clone3 ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[maleadt]

CUDA_Driver_jll@0.10 is where I switched from dlopening the forwards compatible driver (and dlcloseing it again if needed, which turned out to be fragile) to spawning a new process: JuliaPackaging/Yggdrasil#9214. It's not improbable that there's some corner cases related to spawning a process during init that aren't handled properly in the case of PackageCompiler.

[maleadt]

A potential workaround is to disable the forwards-compatible driver by setting the compat=false preference on the driver JLL. If it's caused by the dlopen, of course. The first step to actually fixing this would be to reduce the packages involved to the bare minimum, i.e., creating an MWE that reproduces the issue.

[dimer116]

Yes that seems to be a nice workaround. I ran export JULIA_CUDA_USE_COMPAT=false before starting the program and it worked. Running with JULIA_DEBUG you get spammed this line infinitely otherwise. I will see if I get time to look into this issue more and make a permanent fix. Thanks for your help, appreciate it.

[lpawela]

Something similar happens on my machines. Consider the following example

module MWE

using CUDA

function julia_main()::Cint
    a = CUDA.rand(1000, 1000)
    b = CUDA.rand(1000, 1000)
    a * b
    return 0
end

end

and Project.toml

name = "MWE"
uuid = "75898d63-9d2a-4347-a948-6026498e897d"

[deps]
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"

[compat]
CUDA = "~5.4"

This is compiled with julia -e 'using PackageCompiler; create_app(".", "compiled";)' and runs without problem. When I change the version to CUDA = ~5.8 the compilation seems to go fine, but when I try to launch the produced executable it just consumes all available RAM.

Setting export JULIA_CUDA_USE_COMPAT=false with CUDA = ~5.8 allows the MWE to run without any memory issues.

[maleadt]

Setting export JULIA_CUDA_USE_COMPAT=false with CUDA = ~5.8 allows the MWE to run without any memory issues.

Note that you can also set this using a preference, which may be easier than having to set an env var.

[lpawela]

@maleadt are there any panelties for using this env var? Can this impact performance/stability?

[maleadt]

Yes, it forces you to use the driver you have installed, and disables the userspace upgrade.

Given that the issue here has been fairly well isolated, I think it would be time better spent adding some debugging to CUDA_Driver_jll's init to see why the execution hangs. There's already a workaround in place for PackageCompiler.jl to prevent a loop during init: https://github.com/JuliaPackaging/Yggdrasil/blob/479e01f0e802a203ec06d632ab378ff5c54f40fb/C/CUDA/CUDA_Driver/init.jl#L76-L83

[paulxshen]

works!

for colab when compiling, need LD_LIBRARY_PATH=''

for colab when running the compiled, set JULIA_CUDA_USE_COMPAT='0' and LD_LIBRARY_PATH='' . if other apps use LD_LIBRARY_PATH, set LD_LIBRARY_PATH='a/' and do

mkdir /a
mv /usr/lib64-nvidia/libcu* /a
mv /usr/lib64-nvidia/libnv* /a

which moves cuda stuff out of there but still let other apps use LD_LIBRARY_PATH

btw export doesn't work on colab so need os.environ['LD_LIBRARY_PATH']=...