Could not load library cudnn_ops_infer64_8.dll

[jonniedie]

I'm getting the following error:

julia> using CUDA

julia> A = CuArray(rand(Float32, 100_000))
┌ Error: Error during initialization of CUDA.jl
│   exception =
│    could not load library "C:\Users\jdiegelm\.julia\artifacts\d1e2f396182c9319f1ae2e7f020a3b969ccb8cc6\bin\cudnn_ops_infer64_8.dll"
│    Access is denied.
│    Stacktrace:
│      [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
│        @ Base.Libc.Libdl .\libdl.jl:114
│      [2] dlopen
│        @ .\libdl.jl:114 [inlined]
│      [3] use_artifact_cudnn(release::VersionNumber)
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\deps\bindeps.jl:293
│      [4] use_artifact_cuda()
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\deps\bindeps.jl:198
│      [5] __init_dependencies__()
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\deps\bindeps.jl:379
│      [6] __runtime_init__()
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\initialization.jl:103
│      [7] macro expansion
│        @ ~\.julia\packages\CUDA\Zmd60\src\initialization.jl:32 [inlined]
│      [8] macro expansion
│        @ .\lock.jl:209 [inlined]
│      [9] _functional(show_reason::Bool)
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\initialization.jl:26
│     [10] functional(show_reason::Bool)
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\initialization.jl:19
│     [11] libcuda()
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\initialization.jl:47
│     [12] macro expansion
│        @ ~\.julia\packages\CUDA\Zmd60\lib\cudadrv\libcuda.jl:29 [inlined]
│     [13] macro expansion
│        @ ~\.julia\packages\CUDA\Zmd60\lib\cudadrv\error.jl:102 [inlined]
│     [14] cuDeviceGet
│        @ ~\.julia\packages\CUDA\Zmd60\lib\utils\call.jl:26 [inlined]
│     [15] CuDevice
│        @ ~\.julia\packages\CUDA\Zmd60\lib\cudadrv\devices.jl:25 [inlined]
│     [16] initialize_thread(tid::Int64)
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\state.jl:121
│     [17] prepare_cuda_call()
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\state.jl:80
│     [18] device
│        @ ~\.julia\packages\CUDA\Zmd60\src\state.jl:227 [inlined]
│     [19] alloc
│        @ ~\.julia\packages\CUDA\Zmd60\src\pool.jl:271 [inlined]
│     [20] CuArray{Float32, 1}(#unused#::UndefInitializer, dims::Tuple{Int64})
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\array.jl:20
│     [21] CuArray
│        @ ~\.julia\packages\CUDA\Zmd60\src\array.jl:190 [inlined]
│     [22] CuArray(A::Vector{Float32})
│        @ CUDA ~\.julia\packages\CUDA\Zmd60\src\array.jl:198
│     [23] top-level scope
│        @ REPL[2]:1
│     [24] eval(m::Module, e::Any)
│        @ Core .\boot.jl:360
│     [25] eval_user_input(ast::Any, backend::REPL.REPLBackend)
│        @ REPL C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\REPL\src\REPL.jl:139
│     [26] repl_backend_loop(backend::REPL.REPLBackend)
│        @ REPL C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\REPL\src\REPL.jl:200
│     [27] start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
│        @ REPL C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\REPL\src\REPL.jl:185
│     [28] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
│        @ REPL C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\REPL\src\REPL.jl:317
│     [29] run_repl(repl::REPL.AbstractREPL, consumer::Any)
│        @ REPL C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\REPL\src\REPL.jl:305
│     [30] (::Base.var"#878#880"{Bool, Bool, Bool})(REPL::Module)
│        @ Base .\client.jl:387
│     [31] #invokelatest#2
│        @ .\essentials.jl:707 [inlined]
│     [32] invokelatest
│        @ .\essentials.jl:706 [inlined]
│     [33] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
│        @ Base .\client.jl:372
│     [34] exec_options(opts::Base.JLOptions)
│        @ Base .\client.jl:302
│     [35] _start()
│        @ Base .\client.jl:485
└ @ CUDA C:\Users\jdiegelm\.julia\packages\CUDA\Zmd60\src\initialization.jl:34
ERROR: CUDA.jl did not successfully initialize, and is not usable.
If you did not see any other error message, try again in a new session
with the JULIA_DEBUG environment variable set to 'CUDA'.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] libcuda()
    @ CUDA ~\.julia\packages\CUDA\Zmd60\src\initialization.jl:48
  [3] macro expansion
    @ ~\.julia\packages\CUDA\Zmd60\lib\cudadrv\libcuda.jl:29 [inlined]
  [4] macro expansion
    @ ~\.julia\packages\CUDA\Zmd60\lib\cudadrv\error.jl:102 [inlined]
  [5] cuDeviceGet
    @ ~\.julia\packages\CUDA\Zmd60\lib\utils\call.jl:26 [inlined]
  [6] CuDevice
    @ ~\.julia\packages\CUDA\Zmd60\lib\cudadrv\devices.jl:25 [inlined]
  [7] initialize_thread(tid::Int64)
    @ CUDA ~\.julia\packages\CUDA\Zmd60\src\state.jl:121
  [8] prepare_cuda_call()
    @ CUDA ~\.julia\packages\CUDA\Zmd60\src\state.jl:80
  [9] device
    @ ~\.julia\packages\CUDA\Zmd60\src\state.jl:227 [inlined]
 [10] alloc
    @ ~\.julia\packages\CUDA\Zmd60\src\pool.jl:271 [inlined]
 [11] CuArray{Float32, 1}(#unused#::UndefInitializer, dims::Tuple{Int64})
    @ CUDA ~\.julia\packages\CUDA\Zmd60\src\array.jl:20
 [12] CuArray
    @ ~\.julia\packages\CUDA\Zmd60\src\array.jl:190 [inlined]
 [13] CuArray(A::Vector{Float32})
    @ CUDA ~\.julia\packages\CUDA\Zmd60\src\array.jl:198
 [14] top-level scope
    @ REPL[2]:1

Here's my CUDA version number:

(@v1.6) pkg> st CUDA
      Status `C:\Users\jdiegelm\.julia\environments\v1.6\Project.toml`
  [052768ef] CUDA v2.6.1

And here's my versioninfo:

julia> versioninfo()
Julia Version 1.6.0-rc1
Commit a58bdd9010 (2021-02-06 15:49 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Xeon(R) E-2276M  CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

[maleadt]

Did you try the master branch, as the bug filing template suggests?
What are the full permissions of the library that fails to load, here C:\Users\jdiegelm\.julia\artifacts\d1e2f396182c9319f1ae2e7f020a3b969ccb8cc6\bin\cudnn_ops_infer64_8.dll?

[jonniedie]

No, I hadn't. It looks like I'm not getting the error on master. Thanks.

[visr]

I was just looking at this with @mjr-deltares, who had the same error with the cudnn files on Windows, both on CUDA.jl 3.1.0 and master. He fixed the issue by manually making the cudnn DLL files in the artifact executable, like described here: https://discourse.julialang.org/t/private-package-dll-gives-access-is-denied/58295. Otherwise it would only work when starting julia as an administrator.

I'm also on Windows, and could not reproduce this myself, so I have no idea how it can be that my dll files are executable, and his are not. But just leaving this comment here in case anyone has an idea.

Looking at the Yggdracil build scripts, I see that CUDA itself has

    # Fix permissions
    chmod +x ${bindir}/*.{exe,dll}

but CUDNN does not. Perhaps adding this would help?

[maleadt]

Yeah that sounds like a good fix.

[visr]

Ok, done in JuliaPackaging/Yggdrasil#2932