Could not load CUTENSOR artifact dll on Windows 10

[stillyslalom]

Describe the bug

When installing CUDA#master in Julia 1.5 on Windows 10 (after nuking .julia/ to start fresh), a (504) Gateway Timeout exception is thrown while downloading artifacts, followed by an error resulting from failure to load cutensor.dll, despite the existence of the library at the specified path.

To reproduce

The Minimal Working Example (MWE) for this bug:

PS C:\Users\alexa\AppData\Local\Programs\Julia 1.5.0\bin> Set-Variable -Name "JULIA_DEBUG" -Value "CUDA"

julia> using CUDA
[ Info: Precompiling CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]

julia> CUDA.version()
Downloading artifact: CUDA110
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (504) Gateway Timeout."
At line:12 char:1
+ $webclient.DownloadFile("https://pkg.julialang.org/artifact/4747fe73a ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException
ERROR: could not load library "C:\Users\alexa\.julia\artifacts\804435c7dbe9d002d6caa0a8b3596d11a6faa778\bin\cutensor.dll"
The specified module could not be found.
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Libdl\src\Libdl.jl:109
 [2] dlopen at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Libdl\src\Libdl.jl:109 [inlined] (repeats 2 times)
 [3] use_artifact_cutensor(::VersionNumber) at C:\Users\alexa\.julia\packages\CUDA\YU1DS\deps\bindeps.jl:315
 [4] use_artifact_cuda() at C:\Users\alexa\.julia\packages\CUDA\YU1DS\deps\bindeps.jl:177
 [5] __init_dependencies__() at C:\Users\alexa\.julia\packages\CUDA\YU1DS\deps\bindeps.jl:342
 [6] __runtime_init__() at C:\Users\alexa\.julia\packages\CUDA\YU1DS\src\initialization.jl:112
 [7] (::CUDA.var"#609#610"{Bool})() at C:\Users\alexa\.julia\packages\CUDA\YU1DS\src\initialization.jl:32
 [8] lock(::CUDA.var"#609#610"{Bool}, ::ReentrantLock) at .\lock.jl:161
 [9] _functional(::Bool) at C:\Users\alexa\.julia\packages\CUDA\YU1DS\src\initialization.jl:26
 [10] functional(::Bool) at C:\Users\alexa\.julia\packages\CUDA\YU1DS\src\initialization.jl:19
 [11] libcuda() at C:\Users\alexa\.julia\packages\CUDA\YU1DS\src\initialization.jl:50
 [12] (::CUDA.var"#676#cache_fptr!#7")() at C:\Users\alexa\.julia\packages\CUDA\YU1DS\lib\utils\call.jl:31
 [13] macro expansion at C:\Users\alexa\.julia\packages\CUDA\YU1DS\lib\utils\call.jl:39 [inlined]
 [14] macro expansion at C:\Users\alexa\.julia\packages\CUDA\YU1DS\lib\cudadrv\libcuda.jl:28 [inlined]
 [15] macro expansion at C:\Users\alexa\.julia\packages\CUDA\YU1DS\lib\cudadrv\error.jl:108 [inlined]
 [16] cuDriverGetVersion(::Base.RefValue{Int32}) at C:\Users\alexa\.julia\packages\CUDA\YU1DS\lib\utils\call.jl:93
 [17] version() at C:\Users\alexa\.julia\packages\CUDA\YU1DS\lib\cudadrv\version.jl:10
 [18] top-level scope at REPL[3]:1

julia> ispath(raw"C:\Users\alexa\.julia\artifacts\804435c7dbe9d002d6caa0a8b3596d11a6faa778\bin\cutensor.dll")
true

Manifest.toml

# This file is machine-generated - editing it directly is not advised

[[AbstractFFTs]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "051c95d6836228d120f5f4b984dd5aba1624f716"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "0.5.0"

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "0fac443759fa829ed8066db6cf1077d888bb6573"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "2.0.2"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[BinaryProvider]]
deps = ["Libdl", "Logging", "SHA"]
git-tree-sha1 = "ecdec412a9abc8db54c0efc5548c64dfce072058"
uuid = "b99e7846-7c00-51b0-8f62-c81ae34c0232"
version = "0.5.10"

[[CEnum]]
git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.1"

[[CUDA]]
deps = ["AbstractFFTs", "Adapt", "BinaryProvider", "CEnum", "DataStructures", "ExprTools", "GPUArrays", "GPUCompiler", "LLVM", "Libdl", "LinearAlgebra", "Logging", "MacroTools", "NNlib", "Pkg", "Printf", "Random", "Reexport", "Requires", "SparseArrays", "Statistics", "TimerOutputs"]
git-tree-sha1 = "33975e799350f944295c6ccfb481cd902a54aa20"
repo-rev = "master"
repo-url = "https://github.com/JuliaGPU/CUDA.jl.git"
uuid = "052768ef-5323-5732-b1bb-66c8b64840ba"
version = "1.2.1"

[[DataStructures]]
deps = ["InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "88d48e133e6d3dd68183309877eac74393daa7eb"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.17.20"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[ExprTools]]
git-tree-sha1 = "6f0517056812fd6aa3af23d4b70d5325a2ae4e95"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.1"

[[GPUArrays]]
deps = ["AbstractFFTs", "Adapt", "LinearAlgebra", "Printf", "Random", "Serialization"]
git-tree-sha1 = "9027d6128645d227fa9ba7d5b3aa00af696b9aaf"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "5.0.0"

[[GPUCompiler]]
deps = ["DataStructures", "InteractiveUtils", "LLVM", "Libdl", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "d6db068d8373bb45ac17e14744cc1d3681d0dbc7"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.5.5"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[LLVM]]
deps = ["CEnum", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "a662366a5d485dee882077e8da3e1a95a86d097f"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "2.0.0"

[[LibGit2]]
deps = ["Printf"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "f7d2e3f654af75f01ec49be82c231c382214223a"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.5"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[NNlib]]
deps = ["Libdl", "LinearAlgebra", "Pkg", "Requires", "Statistics"]
git-tree-sha1 = "8ec4693a5422f0b064ce324f59351f24aa474893"
uuid = "872c559c-99b0-510c-b3b7-b6c96a88d5cd"
version = "0.7.4"

[[OrderedCollections]]
git-tree-sha1 = "293b70ac1780f9584c89268a6e2a560d938a7065"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.3.0"

[[Pkg]]
deps = ["Dates", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "UUIDs"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[Reexport]]
deps = ["Pkg"]
git-tree-sha1 = "7b1d07f411bc8ddb7977ec7f377b97b158514fe0"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "0.2.0"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "d37400976e98018ee840e0ca4f9d20baa231dc6b"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.0.1"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[TimerOutputs]]
deps = ["Printf"]
git-tree-sha1 = "f458ca23ff80e46a630922c555d838303e4b9603"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.6"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

Expected behavior

Automatic download/installation of binary artifacts

Version info

Details on Julia:

julia> versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 9 4900HS with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, znver1)
Environment:
  JULIA_NUM_THREADS = 8

Details on CUDA:

julia> CUDA.versioninfo()
ERROR: AssertionError: CUDA.jl did not successfully initialize, and is not usable.
Stacktrace:
 [1] macro expansion at C:\Users\alexa\.julia\packages\CUDA\YU1DS\src\initialization.jl:50 [inlined]
 [2] toolkit_version at C:\Users\alexa\.julia\packages\CUDA\YU1DS\deps\bindeps.jl:25 [inlined]
 [3] versioninfo(::Base.TTY) at C:\Users\alexa\.julia\packages\CUDA\YU1DS\src\utilities.jl:43 (repeats 2 times)
 [4] top-level scope at REPL[5]:1

Additional context

PS C:\Users\alexa> nvidia-smi
Thu Aug 06 12:06:19 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 451.67       Driver Version: 451.67       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 206... WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   27C    P8    12W /  N/A |    164MiB /  6144MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

[maleadt]

Can you list the files in C:\Users\alexa\.julia\artifacts\804435c7dbe9d002d6caa0a8b3596d11a6faa778\bin\?

[stillyslalom]

cutensor.dll is the only file in that folder. There are two other artifact folders containing various other CUDA dlls:

[maleadt]

I'm not sure why the DLL wouldn't load then (The specified module could not be found.), and I don't have much Windows experience to debug that. I'll see if I can reproduce.

[stillyslalom]

The cutensor artifact is intact, which rules out download corruption. CUDA 11 support is a pretty recent development, right? Any chance the artifact itself is broken?

julia> artifact_toml = joinpath(pkgdir(CUDA), "Artifacts.toml")
"C:\\Users\\alexa\\.julia\\packages\\CUDA\\YU1DS\\Artifacts.toml"

julia> artifact_hash("CUTENSOR_CUDA110", artifact_toml) |> verify_artifact
true

This is the subset of dlls that's loaded during CUDA.jl initialization before the error:

julia> setdiff(finaldlls, initdlls)
23-element Array{String,1}:
 "C:\\Users\\alexa\\AppData\\Local\\Programs\\Julia 1.5.0\\bin\\libdSFMT.DLL"
 "C:\\WINDOWS\\SYSTEM32\\nvcuda.DLL"
 "C:\\WINDOWS\\SYSTEM32\\VERSION.DLL"
 "C:\\WINDOWS\\system32\\DriverStore\\FileRepository\\nvami.inf_amd64_aceb5b140d1d131e\\nvcuda64.dll"
 "C:\\WINDOWS\\SYSTEM32\\nvapi64.dll"
 "C:\\WINDOWS\\System32\\SETUPAPI.DLL"
 "C:\\WINDOWS\\System32\\cfgmgr32.dll"
 "C:\\WINDOWS\\System32\\SHLWAPI.DLL"
 "C:\\WINDOWS\\SYSTEM32\\dxgi.dll"
 "C:\\Users\\alexa\\AppData\\Local\\Programs\\Julia 1.5.0\\bin\\libopenlibm.DLL"
 "C:\\Users\\alexa\\.julia\\artifacts\\4747fe73a29e16337cb33702edae26b4e681ae52\\bin\\cublas64_11.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\4747fe73a29e16337cb33702edae26b4e681ae52\\bin\\cublasLt64_11.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\4747fe73a29e16337cb33702edae26b4e681ae52\\bin\\cusparse64_11.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\4747fe73a29e16337cb33702edae26b4e681ae52\\bin\\cusolver64_10.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\4747fe73a29e16337cb33702edae26b4e681ae52\\bin\\cufft64_10.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\4747fe73a29e16337cb33702edae26b4e681ae52\\bin\\curand64_10.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\430f2638c1882d64e7673183c57b1564edaba69f\\bin\\cudnn_ops_infer64_8.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\430f2638c1882d64e7673183c57b1564edaba69f\\bin\\cudnn_ops_train64_8.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\430f2638c1882d64e7673183c57b1564edaba69f\\bin\\cudnn_cnn_infer64_8.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\430f2638c1882d64e7673183c57b1564edaba69f\\bin\\cudnn_cnn_train64_8.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\430f2638c1882d64e7673183c57b1564edaba69f\\bin\\cudnn_adv_infer64_8.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\430f2638c1882d64e7673183c57b1564edaba69f\\bin\\cudnn_adv_train64_8.dll"
 "C:\\Users\\alexa\\.julia\\artifacts\\430f2638c1882d64e7673183c57b1564edaba69f\\bin\\cudnn64_8.dll"

[stillyslalom]

I downloaded cutensor.dll v11 from Nvidia, and the artifact is identical, so that's not the issue

julia> ct1 = read(raw"C:\Users\alexa\.julia\artifacts\804435c7dbe9d002d6caa0a8b3596d11a6faa778\bin\cutensor.dll");

julia> ct2 = read(raw"C:\Program Files\NVIDIA cuTENSOR\v1.2\lib\11.0\cutensor.dll");

julia> ct1 == ct2
true

[stillyslalom]

The issue's coming from CUDA110, which must be improperly linking to cutensor:

julia> artifact_hash("CUDA110", artifact_toml) |> verify_artifact
false

I think it may be related to JuliaLang/Pkg.jl#1892.

[timkimd]

I think I'm having similar issue. (I'm using Windows, and I can't load Flux or DiffEqFlux with Julia 1.4.2 or 1.5.0, but the error doesn't occur when using Julia 1.3.1.)

FluxML/Flux.jl#1313

[charleskawczynski]

I'm also using Windows, and I can't load CUDA with Julia 1.4.2 or 1.5.0, but I can with Julia 1.3.1.

[stillyslalom]

Which version of CUDA are you running? (check with nvidia-smi in the console). I'm on 11.0. It's unclear whether it's just a problem with the 11.0 DLLs, or if it's more general than that.

[charleskawczynski]

Hm, not sure how to use the nvidia-smi command:

bash: nvidia-smi: command not found

But,

julia> CUDA.version()
┌ Warning: `haskey(::TargetIterator, name::String)` is deprecated, use `Target(; name=name) !== nothing` instead.
│   caller = llvm_compat(::VersionNumber) at compatibility.jl:181
└ @ CUDA C:\Users\kawcz\.julia\packages\CUDA\42B9G\deps\compatibility.jl:181
v"10.2.0"
julia> versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

[stillyslalom]

You should be able to run nvidia-smi from one of the built-in Windows terminals (cmd or Powershell), but yeah, you're on CUDA v10.2. What's the error message you see on Julia 1.5?

[charleskawczynski]

I believe it was the same, or very similar to the issue post.