Pkg.test error for cutensor test on Windows

[DanaGalizia]

I'm getting a failure of the cutensor test when I run Pkg.test("CUDA") after installing the package.

julia> Pkg.test("CUDA")

The output of this is then:

Status `C:\Users\danag\AppData\Local\Temp\jl_BYDm4E\Project.toml`
  [621f4979] AbstractFFTs v0.5.0
  [79e6a3ab] Adapt v2.0.2
  [b99e7846] BinaryProvider v0.5.10
  [fa961155] CEnum v0.4.1
  [052768ef] CUDA v1.3.3
  [864edb3b] DataStructures v0.18.4
  [e2ba6199] ExprTools v0.1.2
  [7a1cc6ca] FFTW v1.2.4
  [1a297f60] FillArrays v0.9.6
  [f6369f11] ForwardDiff v0.10.12
  [0c68f7d7] GPUArrays v5.1.0
  [61eb1bfa] GPUCompiler v0.6.1
  [929cbde3] LLVM v2.0.0
  [1914dd2f] MacroTools v0.5.5
  [872c559c] NNlib v0.7.4
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v1.0.2
  [a759f4b9] TimerOutputs v0.5.6
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [8dfed614] Test
Status `C:\Users\danag\AppData\Local\Temp\jl_BYDm4E\Manifest.toml`
  [621f4979] AbstractFFTs v0.5.0
  [79e6a3ab] Adapt v2.0.2
  [b99e7846] BinaryProvider v0.5.10
  [fa961155] CEnum v0.4.1
  [052768ef] CUDA v1.3.3
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v3.15.0
  [e66e0078] CompilerSupportLibraries_jll v0.3.3+0
  [864edb3b] DataStructures v0.18.4
  [163ba53b] DiffResults v1.0.2
  [b552c78f] DiffRules v1.0.1
  [e2ba6199] ExprTools v0.1.2
  [7a1cc6ca] FFTW v1.2.4
  [f5851436] FFTW_jll v3.3.9+5
  [1a297f60] FillArrays v0.9.6
  [f6369f11] ForwardDiff v0.10.12
  [0c68f7d7] GPUArrays v5.1.0
  [61eb1bfa] GPUCompiler v0.6.1
  [1d5cc7b8] IntelOpenMP_jll v2018.0.3+0
  [929cbde3] LLVM v2.0.0
  [856f044c] MKL_jll v2020.2.254+0
  [1914dd2f] MacroTools v0.5.5
  [872c559c] NNlib v0.7.4
  [77ba4419] NaNMath v0.3.4
  [efe28fd5] OpenSpecFun_jll v0.5.3+3
  [bac558e1] OrderedCollections v1.3.0
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v1.0.2
  [276daf66] SpecialFunctions v0.10.3
  [90137ffa] StaticArrays v0.12.4
  [a759f4b9] TimerOutputs v0.5.6
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [b77e0a4c] InteractiveUtils
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
Info: System information:
│ CUDA toolkit 11.0.3, artifact installation
│ CUDA driver 11.0.0
│ NVIDIA driver 452.6.0
│
│ Libraries:
│ - CUBLAS: 11.2.0
│ - CURAND: 10.2.1
│ - CUFFT: 10.2.1
│ - CUSOLVER: 10.6.0
│ - CUSPARSE: 11.1.1
│ - CUPTI: 13.0.0
│ - NVML: 11.0.0+452.6
│ - CUDNN: 8.0.2 (for CUDA 11.0.0)
│ - CUTENSOR: 1.2.0 (for CUDA 11.0.0)
│
│ Toolchain:
│ - Julia: 1.5.1
│ - LLVM: 9.0.1
│ - PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4
│ - Device support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75
│
│ Environment:
│ - JULIA_CUDA_USE_BINARYBUILDER: true
│
│ 1 device:
└   0: GeForce RTX 2070 SUPER (sm_75, 6.941 GiB / 8.000 GiB available)
[ Info: Testing using 1 device(s): 1. GeForce RTX 2070 SUPER (UUID 7d5caace-9c21-a91e-a376-a3f91b42a13f)
                                     |          | ---------------- GPU ---------------- | ---------------- CPU ---------------- |
Test                        (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
initialization                   (2) |     3.04 |   0.00 |  0.0 |       0.00 |      N/A |   0.04 |  1.5 |     221.59 |   503.67 |
cutensor                         (2) |         failed at 2020-09-11T14:50:02.127
      From worker 2:    error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)
      From worker 2:    error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)
      From worker 2:    error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)
      From worker 2:    error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)
apiutils                         (3) |     0.43 |   0.00 |  0.0 |       0.00 |      N/A |   0.02 |  4.3 |      81.68 |   498.77 |

You can see the first signs of the error there. After this, there's a bunch of other tests that run fine. Then at the end of the test, there are hundreds (to the point where I can't even see them all in the REPL window) of error reports that seem to be mostly identical, a typical one which looks like:

Error in testset cutensor:
Error During Test at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\cutensor.jl:224
  Got exception outside of a @test
  CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)
  Stacktrace:
   [1] throw_api_error(::CUDA.cudaError_enum) at C:\Users\danag\.julia\packages\CUDA\dZvbp\lib\cudadrv\error.jl:103
   [2] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\lib\cudadrv\error.jl:110 [inlined]
   [3] cuMemAlloc_v2(::Base.RefValue{CuPtr{Nothing}}, ::Int64) at C:\Users\danag\.julia\packages\CUDA\dZvbp\lib\utils\call.jl:93
   [4] alloc at C:\Users\danag\.julia\packages\CUDA\dZvbp\lib\cudadrv\memory.jl:69 [inlined]
   [5] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool.jl:181 [inlined]
   [6] macro expansion at C:\Users\danag\.julia\packages\TimerOutputs\dVnaw\src\TimerOutput.jl:206 [inlined]
   [7] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool.jl:180 [inlined]
   [8] macro expansion at .\timing.jl:233 [inlined]
   [9] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool.jl:179 [inlined]
   [10] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\state.jl:206 [inlined]
   [11] actual_alloc(::CuDevice, ::Int64) at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool.jl:171
   [12] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool\binned.jl:143 [inlined]
   [13] macro expansion at C:\Users\danag\.julia\packages\TimerOutputs\dVnaw\src\TimerOutput.jl:206 [inlined]
   [14] pool_alloc(::CuDevice, ::Int64) at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool\binned.jl:142
   [15] macro expansion at C:\Users\danag\.julia\packages\TimerOutputs\dVnaw\src\TimerOutput.jl:206 [inlined]
   [16] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool.jl:296 [inlined]
   [17] macro expansion at .\timing.jl:233 [inlined]
   [18] alloc at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool.jl:295 [inlined]
   [19] CuArray{Complex{Float64},5}(::UndefInitializer, ::NTuple{5,Int64}) at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\array.jl:116
   [20] CuArray at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\array.jl:205 [inlined]
   [21] CuArray(::Array{Complex{Float64},5}) at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\array.jl:213
   [22] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\cutensor.jl:435 [inlined]
   [23] macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Test\src\Test.jl:1190 [inlined]
   [24] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\cutensor.jl:424 [inlined]
   [25] macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Test\src\Test.jl:1190 [inlined]
   [26] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\cutensor.jl:423 [inlined]
   [27] macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Test\src\Test.jl:1115 [inlined]
   [28] top-level scope at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\cutensor.jl:417
   [29] include(::String) at .\client.jl:457
   [30] #9 at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\runtests.jl:79 [inlined]
   [31] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\setup.jl:44 [inlined]
   [32] macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Test\src\Test.jl:1115 [inlined]
   [33] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\setup.jl:44 [inlined]
   [34] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\utilities.jl:35 [inlined]
   [35] macro expansion at C:\Users\danag\.julia\packages\CUDA\dZvbp\src\pool.jl:537 [inlined]
   [36] top-level scope at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\setup.jl:43
   [37] eval at .\boot.jl:331 [inlined]
   [38] runtests(::Function, ::String, ::Symbol, ::Nothing) at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\setup.jl:55
   [39] (::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}})() at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:294
   [40] run_work_thunk(::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}}, ::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:79
   [41] macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:294 [inlined]
   [42] (::Distributed.var"#105#107"{Distributed.CallMsg{:call_fetch},Distributed.MsgHeader,Sockets.TCPSocket})() at .\task.jl:356

The only differences I can see are that, in that second line, sometimes it's cutensor.jl:424 or cutensor.jl:511 instead of 224. Finally, the output ends with

ERROR: LoadError: Test run finished with errors
in expression starting at C:\Users\danag\.julia\packages\CUDA\dZvbp\test\runtests.jl:482
ERROR: Package CUDA errored during testing
Stacktrace:
 [1] pkgerror(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:52
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1578
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:328
 [4] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:315
 [5] #test#61 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
 [6] test at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
 [7] #test#60 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
 [8] test at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
 [9] test(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
 [10] test(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
 [11] top-level scope at REPL[11]:1

Manifest.toml

# This file is machine-generated - editing it directly is not advised

[[AbstractFFTs]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "051c95d6836228d120f5f4b984dd5aba1624f716"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "0.5.0"

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "0fac443759fa829ed8066db6cf1077d888bb6573"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "2.0.2"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[BinaryProvider]]
deps = ["Libdl", "Logging", "SHA"]
git-tree-sha1 = "ecdec412a9abc8db54c0efc5548c64dfce072058"
uuid = "b99e7846-7c00-51b0-8f62-c81ae34c0232"
version = "0.5.10"

[[CEnum]]
git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.1"

[[DataStructures]]
deps = ["InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "88d48e133e6d3dd68183309877eac74393daa7eb"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.17.20"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[ExprTools]]
git-tree-sha1 = "6f0517056812fd6aa3af23d4b70d5325a2ae4e95"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.1"

[[GPUArrays]]
deps = ["AbstractFFTs", "Adapt", "LinearAlgebra", "Printf", "Random", "Serialization"]
git-tree-sha1 = "600f45500060894487832c2f00c203fe3e0cb264"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "5.1.0"

[[GPUCompiler]]
deps = ["DataStructures", "InteractiveUtils", "LLVM", "Libdl", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "c5687a4d9ff9fcfce8700b98106d63334dcbb4fe"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.6.0"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[LLVM]]
deps = ["CEnum", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "a662366a5d485dee882077e8da3e1a95a86d097f"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "2.0.0"

[[LibGit2]]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "f7d2e3f654af75f01ec49be82c231c382214223a"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.5"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[NNlib]]
deps = ["Libdl", "LinearAlgebra", "Pkg", "Requires", "Statistics"]
git-tree-sha1 = "8ec4693a5422f0b064ce324f59351f24aa474893"
uuid = "872c559c-99b0-510c-b3b7-b6c96a88d5cd"
version = "0.7.4"

[[OrderedCollections]]
git-tree-sha1 = "293b70ac1780f9584c89268a6e2a560d938a7065"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.3.0"

[[Pkg]]
deps = ["Dates", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Test", "UUIDs"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[Reexport]]
deps = ["Pkg"]
git-tree-sha1 = "7b1d07f411bc8ddb7977ec7f377b97b158514fe0"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "0.2.0"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "d37400976e98018ee840e0ca4f9d20baa231dc6b"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.0.1"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[Test]]
deps = ["Distributed", "InteractiveUtils", "Logging", "Random"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[TimerOutputs]]
deps = ["Printf"]
git-tree-sha1 = "f458ca23ff80e46a630922c555d838303e4b9603"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.6"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

Version info

Details on Julia:

julia> versioninfo()
Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 9 3900X 12-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, znver2)
Environment:
  JULIA_CUDA_USE_BINARYBUILDER = true

Details on CUDA:

julia> CUDA.versioninfo()
CUDA toolkit 11.0.3, artifact installation
CUDA driver 11.0.0
NVIDIA driver 452.6.0

Libraries:
- CUBLAS: 11.2.0
- CURAND: 10.2.1
- CUFFT: 10.2.1
- CUSOLVER: 10.6.0
- CUSPARSE: 11.1.1
- CUPTI: 13.0.0
- NVML: 11.0.0+452.6
- CUDNN: 8.0.2 (for CUDA 11.0.0)
- CUTENSOR: 1.2.0 (for CUDA 11.0.0)

Toolchain:
- Julia: 1.5.1
- LLVM: 9.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4
- Device support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75

1 device:
  0: GeForce RTX 2070 SUPER (sm_75, 6.963 GiB / 8.000 GiB available)

(EDIT: CUDA version info was pasted from the wrong place originally. Fixed now.)

[maleadt]

CUTENSOR has been upgraded to 1.2.1, could you test again on the latest master branch?