LinearAlgebra./ and LinearAlgebra.\ breaks CuArray

[algorithmx]

After using the operator / or \ on two CuArrays, some internal states oc CUDA seems to be broken.
The lines followng it causes strange error messages.
Removing the line using the operator / or \ will eliminate the error.

To reproduce

The Minimal Working Example (MWE) for this bug:

using LinearAlgebra
using CUDA

b = CUDA.rand(Float64,5,5)
A = CUDA.rand(Float64,5,5)
V = (A \ b)  # no command-line errors if this line is absent 

B = CUDA.rand(Float64,5,5)
w = CUDA.rand(Float64,5)
B * w

Manifest.toml

# This file is machine-generated - editing it directly is not advised

[[AbstractFFTs]]
deps = ["ChainRulesCore", "LinearAlgebra"]
git-tree-sha1 = "69f7020bd72f069c219b5e8c236c1fa90d2cb409"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.2.1"

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "af92965fb30777147966f58acb05da51c5616b5f"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.3.3"

[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[BFloat16s]]
deps = ["LinearAlgebra", "Printf", "Random", "Test"]
git-tree-sha1 = "a598ecb0d717092b5539dbbe890c98bac842b072"
uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b"
version = "0.2.0"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[CEnum]]
git-tree-sha1 = "eb4cb44a499229b3b8426dcfb5dd85333951ff90"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.2"

[[ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "2dd813e5f2f7eec2d1268c57cf2373d3ee91fcea"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.15.1"

[[ChangesOfVariables]]
deps = ["ChainRulesCore", "LinearAlgebra", "Test"]
git-tree-sha1 = "1e315e3f4b0b7ce40feded39c73049692126cf53"
uuid = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0"
version = "0.1.3"

[[Compat]]
deps = ["Dates", "LinearAlgebra", "UUIDs"]
git-tree-sha1 = "924cdca592bc16f14d2f7006754a621735280b74"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "4.1.0"

[[CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "b19534d1895d702889b219c382a6e18010797f0b"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.6"

[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[ExprTools]]
git-tree-sha1 = "56559bbef6ca5ea0c0818fa5c90320398a6fbf8d"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.8"

[[GPUArrays]]
deps = ["Adapt", "GPUArraysCore", "LLVM", "LinearAlgebra", "Printf", "Random", "Reexport", "Serialization", "Statistics"]
git-tree-sha1 = "470dcaf29237a0818bc2cc97f0c408f0bc052653"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "8.4.1"

[[GPUArraysCore]]
deps = ["Adapt"]
git-tree-sha1 = "4078d3557ab15dd9fe6a0cf6f65e3d4937e98427"
uuid = "46192b85-c4d5-4398-a991-12ede77f4527"
version = "0.1.0"

[[GPUCompiler]]
deps = ["ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "47f63159f7cb5d0e5e0cfd2f20454adea429bec9"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.16.1"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[InverseFunctions]]
deps = ["Test"]
git-tree-sha1 = "b3364212fb5d870f724876ffcd34dd8ec6d98918"
uuid = "3587e190-3f89-42d0-90ee-14403ec27112"
version = "0.1.7"

[[IrrationalConstants]]
git-tree-sha1 = "7fd44fd4ff43fc60815f8e764c0f352b83c49151"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.1.1"

[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "abc9885a7ca2052a736a600f7fa66209f96506e1"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.4.1"

[[LLVM]]
deps = ["CEnum", "LLVMExtra_jll", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "e7e9184b0bf0158ac4e4aa9daf00041b5909bf1a"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "4.14.0"

[[LLVMExtra_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg", "TOML"]
git-tree-sha1 = "771bfe376249626d3ca12bcd58ba243d3f961576"
uuid = "dad2f222-ce93-54a1-a47d-0025e8a3acab"
version = "0.0.16+0"

[[LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[LogExpFunctions]]
deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "09e4b894ce6a976c354a69041a04748180d43637"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.15"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[OpenLibm_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "05823500-19ac-5b8b-9628-191a04bc5112"

[[OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.5+0"

[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "47e5f437cc0e7ef2ce8406ce1e7e24d44915f88d"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.3.0"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[Random123]]
deps = ["Random", "RandomNumbers"]
git-tree-sha1 = "afeacaecf4ed1649555a19cb2cad3c141bbc9474"
uuid = "74087812-796a-5b5d-8853-05524746bad3"
version = "1.5.0"

[[RandomNumbers]]
deps = ["Random", "Requires"]
git-tree-sha1 = "043da614cc7e95c703498a491e2c21f58a2b8111"
uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143"
version = "1.5.3"

[[Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.3.0"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[SpecialFunctions]]
deps = ["ChainRulesCore", "IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
git-tree-sha1 = "d75bda01f8c31ebb72df80a46c88b25d1c79c56d"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "2.1.7"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[TimerOutputs]]
deps = ["ExprTools", "Printf"]
git-tree-sha1 = "464d64b2510a25e6efe410e7edab14fffdc333df"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.20"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

Expected behavior

Run the code in MWE from command line or Julia REPL will generate the following error message:

(base) dabajabaza@XXXX:~$ julia jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl 
ERROR: LoadError: CUBLASError: the requested feature is not supported (code 15, CUBLAS_STATUS_NOT_SUPPORTED)
Stacktrace:
 [1] throw_api_error(res::CUDA.CUBLAS.cublasStatus_t)
   @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/error.jl:50
 [2] macro expansion
   @ ~/.julia/packages/CUDA/DfvRa/lib/cublas/error.jl:63 [inlined]
 [3] cublasDgemv_v2(handle::Ptr{Nothing}, trans::Char, m::Int64, n::Int64, alpha::Bool, A::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, lda::Int64, x::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, incx::Int64, beta::Bool, y::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, incy::Int64)
   @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/utils/call.jl:26
 [4] gemv!
   @ ~/.julia/packages/CUDA/DfvRa/lib/cublas/wrappers.jl:331 [inlined]
 [5] gemv_dispatch!(Y::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, A::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, B::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, alpha::Bool, beta::Bool)
   @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/linalg.jl:179
 [6] mul!
   @ ~/.julia/packages/CUDA/DfvRa/lib/cublas/linalg.jl:188 [inlined]
 [7] mul!
   @ ~/julia/share/julia/stdlib/v1.7/LinearAlgebra/src/matmul.jl:275 [inlined]
 [8] *(A::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, x::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer})
   @ LinearAlgebra ~/julia/share/julia/stdlib/v1.7/LinearAlgebra/src/matmul.jl:47
 [9] top-level scope
   @ ~/jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl:9
in expression starting at /home/dabajabaza/jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl:9

Version info

Details on Julia:

Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_PKG_SERVER = https://mirrors.tuna.tsinghua.edu.cn/julia

Details on CUDA:

CUDA toolkit 11.7, artifact installation
NVIDIA driver 495.29.5, for CUDA 11.5
CUDA driver 11.5

Libraries: 
- CUBLAS: 11.10.1
- CURAND: 10.2.10
- CUFFT: 10.7.2
- CUSOLVER: 11.3.5
- CUSPARSE: 11.7.3
- CUPTI: 17.0.0
- NVML: 11.0.0+495.29.5
- CUDNN: 8.30.2 (for CUDA 11.5.0)
- CUTENSOR: 1.4.0 (for CUDA 11.5.0)

Toolchain:
- Julia: 1.7.2
- LLVM: 12.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

1 device:
  0: NVIDIA GeForce GTX 1060 (sm_61, 5.387 GiB / 5.936 GiB available)

[algorithmx]

maybe related to #1363

[algorithmx]

one more message: I don't see any errors when executing the code line by line (Shift+Enter) in VSCode + Julia Extension.

[maleadt]

maybe related to #1363

How would that be related?

I can't reproduce this. Please run with JULIA_DEBUG=CUBLAS.

[algorithmx]

I can't reproduce this. Please run with JULIA_DEBUG=CUBLAS.

(base) dabajabaza@XXXX:~$ julia  jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl
typeof(V) = CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}
ERROR: LoadError: ┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasCreate_v2(cublasContext**) called:
│   handle: type=cublasHandle_t; val=POINTER (IN HEX:0x0x7f531489c0e0)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
CUBLASError: ┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasSetStream_v2(cublasHandle_t, cudaStream_t) called:
│   handle: type=cublasHandle_t; val=POINTER (IN HEX:0x0x1c50f360)
│   streamId: type=SOME TYPE; val=POINTER (IN HEX:0x0x1b2ef30)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x0x1c50f360); StreamId=POINTER (IN HEX:0x(nil)) (defaultStream); MathMode=CUBLAS_DEFAULT_MATH
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
the requested feature is not supported┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=0
│   value: type=int; val=POINTER (IN HEX:0x0x7f5314de48a0)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
 (code 15, CUBLAS_STATUS_NOT_SUPPORTED)┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=1
│   value: type=int; val=POINTER (IN HEX:0x0x7f5314de48d0)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222

Stacktrace:┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=2
│   value: type=int; val=POINTER (IN HEX:0x0x7f5314de48e0)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222

┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=0
│   value: type=int; val=POINTER (IN HEX:0x0x7f5314de48f0)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
 ┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=1
│   value: type=int; val=POINTER (IN HEX:0x0x7f5314de4900)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
[1]┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=2
│   value: type=int; val=POINTER (IN HEX:0x0x7f5314de4920)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
 ┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasSetMathMode(cublasHandle_t, cublasMath_t) called:
│   handle: type=cublasHandle_t; val=POINTER (IN HEX:0x0x1c50f360)
│   mode: type=cublasMath_t; val=CUBLAS_DEFAULT_MATH | CUBLAS_MATH_DISALLOW_REDUCED_PRECISION_REDUCTION(16)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x0x1c50f360); StreamId=POINTER (IN HEX:0x0x1b2ef30); MathMode=CUBLAS_DEFAULT_MATH
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
throw_api_error┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasDgemv_v2(cublasHandle_t, cublasOperation_t, int, int, const double*, const double*, int, const double*, int, const double*, double*, int) called:
│   handle: type=cublasHandle_t; val=POINTER (IN HEX:0x0x1c50f360)
│   trans: type=cublasOperation_t; val=CUBLAS_OP_N(0)
│   m: type=int; val=5
│   n: type=int; val=5
│   alpha: type=double; val=POINTER (IN HEX:0x0x7ffd1bca3930)
│   A: type=double; val=POINTER (IN HEX:0x0x302000a00)
│   lda: type=int; val=5
│   x: type=double; val=POINTER (IN HEX:0x0x302000c00)
│   incx: type=int; val=1
│   beta: type=double; val=POINTER (IN HEX:0x0x7ffd1bca3990)
│   y: type=double; val=POINTER (IN HEX:0x0x302000e00)
│   incy: type=int; val=1
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x0x1c50f360); StreamId=POINTER (IN HEX:0x0x1b2ef30); MathMode=CUBLAS_DEFAULT_MATH | CUBLAS_MATH_DISALLOW_REDUCED_PRECISION_REDUCTION
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
│ 
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
(res::CUDA.CUBLAS.cublasStatus_t)
   @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/error.jl:50
 [2] macro expansion
   @ ~/.julia/packages/CUDA/DfvRa/lib/cublas/error.jl:63 [inlined]
 [3] cublasDgemv_v2(handle::Ptr{Nothing}, trans::Char, m::Int64, n::Int64, alpha::Bool, A::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, lda::Int64, x::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, incx::Int64, beta::Bool, y::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, incy::Int64)
   @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/utils/call.jl:26
 [4] gemv!
   @ ~/.julia/packages/CUDA/DfvRa/lib/cublas/wrappers.jl:331 [inlined]
 [5] gemv_dispatch!(Y::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, A::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, B::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer}, alpha::Bool, beta::Bool)
   @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/linalg.jl:179
 [6] mul!
   @ ~/.julia/packages/CUDA/DfvRa/lib/cublas/linalg.jl:188 [inlined]
 [7] mul!
   @ ~/julia/share/julia/stdlib/v1.7/LinearAlgebra/src/matmul.jl:275 [inlined]
 [8] *(A::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, x::CuArray{Float64, 1, CUDA.Mem.DeviceBuffer})
   @ LinearAlgebra ~/julia/share/julia/stdlib/v1.7/LinearAlgebra/src/matmul.jl:47
 [9] top-level scope
   @ ~/jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl:11
in expression starting at /home/dabajabaza/jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl:11
(base) dabajabaza@XXXX:~$ 
(base) dabajabaza@XXXX:~$ 
(base) dabajabaza@XXXX:~$ 
(base) dabajabaza@XXXX:~$ 
(base) dabajabaza@XXXX:~$ 
(base) dabajabaza@XXXX:~$ cat  jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl
ENV["JULIA_DEBUG"] = "CUBLAS"

using LinearAlgebra
using CUDA
b = CUDA.rand(Float64,5,5)
A = CUDA.rand(Float64,5,5)
V = (A \ b)
@show typeof(V)
B = CUDA.rand(Float64,5,5)
w = CUDA.rand(Float64,5)
B * w

[algorithmx]

I can't reproduce this. Please run with JULIA_DEBUG=CUBLAS.

(base) dabajabaza@XXXX:~$ julia  jianguoyun/Nutstore/RigorousCoupledWaveAnalysis.jl-master/examples/test.ma2018.jl
typeof(V) = CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}
ERROR: LoadError: ┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasCreate_v2(cublasContext**) called:
│   handle: type=cublasHandle_t; val=POINTER (IN HEX:0x0x7f531489c0e0)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
CUBLASError: ┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasSetStream_v2(cublasHandle_t, cudaStream_t) called:
│   handle: type=cublasHandle_t; val=POINTER (IN HEX:0x0x1c50f360)
│   streamId: type=SOME TYPE; val=POINTER (IN HEX:0x0x1b2ef30)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x0x1c50f360); StreamId=POINTER (IN HEX:0x(nil)) (defaultStream); MathMode=CUBLAS_DEFAULT_MATH
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
the requested feature is not supported┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=0
│   value: type=int; val=POINTER (IN HEX:0x0x7f5314de48a0)
│  Time: 2022-07-22T18:30:25 elapsed from start 0.000000 minutes or 0.000000 seconds
│ Process=130910; Thread=139996664834944; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
....

"the requested feature is not supported" <<== I don't understand which feature is not supported. Is it cublasSetStream_v2?

[algorithmx]

It is my fault. I have purged my CUDA installation and found that I had installed some incompatible versions of CUBLAS. That causes confusion and hence the errors. Sorry Tim, I have wasted your time...