CuArrays don't seem to display correctly in VS code

[DrChainsaw]

Describe the bug

Seems like CuArrays don't display correctly in the VS code REPL on windows, although the values seem to be ok. Don't have a linux install to test on. Also seems to work fine when running julia in windows terminal.

To reproduce

The Minimal Working Example (MWE) for this bug:

julia> xx = randn(3)
3-element Vector{Float64}:
 -0.2691051505932
 -0.38601362073983303
 -0.16102191893287385

julia> cu(xx)
3-element CuArray{Float32, 1}:
 0.0
 0.0
 0.0

julia> cu(xx) |> Array
3-element Vector{Float32}:
 -0.26910514
 -0.38601363
 -0.16102192

julia> cu(xx) .+ 10 |> Array
3-element Vector{Float32}:
 9.730895
 9.613986
 9.838978

# These seem to work, VS code uses some special MIME or something?
julia> display(cu(xx))
3-element CuArray{Float32, 1}:
 -0.26910514
 -0.38601363
 -0.16102192

julia> show(cu(xx))
Float32[-0.26910514, -0.38601363, -0.16102192]

# This is a bit puzzling...
julia> @show cu(xx)
cu(xx) = Float32[-0.26910514, -0.38601363, -0.16102192]
3-element CuArray{Float32, 1}:
 -0.26910514
 -0.38601363
 -0.16102192

# These are not 100% reproducible. Threading issue?
julia> cu(xx) |> identity
3-element CuArray{Float32, 1}:
 -0.26910514
 -0.38601363
 -0.16102192

julia> cu(xx) .+ 10
3-element CuArray{Float32, 1}:
 -0.26910514
 -0.38601363
 -0.16102192

Manifest.toml

# This file is machine-generated - editing it directly is not advised

[[AbstractFFTs]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "485ee0867925449198280d4af84bdb46a2a404d0"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.0.1"

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "f1b523983a58802c4695851926203b36e28f09db"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.3.0"

[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[BFloat16s]]
deps = ["LinearAlgebra", "Test"]
git-tree-sha1 = "4af69e205efc343068dc8722b8dfec1ade89254a"
uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b"
version = "0.1.0"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[CEnum]]
git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.1"

[[CUDA]]
deps = ["AbstractFFTs", "Adapt", "BFloat16s", "CEnum", "CompilerSupportLibraries_jll", "DataStructures", "ExprTools", "GPUArrays", "GPUCompiler", "LLVM", "LazyArtifacts", "Libdl", "LinearAlgebra", "Logging", "MacroTools", "Memoize", "Printf", "Random", "RandomNumbers", "Reexport", "Requires", "SparseArrays", "SpecialFunctions", "Statistics", "TimerOutputs"]
git-tree-sha1 = "d4fa6486e94c4087f1d081d7be2d501a170bd51d"
uuid = "052768ef-5323-5732-b1bb-66c8b64840ba"
version = "3.1.0"

[[ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "a66109c73612c63b10923ac446fddb0f0d21a593"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "0.9.40"

[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "ac4132ad78082518ec2037ae5770b6e796f7f956"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.27.0"

[[CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "4437b64df1e0adccc3e5d1adbc3ac741095e4677"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.9"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[ExprTools]]
git-tree-sha1 = "10407a39b87f29d47ebaca8edbc75d7c302ff93e"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.3"

[[GPUArrays]]
deps = ["AbstractFFTs", "Adapt", "LinearAlgebra", "Printf", "Random", "Serialization"]
git-tree-sha1 = "9c95b2fd5c16bc7f97371e9f92f0fef77e0f5957"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "6.2.2"

[[GPUCompiler]]
deps = ["DataStructures", "ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "Scratch", "Serialization", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "6eadd2321dc3ac0fc9d530ab01c2caa7fe5d74c6"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.11.4"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"

[[LLVM]]
deps = ["CEnum", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "b616937c31337576360cb9fb872ec7633af7b194"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "3.6.0"

[[LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "6a8a2a625ab0dea913aba95c11370589e0239ff0"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.6"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[Memoize]]
deps = ["MacroTools"]
git-tree-sha1 = "2b1dfcba103de714d31c033b5dacc2e4a12c7caa"
uuid = "c03570c3-d221-55d1-a50c-7939bbd78826"
version = "0.4.4"

[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b9b8b8ed236998f91143938a760c2112dceeb2b4"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.4+0"

[[OrderedCollections]]
git-tree-sha1 = "4fa2ba51070ec13fcc7517db714445b4ab986bdf"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.0"

[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "ea79e4c9077208cd3bc5d29631a26bc0cff78902"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.1"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[RandomNumbers]]
deps = ["Random", "Requires"]
git-tree-sha1 = "441e6fc35597524ada7f85e13df1f4e10137d16f"
uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143"
version = "1.4.0"

[[Reexport]]
git-tree-sha1 = "57d8440b0c7d98fc4f889e478e80f268d534c9d5"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.0.0"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.1.3"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Scratch]]
deps = ["Dates"]
git-tree-sha1 = "ad4b278adb62d185bbcb6864dc24959ab0627bf6"
uuid = "6c6a2e73-6563-6170-7368-637461726353"
version = "1.0.3"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[SpecialFunctions]]
deps = ["ChainRulesCore", "OpenSpecFun_jll"]
git-tree-sha1 = "5919936c0e92cff40e57d0ddf0ceb667d42e5902"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "1.3.0"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[TimerOutputs]]
deps = ["Printf"]
git-tree-sha1 = "32cdbe6cd2d214c25a0b88f985c9e0092877c236"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.8"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

Expected behavior

Values should be represented correctly.

Version info

Details on Julia:

# please post the output of:
julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, haswell)
Environment:
  JULIA_DEPOT_PATH = E:/Programs/julia/.julia
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 12

Details on CUDA:

# please post the output of:
julia> CUDA.version()
v"11.3.0"

Additional context

Add any other context about the problem here.

[zhangcx93]

i cannot reproduce this, same vs code on windows with cuda.jl

[maleadt]

Probably similar to #831 / #837? Does VS Code evaluate stuff in a different task? Is there an API to influence that?

[DrChainsaw]

i cannot reproduce this, same vs code on windows with cuda.jl

Interesting. Its 100% reproduceable for me. I tried setting number of threads to 1 and its still 100%. Any way it could have to do with card type or driver version?

Probably similar to #831 / #837?

Looks like more or less exactly the same symptom. I also sometimes see garbage values, especially when there are more values.

Does this shed any more light?

julia> function fff(x)
       cx = cu(x)
       1+2
       return cx
       end
fff (generic function with 1 method)

julia> fff(xx)
3-element CuArray{Float32, 1}:
 0.0
 0.0
 0.0

julia> fff(xx)
3-element CuArray{Float32, 1}:
 0.0
 0.0
 0.0

julia> function fff(x)
       cx = cu(x)
       CUDA.device_synchronize()
       return cx
       end
fff (generic function with 1 method)

julia> fff(xx)
3-element CuArray{Float32, 1}:
  1.190589
 -0.6828699
  0.05489877

julia> fff(xx)
3-element CuArray{Float32, 1}:
  1.190589
 -0.6828699
  0.05489877

julia> xx = randn(5, 10)
5×10 Matrix{Float64}:
  1.67942    0.8585     1.27553    0.371574  -1.39436   -0.374869  -0.0694932  -1.40259    1.60282   -0.928131
  0.873455  -0.284457   0.968319   1.06297   -0.877972  -2.09631   -1.63726    -0.652214   1.78131    2.5976
  1.94723    0.762199   1.16164   -0.36343   -0.827762  -1.28091   -0.777818   -0.496266  -0.663625  -1.69308
 -0.668761   1.34078    0.295551   2.05289    1.75167    1.87313   -0.591423    1.1398    -0.638453   0.685368
 -0.44666    0.954526  -0.774744   0.114682  -0.37361    0.251102  -0.446602    1.09315   -0.642535  -0.254417

julia> cu(xx) # Now old values show up here!
5×10 CuArray{Float32, 2}:
  1.19059    0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 -0.68287    0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
  0.0548988  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
  0.0        0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
  0.0        0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0

julia> CUDA.device_synchronize()

julia> cu(xx)
5×10 CuArray{Float32, 2}:
  1.19059    0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 -0.68287    0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
  0.0548988  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
  0.0        0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
  0.0        0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0

Does VS Code evaluate stuff in a different task? Is there an API to influence that?

I don't know much about VS Code, but I think I have read somewhere in some issue that there is task-y stuff going on. @pfitzseb or @davidanthoff?

[pfitzseb]

Does VS Code evaluate stuff in a different task? Is there an API to influence that?

Yes and no. We should however call the REPL ast transforms properly (although maybe only on master).

[pfitzseb]

So is there a reason why show doesn't synchronize the array? Relying on the REPL hooks seems brittle, no?

[maleadt]

Synchronization is automatic: show copies memory from the device to the host, which synchronizes the task-local stream. It'd seem weird to me if show needs to be special cased to synchronize all streams.

Why does the REPL/vscode want to display on another task? We use task-local storage for many things, including device selection, library handles, etc. I can imagine other scenarios where using different tasks to compute and show could even lead to to crashes.

[pfitzseb]

It's not strictly necessary for us to do it like that. Currently we're evaluating user code in a separate task that's created purely for that purpose (so we can schedule interrupts on it). We could conceivably call display/show on the object there too, but that might be a bit annoying to implement.

Also, FWIW, I still can't repro this issue, even with CUDA.jl@3.1.

[maleadt]

I couldn't repro the REPL-related issue on Linux either, but could on Windows (it's timing sensitive, obviously).

[maleadt]

Why does the REPL/vscode want to display on another task? We use task-local storage for many things, including device selection, library handles, etc. I can imagine other scenarios where using different tasks to compute and show could even lead to to crashes.

And indeed, by @mcabbott:

julia> CUDA.device!(1)
CuContext(0x0000561874835820, instance b053160cb92aac29)

julia> sum(cu([1 2; 3 4.0]) .+ 1)
14.0f0

julia> cu([1 2; 3 4.0]) .+ 1
2×2 CuArray{Float32, 2}:
Error showing value of type CuArray{Float32, 2}:
ERROR: Out of GPU memory
Effective GPU memory usage: 99.99% (31.745 GiB/31.749 GiB)

Here device 0 doesn't have any free memory, leading to the OOM.

[pfitzseb]

Ok, so in theory we can fix this by calling display in the same task that evaluates the user code, but

• this breaks ans
• I don't see why it's necessary, considering the synchronize call in ast_transforms.

[maleadt]

You can switch devices, and that's a task local property too. So the returned array can contain a pointer from another device, and fail to show regardless of synchronization. Now, I'm hoping to add automatic cross-device transfers in the future (where data would know which device it is bound to), but the point remains that a package may assume task-local state to remain the same between evaluation and display.

Why would this break ans?

[pfitzseb]

Right, but the normal REPL already uses two different tasks for display and evaluation, so I'm not sure how that's different? We simply add yet another task into the mix.

Why would this break ans?

Basically, VSCode uses an AST transform to evaluate code in a different Task. If we also use that task to call display, then we need to return nothing so results aren't displayed twice. And that then sets ans to nothing, regardless of the actual value and after our injected code has returned.