Memory leaks with repeated use of `fft` of a CUDA Array

[roflmaostc]

Hey,

the following setup:

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen 5 5600X 6-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, generic)
Environment:
  JULIA_NUM_THREADS = 12

julia> CUDA.versioninfo()
CUDA toolkit 11.2.2, artifact installation
CUDA driver 11.2.0
NVIDIA driver 460.73.1

Libraries: 
- CUBLAS: 11.4.1
- CURAND: 10.2.3
- CUFFT: 10.4.1
- CUSOLVER: 11.1.0
- CUSPARSE: 11.4.1
- CUPTI: 14.0.0
- NVML: 11.0.0+460.73.1
- CUDNN: 8.10.0 (for CUDA 11.2.0)
- CUTENSOR: 1.2.2 (for CUDA 11.1.0)

Toolchain:
- Julia: 1.6.1
- LLVM: 11.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

1 device:
  0: GeForce RTX 2060 SUPER (sm_75, 7.183 GiB / 7.773 GiB available)

Manifest.toml

# This file is machine-generated - editing it directly is not advised

[[AbstractFFTs]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "485ee0867925449198280d4af84bdb46a2a404d0"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.0.1"

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "f1b523983a58802c4695851926203b36e28f09db"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.3.0"

[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[BFloat16s]]
deps = ["LinearAlgebra", "Test"]
git-tree-sha1 = "4af69e205efc343068dc8722b8dfec1ade89254a"
uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b"
version = "0.1.0"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[CEnum]]
git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.1"

[[CUDA]]
deps = ["AbstractFFTs", "Adapt", "BFloat16s", "CEnum", "CompilerSupportLibraries_jll", "DataStructures", "ExprTools", "GPUArrays", "GPUCompiler", "LLVM", "LazyArtifacts", "Libdl", "LinearAlgebra", "Logging", "MacroTools", "Memoize", "Printf", "Random", "RandomNumbers", "Reexport", "Requires", "SparseArrays", "SpecialFunctions", "Statistics", "TimerOutputs"]
git-tree-sha1 = "d4fa6486e94c4087f1d081d7be2d501a170bd51d"
uuid = "052768ef-5323-5732-b1bb-66c8b64840ba"
version = "3.1.0"

[[ChainRules]]
deps = ["ChainRulesCore", "Compat", "LinearAlgebra", "Random", "Reexport", "Requires", "Statistics"]
git-tree-sha1 = "729a17113c67d941a54169da6a81c5ec890f5834"
uuid = "082447d4-558c-5d27-93f4-14fc19e9eca2"
version = "0.7.63"

[[ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "bd0cc939d94b8bd736dce5bbbe0d635db9f94af7"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "0.9.41"

[[ColorTypes]]
deps = ["FixedPointNumbers", "Random"]
git-tree-sha1 = "024fe24d83e4a5bf5fc80501a314ce0d1aa35597"
uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
version = "0.11.0"

[[Colors]]
deps = ["ColorTypes", "FixedPointNumbers", "Reexport"]
git-tree-sha1 = "417b0ed7b8b838aa6ca0a87aadf1bb9eb111ce40"
uuid = "5ae59095-9a9b-59fe-a467-6f913c188581"
version = "0.12.8"

[[CommonSubexpressions]]
deps = ["MacroTools", "Test"]
git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7"
uuid = "bbf7d656-a473-5ed7-a52c-81e309532950"
version = "0.3.0"

[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "0a817fbe51c976de090aa8c997b7b719b786118d"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.28.0"

[[CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "4437b64df1e0adccc3e5d1adbc3ac741095e4677"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.9"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[DiffResults]]
deps = ["StaticArrays"]
git-tree-sha1 = "c18e98cba888c6c25d1c3b048e4b3380ca956805"
uuid = "163ba53b-c6d8-5494-b064-1a9d43ac40c5"
version = "1.0.3"

[[DiffRules]]
deps = ["NaNMath", "Random", "SpecialFunctions"]
git-tree-sha1 = "214c3fcac57755cfda163d91c58893a8723f93e9"
uuid = "b552c78f-8df3-52c6-915a-8e097449b14b"
version = "1.0.2"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[ExprTools]]
git-tree-sha1 = "10407a39b87f29d47ebaca8edbc75d7c302ff93e"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.3"

[[FFTW]]
deps = ["AbstractFFTs", "FFTW_jll", "LinearAlgebra", "MKL_jll", "Preferences", "Reexport"]
git-tree-sha1 = "746f68839306977040653ebbd249e39c15420b8a"
uuid = "7a1cc6ca-52ef-59f5-83cd-3a7055c09341"
version = "1.4.1"

[[FFTW_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "5a0d4b6a22a34d17d53543bd124f4b08ed78e8b0"
uuid = "f5851436-0d7a-5f13-b9de-f02708fd171a"
version = "3.3.9+7"

[[FillArrays]]
deps = ["LinearAlgebra", "Random", "SparseArrays"]
git-tree-sha1 = "31939159aeb8ffad1d4d8ee44d07f8558273120a"
uuid = "1a297f60-69ca-5386-bcde-b61e274b549b"
version = "0.11.7"

[[FixedPointNumbers]]
deps = ["Statistics"]
git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc"
uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
version = "0.8.4"

[[ForwardDiff]]
deps = ["CommonSubexpressions", "DiffResults", "DiffRules", "LinearAlgebra", "NaNMath", "Printf", "Random", "SpecialFunctions", "StaticArrays"]
git-tree-sha1 = "e2af66012e08966366a43251e1fd421522908be6"
uuid = "f6369f11-7733-5829-9624-2563aa707210"
version = "0.10.18"

[[GPUArrays]]
deps = ["AbstractFFTs", "Adapt", "LinearAlgebra", "Printf", "Random", "Serialization", "Statistics"]
git-tree-sha1 = "711b448634c8e1c3597eca3a72aeecf0d19c7d3b"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "6.4.0"

[[GPUCompiler]]
deps = ["DataStructures", "ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "Scratch", "Serialization", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "42d635f6d87af125b86288df3819f805fb4d851a"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.11.5"

[[Graphics]]
deps = ["Colors", "LinearAlgebra", "NaNMath"]
git-tree-sha1 = "2c1cf4df419938ece72de17f368a021ee162762e"
uuid = "a2bd30eb-e257-5431-a919-1863eab51364"
version = "1.1.0"

[[IRTools]]
deps = ["InteractiveUtils", "MacroTools", "Test"]
git-tree-sha1 = "c67e7515a11f726f44083e74f218d134396d6510"
uuid = "7869d1d1-7146-5819-86e3-90919afe41df"
version = "0.4.2"

[[IntelOpenMP_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "d979e54b71da82f3a65b62553da4fc3d18c9004c"
uuid = "1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0"
version = "2018.0.3+2"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"

[[LLVM]]
deps = ["CEnum", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "a220efe4a6bc1c71809d002eb9ed9209ce5a86fb"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "3.7.0"

[[LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[MKL_jll]]
deps = ["Artifacts", "IntelOpenMP_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"]
git-tree-sha1 = "c253236b0ed414624b083e6b72bfe891fbd2c7af"
uuid = "856f044c-d86e-5d09-b602-aeab76dc8ba7"
version = "2021.1.1+1"

[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "6a8a2a625ab0dea913aba95c11370589e0239ff0"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.6"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[Memoize]]
deps = ["MacroTools"]
git-tree-sha1 = "2b1dfcba103de714d31c033b5dacc2e4a12c7caa"
uuid = "c03570c3-d221-55d1-a50c-7939bbd78826"
version = "0.4.4"

[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[NFFT]]
deps = ["CUDA", "Distributed", "FFTW", "Graphics", "LinearAlgebra", "Random", "SparseArrays", "SpecialFunctions"]
git-tree-sha1 = "3c4e0300f7a6765df09d340ccf049acbd6db572d"
repo-rev = "master"
repo-url = "https://github.com/tknopp/NFFT.jl"
uuid = "efe261a4-0d2b-5849-be55-fc731d526b0d"
version = "0.6.1"

[[NaNMath]]
git-tree-sha1 = "bfe47e760d60b82b66b61d2d44128b62e3a369fb"
uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
version = "0.3.5"

[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b9b8b8ed236998f91143938a760c2112dceeb2b4"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.4+0"

[[OrderedCollections]]
git-tree-sha1 = "4fa2ba51070ec13fcc7517db714445b4ab986bdf"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.0"

[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "ea79e4c9077208cd3bc5d29631a26bc0cff78902"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.1"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[RandomNumbers]]
deps = ["Random", "Requires"]
git-tree-sha1 = "441e6fc35597524ada7f85e13df1f4e10137d16f"
uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143"
version = "1.4.0"

[[Reexport]]
git-tree-sha1 = "57d8440b0c7d98fc4f889e478e80f268d534c9d5"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.0.0"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.1.3"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Scratch]]
deps = ["Dates"]
git-tree-sha1 = "ad4b278adb62d185bbcb6864dc24959ab0627bf6"
uuid = "6c6a2e73-6563-6170-7368-637461726353"
version = "1.0.3"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[SpecialFunctions]]
deps = ["ChainRulesCore", "OpenSpecFun_jll"]
git-tree-sha1 = "5919936c0e92cff40e57d0ddf0ceb667d42e5902"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "1.3.0"

[[StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "2653e9c769343808781a8bd5010ee7a17c01152e"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.1.2"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[TimerOutputs]]
deps = ["Printf"]
git-tree-sha1 = "32cdbe6cd2d214c25a0b88f985c9e0092877c236"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.8"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[Zygote]]
deps = ["AbstractFFTs", "ChainRules", "ChainRulesCore", "DiffRules", "Distributed", "FillArrays", "ForwardDiff", "IRTools", "InteractiveUtils", "LinearAlgebra", "MacroTools", "NaNMath", "Random", "Requires", "SpecialFunctions", "Statistics", "ZygoteRules"]
path = "/home/fxw/.julia/dev/Zygote"
uuid = "e88e6eb3-aa80-5325-afca-941959d7151f"
version = "0.6.10"

[[ZygoteRules]]
deps = ["MacroTools"]
git-tree-sha1 = "9e7a1e8ca60b742e508a315c17eef5211e7fbfd7"
uuid = "700de1a5-db45-46bc-99cf-38207098b444"
version = "0.2.1"

[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

As discussed on Discourse, after several fft calls, memory doesn't get freed anymore:

julia> using CUDA, FFTW

julia> x = CUDA.rand(ComplexF32, (512, 512, 512)); # 1GiB memory

julia> CUDA.memory_status()
Effective GPU memory usage: 24.61% (1.913 GiB/7.773 GiB)
CUDA allocator usage: 1.000 GiB
Memory pool usage: 1.000 GiB (1.000 GiB allocated, 0 bytes cached)

julia> CUDA.@time y = fft(x);
  0.995551 seconds (2.68 M CPU allocations: 147.661 MiB, 2.32% gc time) (2 GPU allocations: 2.000 GiB, 4.73% gc time of which 9.55% spent allocating)

julia> CUDA.memory_status()
Effective GPU memory usage: 50.96% (3.962 GiB/7.773 GiB)
CUDA allocator usage: 2.000 GiB
Memory pool usage: 2.000 GiB (2.000 GiB allocated, 0 bytes cached)

julia> GC.gc(true)

julia> CUDA.memory_status()
Effective GPU memory usage: 63.83% (4.962 GiB/7.773 GiB)
CUDA allocator usage: 2.000 GiB
Memory pool usage: 2.000 GiB (2.000 GiB allocated, 0 bytes cached)

julia> CUDA.@time y = fft(x);
  0.027007 seconds (329.92 k CPU allocations: 5.035 MiB) (2 GPU allocations: 2.000 GiB, 0.05% gc time of which 61.26% spent allocating)

julia> CUDA.@time y = fft(x);
  0.083744 seconds (251.18 k CPU allocations: 3.833 MiB, 4.40% gc time) (2 GPU allocations: 2.000 GiB, 68.15% gc time of which 85.60% spent allocating)

julia> CUDA.@time y = fft(x);
  0.037591 seconds (373.56 k CPU allocations: 5.701 MiB, 7.31% gc time) (2 GPU allocations: 2.000 GiB, 18.91% gc time of which 0.13% spent allocating)

julia> CUDA.@time y = fft(x);
  0.037605 seconds (371.99 k CPU allocations: 5.677 MiB, 7.70% gc time) (2 GPU allocations: 2.000 GiB, 19.19% gc time of which 0.14% spent allocating)

julia> CUDA.@time y = fft(x);
  0.038422 seconds (365.12 k CPU allocations: 5.573 MiB, 8.85% gc time) (2 GPU allocations: 2.000 GiB, 20.55% gc time of which 0.16% spent allocating)

julia> CUDA.@time y = fft(x);
  0.038034 seconds (362.32 k CPU allocations: 5.529 MiB, 8.32% gc time) (2 GPU allocations: 2.000 GiB, 20.03% gc time of which 0.15% spent allocating)

julia> GC.gc(true)

julia> CUDA.memory_status()
Effective GPU memory usage: 76.69% (5.962 GiB/7.773 GiB)
CUDA allocator usage: 2.000 GiB
Memory pool usage: 2.000 GiB (2.000 GiB allocated, 0 bytes cached)

julia> CUDA.@time y = fft(x);
  0.027007 seconds (322.19 k CPU allocations: 4.917 MiB) (2 GPU allocations: 2.000 GiB, 0.05% gc time of which 66.57% spent allocating)

julia> CUDA.@time y = fft(x);
ERROR: CUFFTError: driver or internal cuFFT library error (code 5, CUFFT_INTERNAL_ERROR)
Stacktrace:
  [1] throw_api_error(res::CUDA.CUFFT.cufftResult_t)
    @ CUDA.CUFFT ~/.julia/packages/CUDA/k52QH/lib/cufft/error.jl:64
  [2] macro expansion
    @ ~/.julia/packages/CUDA/k52QH/lib/cufft/error.jl:81 [inlined]
  [3] cufftMakePlan3d(plan::Int32, nx::Int64, ny::Int64, nz::Int64, type::CUDA.CUFFT.cufftType_t, workSize::Base.RefValue{UInt64})
    @ CUDA.CUFFT ~/.julia/packages/CUDA/k52QH/lib/utils/call.jl:26
  [4] create_plan(xtype::CUDA.CUFFT.cufftType_t, xdims::Tuple{Int64, Int64, Int64}, region::UnitRange{Int64})
    @ CUDA.CUFFT ~/.julia/packages/CUDA/k52QH/lib/cufft/fft.jl:137
  [5] plan_fft
    @ ~/.julia/packages/CUDA/k52QH/lib/cufft/fft.jl:293 [inlined]
  [6] #plan_fft#10
    @ ~/.julia/packages/FFTW/Iu2GG/src/fft.jl:693 [inlined]
  [7] plan_fft
    @ ~/.julia/packages/FFTW/Iu2GG/src/fft.jl:693 [inlined]
  [8] fft(x::CuArray{ComplexF32, 3})
    @ AbstractFFTs ~/.julia/packages/AbstractFFTs/JebmH/src/definitions.jl:50
  [9] macro expansion
    @ ~/.julia/packages/CUDA/k52QH/src/utilities.jl:28 [inlined]
 [10] top-level scope
    @ ~/.julia/packages/CUDA/k52QH/src/pool.jl:572 [inlined]
 [11] top-level scope
    @ ./REPL[18]:0
 [12] top-level scope
    @ ~/.julia/packages/CUDA/k52QH/src/initialization.jl:81

Any more info I can provide?

Thanks,

Felix

[roflmaostc]

Turns out the issue is even more strange, even fft! leaks memory 😕

julia> using CUDA, FFTW

julia> x = CUDA.rand(ComplexF32, (512, 512, 512)); # 1GiB memory

julia> CUDA.memory_status()
Effective GPU memory usage: 23.86% (1.855 GiB/7.773 GiB)
CUDA allocator usage: 1.000 GiB
Memory pool usage: 1.000 GiB (1.000 GiB allocated, 0 bytes cached)

julia> CUDA.@time y = fft!(x);
  0.617843 seconds (2.19 M CPU allocations: 117.850 MiB, 4.14% gc time)

julia> CUDA.@time y = fft!(x);
  0.001098 seconds (24 CPU allocations: 688 bytes)

julia> CUDA.@time y = fft!(x);
  0.001115 seconds (24 CPU allocations: 688 bytes)

julia> CUDA.@time y = fft!(x);
  0.000762 seconds (24 CPU allocations: 688 bytes)

julia> CUDA.@time y = fft!(x);
  0.000774 seconds (24 CPU allocations: 688 bytes)

julia> CUDA.@time y = fft!(x);
  0.000770 seconds (24 CPU allocations: 688 bytes)

julia> CUDA.@time y = fft!(x);
  0.002400 seconds (24 CPU allocations: 688 bytes)

julia> CUDA.@time y = fft!(x);
ERROR: CUFFTError: driver or internal cuFFT library error (code 5, CUFFT_INTERNAL_ERROR)
Stacktrace:
  [1] throw_api_error(res::CUDA.CUFFT.cufftResult_t)
    @ CUDA.CUFFT ~/.julia/packages/CUDA/k52QH/lib/cufft/error.jl:64
  [2] macro expansion
    @ ~/.julia/packages/CUDA/k52QH/lib/cufft/error.jl:81 [inlined]
  [3] cufftMakePlan3d(plan::Int32, nx::Int64, ny::Int64, nz::Int64, type::CUDA.CUFFT.cufftType_t, workSize::Base.RefValue{UInt64})
    @ CUDA.CUFFT ~/.julia/packages/CUDA/k52QH/lib/utils/call.jl:26
  [4] create_plan(xtype::CUDA.CUFFT.cufftType_t, xdims::Tuple{Int64, Int64, Int64}, region::UnitRange{Int64})
    @ CUDA.CUFFT ~/.julia/packages/CUDA/k52QH/lib/cufft/fft.jl:137
  [5] plan_fft!
    @ ~/.julia/packages/CUDA/k52QH/lib/cufft/fft.jl:272 [inlined]
  [6] #plan_fft!#11
    @ ~/.julia/packages/FFTW/Iu2GG/src/fft.jl:695 [inlined]
  [7] plan_fft!
    @ ~/.julia/packages/FFTW/Iu2GG/src/fft.jl:695 [inlined]
  [8] fft!(x::CuArray{ComplexF32, 3})
    @ AbstractFFTs ~/.julia/packages/AbstractFFTs/JebmH/src/definitions.jl:50
  [9] macro expansion
    @ ~/.julia/packages/CUDA/k52QH/src/utilities.jl:28 [inlined]
 [10] top-level scope
    @ ~/.julia/packages/CUDA/k52QH/src/pool.jl:572 [inlined]
 [11] top-level scope
    @ ./REPL[5]:0
 [12] top-level scope
    @ ~/.julia/packages/CUDA/k52QH/src/initialization.jl:81

julia> CUDA.memory_status()
Effective GPU memory usage: 88.81% (6.904 GiB/7.773 GiB)
CUDA allocator usage: 1.000 GiB
Memory pool usage: 1.000 GiB (1.000 GiB allocated, 0 bytes cached)

[maleadt]

As I explained on Discourse (or was it Slack?), it's not a memory leak, so calling the in-place methods won't help. CUFFT needs its own work area, and allocating that probably goes wrong here. I'll try to have a look, maybe allocating our own workspace could help here.

[ali-ramadhan]

I assume you want to be using CUDA.CUFFT.plan_fft! to plan the FFTs then apply them later (which shouldn't allocate extra memory)?

[maleadt]

I could reproduce, and put up a fix in #902. Please verify.

That said, you will want to create a plan and execute that iteratively like @ali-ramadhan suggests.

[roflmaostc]

I'll check out it, thanks a lot for the quick help.

I'm aware of FFT plans but didn't know that they prevent the memory issue, I'll try that as well!

[roflmaostc]

Your fix seems to work.

However, old state:

julia> CUDA.@time fft(x);
  0.274786 seconds (676.42 k CPU allocations: 26.996 MiB, 66.27% gc time) (2 GPU allocations: 2.000 GiB, 103.69% gc time of which 1.01% spent allocating)

julia> CUDA.@time fft(x);
  0.224292 seconds (336.18 k CPU allocations: 5.130 MiB, 80.05% gc time) (2 GPU allocations: 2.000 GiB, 81.86% gc time of which 1.73% spent allocating)

julia> CUDA.@time fft(x);
  0.201762 seconds (336.40 k CPU allocations: 5.134 MiB, 86.41% gc time) (2 GPU allocations: 2.000 GiB, 86.76% gc time of which 0.01% spent allocating)

After fix:

julia> x = CUDA.rand(ComplexF32, (512, 512, 512));

julia> CUDA.@time fft(x);
  1.335657 seconds (3.01 M CPU allocations: 167.445 MiB, 24.74% gc time) (3 GPU allocations: 3.000 GiB, 26.55% gc time of which 2.04% spent allocating)

julia> CUDA.@time fft(x);
  0.353090 seconds (129.30 k CPU allocations: 7.722 MiB, 79.66% gc time) (3 GPU allocations: 3.000 GiB, 89.23% gc time of which 0.76% spent allocating)

julia> CUDA.@time fft(x);
  0.303626 seconds (12.03 k CPU allocations: 736.740 KiB, 87.67% gc time) (3 GPU allocations: 3.000 GiB, 90.42% gc time of which 1.00% spent allocating)

Initially it showed an allocated memory of 2GiB (in comparison FFTW does only allocate 1GiB). Why is the memory consumption twice as high?

After the fix it is worse with 3GiB per call.

[maleadt]

It needs 1GB workspace (which was previously allocated internally, so CUDA.@time didn't report it but the allocation was there), 1GB output, and 1GB workaround due to an NVIDIA bug, JuliaGPU/CuArrays.jl#345. Performance is identical here though, please make sure you measure accurately (e.g. wrapping in CUDA.@sync, using BenchmarkTools if possible).

[roflmaostc]

Hm, I'm still missing a explanation for one part:

For FFTW there is no significant allocation:

julia> x = randn(ComplexF32, (512, 512));

julia> varinfo(r"x")
  name      size summary                   
  –––– ––––––––– ––––––––––––––––––––––––––
  x    2.000 MiB 512×512 Matrix{ComplexF32}

julia> @btime fft!($x);
  8.301 ms (31 allocations: 2.44 KiB)

For CUFFT I still observe one allocation in the size of the array. This isn't explained by the NVIDIA bug you work around (was for out of place?), is it?

julia> x = CUDA.zeros(ComplexF32, 512, 512);

julia> CUDA.@time (CUDA.@sync fft!(x));
  0.000332 seconds (548 CPU allocations: 8.891 KiB) (1 GPU allocation: 2.000 MiB, 3.52% gc time of which 74.06% spent allocating)

julia> CUDA.@time (CUDA.@sync fft!(x));
  0.000339 seconds (514 CPU allocations: 8.359 KiB) (1 GPU allocation: 2.000 MiB, 4.36% gc time of which 70.57% spent allocating)

[maleadt]

It needs 1GB workspace (which was previously allocated internally, so CUDA.@time didn't report it but the allocation was there)

[roflmaostc]

But why is this workspace needed at all? Technically, the in-place version doesn't need additional memory?

EDIT:
CUFFT

x = CUDA.rand(ComplexF32, 512, 512,32);

julia> CUDA.@time p = plan_fft!(x);
  0.000216 seconds (44 CPU allocations: 1.062 KiB) (1 GPU allocation: 64.000 MiB, 5.79% gc time of which 73.70% spent allocating)

FFTW on Array:

julia> x = randn(ComplexF32, (512, 512,32));

julia> varinfo(r"x")
  name       size summary                        
  –––– –––––––––– –––––––––––––––––––––––––––––––
  x    64.000 MiB 512×512×32 Array{ComplexF32, 3}

julia> @time p = plan_fft!(x);
  0.000109 seconds (32 allocations: 2.703 KiB)

So apparently the plan creation of CUFFT (which of course is called by plan_fft) is the reason for the additional memory allocation.

[maleadt]

But why is this workspace needed at all? Technically, the in-place version doesn't need additional memory?

That's what CUFFT requires. File a bug with NVIDIA if you want that to change.

As has been said multiple times, if you want to reduce the cost of that, create a plan once and execute it multiple times. But even without doing that, the overhead of allocating the workspace is small, so instead of focusing on the output of CUDA.@time have a look at whether this even matters.

[roflmaostc]

Yeah I'm already using plans, it relaxes the situation.

And yes it actually did matter at several locations in my source code, (accidentally) I was creating ~100 plans (each was called hundred times again) and I was OOM again.
I'm now dragging the same plan through all functions, seems to be good now.

Thanks for the merge and your patience :)