LoadError: LoadError: UndefVarError: libcudnn not defined

[YolCruz]

I'm having the same problem as in #846 in Google colab when two weeks before It worked perfectly

[zenna]

Likewise.

I think the issue might be:

• CUDANative is failing (but is still being loaded). The recent update to CUDANative seems to be making people's CUDA fail.
• Since CUDANative fails, CuArrays also fails, but also in this weird weird way that allows it to still load but it doesn't load all the objects it should, like libcudnn
• Flux 0.9 uses CUDAapi.has_cuda() to see whether to do CUDA related stuff. This is still returning true even though CUDANative is failing to load, and hence you get the libcudnn error.

Flux#master has some more robust code for checking whether CUDA has loaded. But Flux#master relies on Zygote 3.4, which is broken due to another issue. Zygote 4.0 fixes that issue, but is incompatible with Flux#master.

[YolCruz]

So I have to wait for an update then?

[zenna]

I don't know. I don't work on Flux. And what I wrote may not actually be your error

[aussetg]

I don't think it's due to CUDAnative failing, CUDAnative passes all tests and I still have the error.

CuArrays also passes all tests including Libcudnn related tests.

[maleadt]

The problem is this TODO that nobody fixed:

Flux.jl/src/cuda/cuda.jl

Line 5 in 7104fd9

if CuArrays.libcudnn !== nothing # TODO: use CuArrays.has_cudnn()

That constant went away a while ago, but there was a backwards-compatible definition. That definition is now also gone due to the changes in how CuArrays is loaded.

For now, please downgrade to CuArrays 1.3, where this constant is still available. I've made a PR to retroactively add an upper bound to the current Flux release: JuliaRegistries/General#5066

CUDANative is failing (but is still being loaded). The recent update to CUDANative seems to be making people's CUDA fail.

No, CUDAnative should not fail to load. If it does, please file an issue.

[Sleort]

@maleadt The upper bound you added leads to Flux automatically being downgraded (while CuArrays is still being upgraded):

(v1.2) pkg> up
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating git-repo `https://github.com/FluxML/NNlib.jl.git`
 Resolving package versions...
 Installed Tracker ──── v0.2.4
 Installed CUDAdrv ──── v4.0.2
 Installed CUDAnative ─ v2.5.1
 Installed CuArrays ─── v1.4.1
  Updating `~/.julia/environments/v1.2/Project.toml`
  [c5f51814] ↑ CUDAdrv v4.0.1 ⇒ v4.0.2
  [be33ccc6] ↑ CUDAnative v2.5.0 ⇒ v2.5.1
  [3a865a2d] ↑ CuArrays v1.4.0 ⇒ v1.4.1
  [587475ba] ↓ Flux v0.9.0 ⇒ v0.8.3
  [9f7883ad] ↑ Tracker v0.2.3 ⇒ v0.2.4
  Updating `~/.julia/environments/v1.2/Manifest.toml`
  [c5f51814] ↑ CUDAdrv v4.0.1 ⇒ v4.0.2
  [be33ccc6] ↑ CUDAnative v2.5.0 ⇒ v2.5.1
  [3a865a2d] ↑ CuArrays v1.4.0 ⇒ v1.4.1
  [587475ba] ↓ Flux v0.9.0 ⇒ v0.8.3
  [9f7883ad] ↑ Tracker v0.2.3 ⇒ v0.2.4

hence, the problem persists (at least for someone who isn't aware of what's going on behind the scenes):

julia> using Flux, CuArrays
┌ Warning: Could not find libcutensor, CuArrays.CUTENSOR will be unavailable.
└ @ CuArrays ~/.julia/packages/CuArrays/ir1DU/src/CuArrays.jl:104
┌ Warning: Error requiring CuArrays from Flux:
│ LoadError: UndefVarError: libcudnn not defined
│ Stacktrace:
│  [1] getproperty(::Module, ::Symbol) at ./Base.jl:13
│  [2] top-level scope at /home/troels/.julia/packages/Flux/qXNjB/src/cuda/cuda.jl:26
│  [3] include at ./boot.jl:328 [inlined]
│  [4] include_relative(::Module, ::String) at ./loading.jl:1094
│  [5] include at ./Base.jl:31 [inlined]
│  [6] include(::String) at /home/troels/.julia/packages/Flux/qXNjB/src/Flux.jl:1
│  [7] top-level scope at REPL[1]:1
│  [8] eval at ./boot.jl:330 [inlined]
│  [9] eval at /home/troels/.julia/packages/Flux/qXNjB/src/Flux.jl:1 [inlined]
│  [10] (::getfield(Flux, Symbol("##150#154")))() at /home/troels/.julia/packages/Requires/9Jse8/src/require.jl:67
│  [11] err(::getfield(Flux, Symbol("##150#154")), ::Module, ::String) at /home/troels/.julia/packages/Requires/9Jse8/src/require.jl:38
│  [12] #149 at /home/troels/.julia/packages/Requires/9Jse8/src/require.jl:66 [inlined]
│  [13] withpath(::getfield(Flux, Symbol("##149#153")), ::String) at /home/troels/.julia/packages/Requires/9Jse8/src/require.jl:28
│  [14] (::getfield(Flux, Symbol("##148#152")))() at /home/troels/.julia/packages/Requires/9Jse8/src/require.jl:65
│  [15] #invokelatest#1 at ./essentials.jl:790 [inlined]
│  [16] invokelatest at ./essentials.jl:789 [inlined]
│  [17] #3 at /home/troels/.julia/packages/Requires/9Jse8/src/require.jl:19 [inlined]
│  [18] iterate at ./generator.jl:47 [inlined]
│  [19] collect_to!(::Array{Function,1}, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Int64, ::Int64) at ./array.jl:651
│  [20] collect_to!(::Array{typeof(CUDAnative.cudaconvert),1}, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Int64, ::Int64) at ./array.jl:659
│  [21] collect_to_with_first!(::Array{typeof(CUDAnative.cudaconvert),1}, ::Function, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Int64) at ./array.jl:630
│  [22] _collect(::Array{Function,1}, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:624
│  [23] map at ./array.jl:548 [inlined]
│  [24] loadpkg(::Base.PkgId) at /home/troels/.julia/packages/Requires/9Jse8/src/require.jl:19
│  [25] #invokelatest#1 at ./essentials.jl:790 [inlined]
│  [26] invokelatest at ./essentials.jl:789 [inlined]
│  [27] require(::Base.PkgId) at ./loading.jl:914
│  [28] require(::Module, ::Symbol) at ./loading.jl:906
│  [29] eval(::Module, ::Any) at ./boot.jl:330
│  [30] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:86
│  [31] run_backend(::REPL.REPLBackend) at /home/troels/.julia/packages/Revise/439di/src/Revise.jl:975
│  [32] (::getfield(Revise, Symbol("##80#82")){REPL.REPLBackend})() at ./task.jl:268
│ in expression starting at /home/troels/.julia/packages/Flux/qXNjB/src/cuda/cuda.jl:26
└ @ Requires ~/.julia/packages/Requires/9Jse8/src/require.jl:40

[Sleort]

Oh, and just for the record, downgrading to CuArrays 1.3 isn't straightforward either:

(v1.2) pkg> add CuArrays@1.3
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package CuArrays [3a865a2d]:
 CuArrays [3a865a2d] log:
 ├─possible versions are: [0.2.1, 0.3.0, 0.4.0, 0.5.0, 0.6.0-0.6.2, 0.7.0-0.7.3, 0.8.0-0.8.1, 0.9.0-0.9.1, 1.0.0-1.0.2, 1.1.0, 1.2.0-1.2.1, 1.3.0, 1.4.0-1.4.1] or uninstalled
 ├─restricted to versions 1.3 by an explicit requirement, leaving only versions 1.3.0
 └─restricted by compatibility requirements with CUDAdrv [c5f51814] to versions: 1.4.0-1.4.1 or uninstalled — no versions left
   └─CUDAdrv [c5f51814] log:
     ├─possible versions are: [0.8.0-0.8.6, 0.9.0, 1.0.0-1.0.1, 2.0.0, 3.0.0-3.0.1, 3.1.0, 4.0.0-4.0.2] or uninstalled
     └─restricted to versions 4.0.2 by an explicit requirement, leaving only versions 4.0.2

[dsweber2]

For what it's worth, a simple hack if you have dev'd Flux and know you have cudnn installed is to just remove the check. Obviously not a long term solution.

[maleadt]

Hm, I had hoped the resolver would react differently. In any case, it should prevent new users running into this, as IIUC it should prevent the upgrade in the first place.

[YolCruz]

I actually downgraded to 1.3 and works perfectly for me

[maleadt]

@Sleort: Try downgrading CUDAdrv too

(v1.2) pkg> st
    Status `/tmp/tmp.p0oy8HeJpN/environments/v1.2/Project.toml`
  [587475ba] Flux v0.9.0

# broken upgrade
(v1.2) pkg> up
  Updating `/tmp/tmp.p0oy8HeJpN/environments/v1.2/Manifest.toml`
  [c5f51814] ↑ CUDAdrv v3.1.0 ⇒ v4.0.2
  [be33ccc6] ↑ CUDAnative v2.4.0 ⇒ v2.5.1
  [3a865a2d] ↑ CuArrays v1.2.0 ⇒ v1.4.1

# failure to downgrade
(v1.2) pkg> add CuArrays@1.3
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package CuArrays [3a865a2d]:
 CuArrays [3a865a2d] log:
 ├─possible versions are: [0.2.1, 0.3.0, 0.4.0, 0.5.0, 0.6.0-0.6.2, 0.7.0-0.7.3, 0.8.0-0.8.1, 0.9.0-0.9.1, 1.0.0-1.0.2, 1.1.0, 1.2.0-1.2.1, 1.3.0, 1.4.0-1.4.1] or uninstalled
 ├─restricted to versions 1.3 by an explicit requirement, leaving only versions 1.3.0
 └─restricted by compatibility requirements with CUDAdrv [c5f51814] to versions: 1.4.0-1.4.1 or uninstalled — no versions left
   └─CUDAdrv [c5f51814] log:
     ├─possible versions are: [0.8.0-0.8.6, 0.9.0, 1.0.0-1.0.1, 2.0.0, 3.0.0-3.0.1, 3.1.0, 4.0.0-4.0.2] or uninstalled
     └─restricted to versions 4.0.2 by an explicit requirement, leaving only versions 4.0.2

# success
(v1.2) pkg> add CuArrays@1.3 CUDAdrv@3.1
 Resolving package versions...
  Updating `/tmp/tmp.p0oy8HeJpN/environments/v1.2/Project.toml`
  [c5f51814] ↓ CUDAdrv v4.0.2 ⇒ v3.1.0
  [3a865a2d] ↓ CuArrays v1.4.1 ⇒ v1.3.0
  Updating `/tmp/tmp.p0oy8HeJpN/environments/v1.2/Manifest.toml`
  [c5f51814] ↓ CUDAdrv v4.0.2 ⇒ v3.1.0
  [be33ccc6] ↓ CUDAnative v2.5.1 ⇒ v2.4.0
  [3a865a2d] ↓ CuArrays v1.4.1 ⇒ v1.3.0

[Sleort]

Yeah... I just figured out that CUDAnative was the one blocking the downgrade. Now I've got everything up and running again.

Thanks!

[maleadt]

With #916 (fixing master) and JuliaRegistries/General#5066 (fixing 0.9) I think we can close this.

[jchen975]

Could you please provide the versions for CUDAapi, CUDAnative and Flux that won't result in any errors too? Currently my laptop has Julia v1.1.1, and the following

  [3895d2a7] CUDAapi v2.0.0
  [c5f51814] CUDAdrv v4.0.4
  [be33ccc6] CUDAnative v2.5.5
  [3a865a2d] CuArrays v1.4.7
  [587475ba] Flux v0.8.3

And it works fine with CUDA 9.0, and cuDNN 7.1.4.

On the compute cluster which I don't have admin access, I used Julia 1.2.0 and the same versions for these 5 packages, but with CUDA 10.0.130 and cuDNN 7.6, and I keep getting this error message (both in Pkg.test("CuArrays") and during training, but Pkg.build() are successful for all):

Warning: Error requiring CuArrays from Flux:
│ LoadError: UndefVarError: libcudnn not defined

Any help would be appreciated!