Conda Pytorch (Pytorch channel) in WSL2 Ubuntu can't find libcudnn shared objects

[mawright]

🐛 Describe the bug

This is on a fresh install of Ubuntu 22,04 in WSL. All I did before running this was an apt-get upgrade and wget and install the Linux Miniconda installer. I did not download any Cuda packages from Nvidia, or the Ubuntu nvidia-cuda-toolkit package.

conda create -n torch pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

(skip install text)

conda activate torch
python test.py

test.py:

import torch

print(f"{torch.cuda.is_available()=}")

f = torch.nn.Conv2d(3, 8, 3, device="cuda")
X = torch.randn(2, 3, 4, 4, device="cuda")

Y = X @ X
print(f"{Y.shape=}")
print("matrix multiply works")

Y = f(X)
print(f"{Y.shape=}")
print("Conv2d works")

Console output;

torch.cuda.is_available()=True
Y.shape=torch.Size([2, 3, 4, 4])
matrix multiply works
Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory
Please make sure libcudnn_cnn_infer.so.8 is in your library path!
Aborted

The shared objects are where I would expect them to be:

$ ls $CONDA_PREFIX/lib/python3.10/site-packages/torch/lib
libc10_cuda.so           libcudnn_adv_train.so.8  libcudnn_ops_train.so.8    libtorch_cpu.so          libtorch_cuda.so
libc10.so                libcudnn_cnn_infer.so.8  libcudnn.so.8              libtorch_cuda_cpp.so     libtorch_global_deps.so
libcaffe2_nvrtc.so       libcudnn_cnn_train.so.8  libcupti-53b4cc5d.so.11.3  libtorch_cuda_cu.so      libtorch_python.so
libcudnn_adv_infer.so.8  libcudnn_ops_infer.so.8  libshm.so                  libtorch_cuda_linalg.so  libtorch.so

This error does not happen when running Pytorch on the Windows that's hosting this WSL container. On Windows, everything works fine.

This is a different bug from #73487. . They report Pytorch not finding Cuda, while what I've found is Pytorch crashing on a call to, e.g., a forward pass of a Conv2d layer with an error saying it's unable to load the libcudnn shared object.

Versions

$ python collect_env.py
Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 516.94
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.13.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0           py310h7f8727e_0
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0
[conda] mkl_random                1.2.2           py310h00e6091_0
[conda] numpy                     1.23.1          py310h1794996_0
[conda] numpy-base                1.23.1          py310hcba007f_0
[conda] pytorch                   1.12.1          py3.10_cuda11.3_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.1              py310_cu113    pytorch
[conda] torchvision               0.13.1              py310_cu113    pytorch

cc @malfet @seemethere @ngimel

[mawright]

Here's the collect_env.py output of a conda env on the Windows host, if that's relevant at all. The test.py script runs fine on Windows.

PS C:\Users\matt> python .\collect_env.py
Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.4 | packaged by conda-forge | (main, Mar 30 2022, 08:38:02) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22000-SP0
Is CUDA available: True
CUDA runtime version: 11.7.99
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 516.94
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] pytorch-lightning==1.7.6
[pip3] torch==1.12.1
[pip3] torchdiffeq==0.2.3
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h59b6b97_2
[conda] mkl                       2021.4.0           haa95532_640
[conda] mkl-service               2.4.0           py310h2bbff1b_0
[conda] mkl_fft                   1.3.1           py310ha0764ea_0
[conda] mkl_random                1.2.2           py310h4ed8f06_0
[conda] numpy                     1.21.5          py310h6d2d95c_3
[conda] numpy-base                1.21.5          py310h206c741_3
[conda] pytorch                   1.12.1          py3.10_cuda11.3_cudnn8_0    pytorch
[conda] pytorch-lightning         1.7.6                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchdiffeq               0.2.3                    pypi_0    pypi
[conda] torchmetrics              0.9.3                    pypi_0    pypi
[conda] torchvision               0.13.1              py310_cu113    pytorch

[ptrblck]

I did not download any Cuda packages from Nvidia, or the Ubuntu nvidia-cuda-toolkit package.

I guess your WSL2 setup might not be able to use the GPU and you might need to install the drivers as described in the user guide.

[mawright]

As you suggested, I tried installing the Nvidia-packaged copy of WSL2 CUDA using those instructions, and while it installs correctly, the issue still remains.

My example shows that CUDA matmul works fine (it was also working earlier because conda had installed its own cudatoolkit package). The problem seems to be only with the CUDNN ops.

[OYJ-2000]

I'm using Jax in WSL in the windows system. In my case, torch works well for everything, while for Jax, conv1d and conv2d don't work.
And reported the same error.

[jasonYpretty]

I try to fix symbolic link problem and solve it.
sudo ldconfig

/usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

Firstly,
cd /usr/lib/wsl/lib/
backup "libcuda.so.1" and "libcuda.so",
and
rm -r libcuda.so.1
rm -r libcuda.so
Then,
ln -s libcuda.so.1.1 libcuda.so.1
ln -s libcuda.so.1.1 libcuda.so
sudo ldconfig
done.
Now works fine.

I think the issue is on the Windows side. I followed the instructions in this post and it solved the issue.
microsoft/WSL#5663 (comment)

[shimachao]

I think the issue is on the Windows side. I followed the instructions in this post and it solved the issue. microsoft/WSL#5663 (comment)

thinks，this method works for me

[nickchomey]

I suspect this is not a WSL2 issue, but just the fact that libcudnn is a separate thing from cuda. Whatever the case, this link has install instructions for all systems - I followed the Ubuntu steps and it worked for me https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html

[db0]

I am having the same issue using a micromamba local env in Ubuntu22.04, (no WSL, just normal install)

my requirements.txt is

--extra-index-url https://download.pytorch.org/whl/cu118
torch

my conda environment.yml is

name: ldm
channels:
  - nvidia
  - conda-forge
  - defaults
dependencies:
  - cudatoolkit==11.8.0
  - git
  - pip
  - python==3.10

Running the test posted above, I get a similar response

Install

bin/micromamba create --no-shortcuts -r conda -n linux -f environment.yaml -y
bin/micromamba run -r conda -n linux python -s -m pip install -r requirements.txt

Use

$ bin/micromamba run -r conda -n linux python -s test.py
torch.cuda.is_available()=True
/home/db0/projects/torchtest/conda/envs/linux/lib/python3.10/site-packages/torch/nn/modules/conv.py:137: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
  self.weight = Parameter(torch.empty(
Y.shape=torch.Size([2, 3, 4, 4])
matrix multiply works
Could not load library libcudnn_cnn_infer.so.8. Error: libnvrtc.so: cannot open shared object file: No such file or directory

However, if I instead switch to index https://download.pytorch.org/whl/cu117, it all works normally!

[db0]

I switched to index https://download.pytorch.org/whl/cu119 and it also works

[ptrblck]

@db0 your issue is unrelated to WSL2, tracked here, and fixed here with a workaround which is available in the nightly binaries.