The speed of pytorch with cudatoolkit 11.0 is slower than cudatoolkit 10.2

[klyjm]

🐛 Bug

When I update the pytorch to 1.7, the cudatoolkit is updated automaticlly to 11.0, and I find the speed of the same code is slower too much than before. So I change the version of the cudatoolkit back to 10.2, the speed is normal. Maybe I should update the cudnn version in Ubuntu?

To Reproduce

I just use the same code in the same device with the same environment only change the version of the cudatoolkit, the speed is slower too much.

cudatoolkit 10.2
Speed:  13.4/1.3/14.6 ms inference/NMS/total per 640x640 image at batch-size 1

cudatoolkit 11.0
Speed: 27.0/1.2/28.2 ms inference/NMS/total per 640x640 image at batch-size 1

Expected behavior

The speed of 11.0 should be no more slower than 10.2.

Environment

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

Collecting environment information...
PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration:
GPU 0: TITAN V
GPU 1: TITAN V
GPU 2: TITAN V
GPU 3: TITAN V
GPU 4: TITAN V
GPU 5: TITAN V

Nvidia driver version: 450.80.02
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.7.0
[pip3] torchvision==0.8.1
[conda] blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] cudatoolkit               11.0.221             h6bb024c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] libblas                   3.8.0                    20_mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] libcblas                  3.8.0                    20_mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] liblapack                 3.8.0                    20_mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] liblapacke                3.8.0                    20_mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] mkl                       2020.2                      256    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl-service               2.3.0            py38he904b0f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl_fft                   1.2.0            py38h23d657b_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl_random                1.1.1            py38h0573a6f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] numpy                     1.19.1           py38hbc911f0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] numpy-base                1.19.1           py38hfa32c7d_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] pytorch                   1.7.0           py3.8_cuda11.0.221_cudnn8.0.3_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torchvision               0.8.1                py38_cu110    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

Additional context

cc @ngimel @VitalyFedyunin

[jmuchovej]

I can confirm this is the case as well. Recently had a 2x speedup downgrading from CUDA 11 to CUDA 10.2 on a GTX 1080 Ti.

Collecting environment information...
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Pop!_OS 20.04 LTS (x86_64)
GCC version: (Homebrew GCC 5.5.0_7) 5.5.0
Clang version: 11.0.0
CMake version: version 3.18.4

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89 (11.1 installed by Pop!_OS)
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 455.38
cuDNN version: 7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.4
[pip3] pytorch-lightning==1.0.5
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] blas                      2.20                        mkl    conda-forge
[conda] cudatoolkit               10.2.89              hfd86e86_1    anaconda
[conda] libblas                   3.8.0                    20_mkl    conda-forge
[conda] libcblas                  3.8.0                    20_mkl    conda-forge
[conda] liblapack                 3.8.0                    20_mkl    conda-forge
[conda] liblapacke                3.8.0                    20_mkl    conda-forge
[conda] mkl                       2020.2                      256    conda-forge
[conda] numpy                     1.19.4           py38hf0fd68c_0    conda-forge
[conda] pytorch                   1.6.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] pytorch-lightning         1.0.5              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.7.0                py38_cu102    pytorch

This is output within the conda environment I experienced this speedup in.

[LukeAI]

@ionlights @klyjm do you know if this is still the case with pytorch 1.7.1 ?

[klyjm]

@ionlights @klyjm do you know if this is still the case with pytorch 1.7.1 ?

@LukeAI It is not stable, sometimes the speeds are same, sometimes the 11.0 is slower. So, I still use 10.2.

[benfei]

@ionlights @klyjm do you know if this is still the case with pytorch 1.7.1 ?

It is for me (about 20% slower on CUDA 11.0 compared to to CUDA 10.1).

Here is my first logs (CUDA=10.1):

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
Clang version: Could not collect
CMake version: version 2.8.12.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.1.243             h036e899_6    conda-forge
[conda] mkl                       2020.4             h726a3e6_304    conda-forge
[conda] mkl-service               2.3.0            py38h1e0a361_2    conda-forge
[conda] mkl_fft                   1.2.0            py38hab2c0dc_1    conda-forge
[conda] mkl_random                1.2.0            py38hc5bc63f_1    conda-forge
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] pytorch                   1.7.1           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchaudio                0.7.2                      py38    pytorch
[conda] torchvision               0.8.2                py38_cu101    pytorch

Here is my second logs (CUDA=11.0):

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
Clang version: Could not collect
CMake version: version 2.8.12.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.0.3               h15472ef_6    conda-forge
[conda] mkl                       2020.4             h726a3e6_304    conda-forge
[conda] mkl-service               2.3.0            py38h1e0a361_2    conda-forge
[conda] mkl_fft                   1.2.0            py38hab2c0dc_1    conda-forge
[conda] mkl_random                1.2.0            py38hc5bc63f_1    conda-forge
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] pytorch                   1.7.1           py3.8_cuda11.0.221_cudnn8.0.5_0    pytorch
[conda] torchaudio                0.7.2                      py38    pytorch
[conda] torchvision               0.8.2                py38_cu110    pytorch

[klyjm]

@ionlights @klyjm do you know if this is still the case with pytorch 1.7.1 ?

It is for me (about 20% slower on CUDA 11.0 compared to to CUDA 10.1).

Here is my first logs (CUDA=10.1):

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
Clang version: Could not collect
CMake version: version 2.8.12.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.1.243             h036e899_6    conda-forge
[conda] mkl                       2020.4             h726a3e6_304    conda-forge
[conda] mkl-service               2.3.0            py38h1e0a361_2    conda-forge
[conda] mkl_fft                   1.2.0            py38hab2c0dc_1    conda-forge
[conda] mkl_random                1.2.0            py38hc5bc63f_1    conda-forge
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] pytorch                   1.7.1           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchaudio                0.7.2                      py38    pytorch
[conda] torchvision               0.8.2                py38_cu101    pytorch

Here is my second logs (CUDA=11.0):

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
Clang version: Could not collect
CMake version: version 2.8.12.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.0.3               h15472ef_6    conda-forge
[conda] mkl                       2020.4             h726a3e6_304    conda-forge
[conda] mkl-service               2.3.0            py38h1e0a361_2    conda-forge
[conda] mkl_fft                   1.2.0            py38hab2c0dc_1    conda-forge
[conda] mkl_random                1.2.0            py38hc5bc63f_1    conda-forge
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] pytorch                   1.7.1           py3.8_cuda11.0.221_cudnn8.0.5_0    pytorch
[conda] torchaudio                0.7.2                      py38    pytorch
[conda] torchvision               0.8.2                py38_cu110    pytorch

@feinsteinben Yeah， I have just test on my code, the speed of 1.7.1 in cuda 11.0 is about 25% slower than 10.2, more faster than 1.7.0, but still slower.

[thecooltechguy]

Hi, I'm also facing the same issue (tried on A100 GPUs which I think need cuda >= 11). Was anybody able to overcome this issue? Thanks

[Godricly]

Hi, I'm also facing the same issue on 1080 when debugging a Libtorch segmentation model. The pytorch branch is v1.7.1.

CUDA10.1-cudnn7	CUDA11.1-cudnn8
187ms	320ms

[klyjm]

Hi, I'm also facing the same issue on 1080 when debugging a Libtorch segmentation model. The pytorch branch is v1.7.1.

CUDA10.1-cudnn7 CUDA11.1-cudnn8
187ms 320ms

The speed is really still slower when using CUDA 11, I don't know what makes.

[Godricly]

Hi, I'm also facing the same issue on 1080 when debugging a Libtorch segmentation model. The pytorch branch is v1.7.1.
CUDA10.1-cudnn7 CUDA11.1-cudnn8
187ms 320ms

The speed is really still slower when using CUDA 11, I don't know what makes.

Have you tried to build your own PyTorch with cuda11.1(cuda11.2 is released but no cudnn support yet.) or use daily built PyTorch?
My colleague duplicated my comparision on RTX 2080Ti, but no difference is observed. I guess this is related to the device arch.

[y0ast]

I am also getting a 2x slowdown with cuda 11 vs 10.2 on pytorch 1.7.1 on a GTX1080Ti.

[opeide]

Same problem on ubuntu 18.04 using titan rtx. Almost 2x speed up on batch size 5 when using conda and downgrading from
pytorch-1.7.1-py3.7_cuda11.0.221_cudnn8.0.5_0
to
pytorch-1.7.0-py3.7_cuda10.2.89_cudnn7.6.5_0.
Driver version is 460. Should i perhaps downgrade to 455? cudnn support matrix confuses me a bit.

edit: tested the 1.8 nightly, which came with cuda 11.0 and cudnn 8.0.3, and did not encounter speed issues.

[jucor]

Is it happening also with Cuda 11.2 (supported by cudnn 8.1.0 since January 26th)?