It is strange that PyTorch is slow on RTX 3090

[Nyohohoho]

With the same CNN, I tested the time spent in inference on several GPUs with different versions of PyTorch.

• GTX 1080 Ti + PyTorch 1.8.0 + CUDA 10.2 = 1.2 ms
• RTX 2080 Ti + PyTorch 1.8.0 + CUDA 10.2 = 0.6 ms
• RTX 3090 + PyTorch 1.8.0 + CUDA 11.1 = 0.94 ms
• RTX 3090 + PyTorch 1.7.0 + CUDA 11.0 = 1.12 ms

I also found in #47908 that PyTorch with CUDA 11.0 has problem in efficiency, right?
Since RTX 3090 cannot use PyTorch with CUDA 10.2, it is a little pity that the latest GPU cannot show itself with full power...

cc @ngimel @VitalyFedyunin

[skyline75489]

What is the "runtime" you are testing exactly? To me these numbers (millisecond level) of runtime may just be the result of measuring error. It does not really show that 3090 is slower, because the workload is just too lightweight.

[Nyohohoho]

What is the "runtime" you are testing exactly? To me these numbers (millisecond level) of runtime may just be the result of measuring error. It does not really show that 3090 is slower, because the workload is just too lightweight.

I want to say "the time spent in inference".
I tested the time on each GPU for 1000 rounds.
I don't think it is measuring error.
I do not mean RTX 3090 itself is low.
Since I also read #47908 , I guess it is because PyTorch with CUDA 11 has the problem in efficiency, and RTX 3090 has not choice but to use it.
If you are interested, you may also try to do so on your machines.

[ngimel]

cc @xwang233, likely related to #50153

[xwang233]

Thanks for reporting the issue. #50153 might be related. To verify this, you can run your benchmark script on a pytorch source build. For example, try a pytorch container built by Nvidia https://ngc.nvidia.com/catalog/containers/nvidia:pytorch

[mctigger]

Just want to add that I can confirm that at least my code runs ~50% faster with https://ngc.nvidia.com/catalog/containers/nvidia:pytorch on my RTX3090 compared to the standard conda install of 1.8 or nightly.

[FrancescoSaverioZuppichini]

I can confirm it as well