CUFFT_INTERNAL_ERROR on RTX 4090

[Yujia-Yan]

🐛 Describe the bug

>>> import torch
>>> torch.fft.rfft(torch.randn(1000).cuda())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

There is a discussion on https://forums.developer.nvidia.com/t/bug-ubuntu-on-wsl2-rtx4090-related-cufft-runtime-error/230883/7 .

Versions

Using pytorch installed with
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

cc @ezyang @gchanan @zou3519 @peterjc123 @mszhanyi @skyline75489 @nbcsm @ngimel @mruberry @peterbell10

[malfet]

Just to clarify, this happen only in Windows Subsystem for Linux or elsewhere as well?

[malfet]

@ptrblck can you please confirm that this indeed happens when with 4090 on Linux or only in WSL config?

[Yujia-Yan]

Just to clarify, this happen only in Windows Subsystem for Linux or elsewhere as well?

Actually I am using ubuntu Server 22.04.
Update:
I later tried to compile pytorch with CUDA 11.8 and this problem disappears.

[ngimel]

So in this case it looks like cufft library doesn't support forward compatibility guarantee (you can run code compiled with older toolkit version, as long as driver on the system supports the new hardware). cc @ptrblck, and we should start producing 11.8 nightlies.

[Blackhex]

I don't have 4090 available so I can only add that it is not reproducible on Windows 11 and Ubuntu WSL with 3080.

[ptrblck]

can you please confirm that this indeed happens when with 4090 on Linux or only in WSL config?

Yes, this is a cuFFT error which is also visible on Linux.

and we should start producing 11.8 nightlies.

Also yes, and I've already started with its bringup, e.g. in: pytorch/builder#1186

[malfet]

Some updates:

• this seems to be the bug in CuFFT in CUDA-11.7 that happens on both Linux and Windows, but seems to be fixed in 11.8
• It worth trying (and I think some investigation has already been done) to use CuFFT from 11.8 in 11.7 build to see if the fix could be deployed/verified to nightlies first

[malfet]

Let's create a frankenbuild for cuda-11.7 for nightlies and see what will happen

[malfet]

First thing, that worries me a lot is 2x binary size increase for cuda-11.8 and nvprune does not help much:

$ ls -lah 11.7/libcufft/lib64/libcufft.so.10.7.2.50 11.8/libcufft/lib64/libcufft.so.10.9.0.58 
-rwxr-xr-x 1 nshulga nshulga 131M Nov 29 21:48 11.7/libcufft/lib64/libcufft.so.10.7.2.50
-rwxr-xr-x 1 nshulga nshulga 267M Nov 29 21:49 11.8/libcufft/lib64/libcufft.so.10.9.0.58

Considering that, I'm not sure if it will be safe to include it as an update in 1.13.1

[malfet]

Removing high priority label, as this is a bug in a 3rd party library and there were big changes between cufft in 11.7 and 11.8

[pranavmalikk]

Still getting this error on RTX 4090 with Cuda 11.7 on Ubuntu 22.04, any recommendations?

[ptrblck]

@pranavmalikk Yes, please use the nightly binaries with CUDA 11.8.

[pranavmalikk]

@pranavmalikk Yes, please use the nightly binaries with CUDA 11.8.

I'm still getting this on CUDA 11.8:

----> 1 torch.fft.rfft(torch.randn(1000).cuda())

RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

[ptrblck]

Could you post the output of torch.__version__ as well as python -m torch.utils.collect_env, please, as I cannot reproduce the error in the CUDA 11.8 nightly binaries anymore:

import torch
print(torch.__version__)
out = torch.fft.rfft(torch.randn(1000).cuda())
print(out.sum())

with 11.7 it fails as reported:

python tmp.pt 
2.0.0.dev20230204+cu117
Traceback (most recent call last):
  File "tmp.pt", line 3, in <module>
    torch.fft.rfft(torch.randn(1000).cuda())
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

Update the nightlies to the 11.8 build:

pip uninstall torch -y
Found existing installation: torch 2.0.0.dev20230204+cu117
Uninstalling torch-2.0.0.dev20230204+cu117:
  Successfully uninstalled torch-2.0.0.dev20230204+cu117
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118
...
Successfully installed torch-2.0.0.dev20230204+cu118

and it works:

python tmp.py 
2.0.0.dev20230204+cu118
tensor(670.6870+11.1756j, device='cuda:0')

[pranavmalikk]

Could you post the output of torch.__version__ as well as python -m torch.utils.collect_env, please, as I cannot reproduce the error in the CUDA 11.8 nightly binaries anymore:

import torch
print(torch.__version__)
out = torch.fft.rfft(torch.randn(1000).cuda())
print(out.sum())

with 11.7 it fails as reported:

python tmp.pt 
2.0.0.dev20230204+cu117
Traceback (most recent call last):
  File "tmp.pt", line 3, in <module>
    torch.fft.rfft(torch.randn(1000).cuda())
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

Update the nightlies to the 11.8 build:

pip uninstall torch -y
Found existing installation: torch 2.0.0.dev20230204+cu117
Uninstalling torch-2.0.0.dev20230204+cu117:
  Successfully uninstalled torch-2.0.0.dev20230204+cu117
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118
...
Successfully installed torch-2.0.0.dev20230204+cu118

and it works:

python tmp.py 
2.0.0.dev20230204+cu118
tensor(670.6870+11.1756j, device='cuda:0')

Thank you for the help, it works now as i mistakenly did 'pip install torchaudio' so it set me back to an older version of torch. I fixed this by installing the nightly version of torchaudio

[bensonbs]

Due to package dependency issues, I am limited to using versions of PyTorch that are below 2.0.0. I understand that PyTorch 1.13.1 supports up to CUDA 11.7. Could you kindly advise if there are any alternative solutions apart from upgrading to CUDA 11.8?