[Bug] cuFFT error

[mesut92]

Describe the bug

I am trying to train vits with ljspeech on 4090. i am getting that error, i could not fix. I update the torch and nvidia drivers.

To Reproduce

run this code: python recipes/turk/vits_tts/train_vits.py

getting this error
/usr/local/lib/python3.8/dist-packages/torch/functional.py:632: UserWarning: stft will soon require the return_complex parameter be given for real inputs, and will further require that return_complex=True in a future PyTorch release. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:801.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
! Run is removed from /media/mesut/Depo1/works/TTS/recipes/turk/vits_tts/vits_ljspeech-February-26-2023_08+55AM-0000000
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/trainer/trainer.py", line 1591, in fit
self._fit()
File "/usr/local/lib/python3.8/dist-packages/trainer/trainer.py", line 1544, in _fit
self.train_epoch()
File "/usr/local/lib/python3.8/dist-packages/trainer/trainer.py", line 1309, in train_epoch
_, _ = self.train_step(batch, batch_num_steps, cur_step, loader_start_time)
File "/usr/local/lib/python3.8/dist-packages/trainer/trainer.py", line 1126, in train_step
batch = self.format_batch(batch)
File "/usr/local/lib/python3.8/dist-packages/trainer/trainer.py", line 926, in format_batch
batch = self.model.format_batch_on_device(batch)
File "/media/mesut/Depo1/works/TTS/TTS/tts/models/vits.py", line 1503, in format_batch_on_device
batch["spec"] = wav_to_spec(wav, ac.fft_size, ac.hop_length, ac.win_length, center=False)
File "/media/mesut/Depo1/works/TTS/TTS/tts/models/vits.py", line 123, in wav_to_spec
spec = torch.stft(
File "/usr/local/lib/python3.8/dist-packages/torch/functional.py", line 632, in stft
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

Expected behavior

start to train

Logs

No response

Environment

{
    "CUDA": {
        "GPU": [
            "NVIDIA GeForce RTX 4090"
        ],
        "available": true,
        "version": "11.7"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "1.13.1+cu117",
        "TTS": "0.11.1",
        "numpy": "1.21.6"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.8.10",
        "version": "#66~20.04.1-Ubuntu SMP Wed Jan 25 09:41:30 UTC 2023"
    }
}

Additional context

No response

[erogol]

can't reproduce. In general it is OOM issue

[pathnirvana]

I am getting the same error on a rtx 4090 on the ljspeech dataset using the !CUDA_VISIBLE_DEVICES=0 python3 recipes/ljspeech/vits_tts/train_vits.py

edit: a solution is mentioned here