TTS Training Setup with 3080

[scott-vector]

I have spent longer than I care to admit trying to figure out how to get TTS set up for VITS training on a 3080 card. I've installed a base of Ubuntu 20.04 and used the make commands to install deps and set things up. I get a range of errors related to either a bad CUDA version, pytorch issues, etc. It seems I can't get the correct combination of dependencies installed to complete training.

This is after fighting WSL2 and nuking that, as I thought it would be easier to set up in Ubuntu (desktop).

Please, if anyone has a tried and true setup to train with a 30 series card, I would be very grateful if you could share it here. I can provide more specifics of the errors, but I wanted to put this out first in the hope of getting the magic / TLDR-type setup.

[neurlang]

I have 20.04 also. I have the 510 driver. CUDA 11.6.

Here is my cuda debs download folder, I've fetched the debs manually and installed one by one lol. But cfft I use just the newer one (upgraded to test it), actually both cufft versions suck on my gpu.

cuda-11-6_11.6.0-1_amd64.deb
cuda-cccl-11-6_11.6.55-1_amd64.deb
cuda-command-line-tools-11-6_11.6.0-1_amd64.deb
cuda-compiler-11-6_11.6.0-1_amd64.deb
cuda-cudart-11-6_11.6.55-1_amd64.deb
cuda-cudart-dev-11-6_11.6.55-1_amd64.deb
cuda-cuobjdump-11-6_11.6.55-1_amd64.deb
cuda-cupti-11-6_11.6.55-1_amd64.deb
cuda-cupti-dev-11-6_11.6.55-1_amd64.deb
cuda-cuxxfilt-11-6_11.6.55-1_amd64.deb
cuda-demo-suite-11-6_11.6.55-1_amd64.deb
cuda-documentation-11-6_11.6.55-1_amd64.deb
cuda-driver-dev-11-6_11.6.55-1_amd64.deb
cuda-drivers-510_510.47.03-1_amd64.deb
cuda-gdb-11-6_11.6.55-1_amd64.deb
cuda-libraries-11-6_11.6.0-1_amd64.deb
cuda-libraries-dev-11-6_11.6.0-1_amd64.deb
cuda-memcheck-11-6_11.6.55-1_amd64.deb
cuda-nsight-11-6_11.6.55-1_amd64.deb
cuda-nsight-compute-11-6_11.6.0-1_amd64.deb
cuda-nsight-systems-11-6_11.6.0-1_amd64.deb
cuda-nvcc-11-6_11.6.55-1_amd64.deb
cuda-nvdisasm-11-6_11.6.55-1_amd64.deb
cuda-nvml-dev-11-6_11.6.55-1_amd64.deb
cuda-nvprof-11-6_11.6.55-1_amd64.deb
cuda-nvprune-11-6_11.6.55-1_amd64.deb
cuda-nvrtc-11-6_11.6.55-1_amd64.deb
cuda-nvrtc-dev-11-6_11.6.55-1_amd64.deb
cuda-nvtx-11-6_11.6.55-1_amd64.deb
cuda-nvvp-11-6_11.6.58-1_amd64.deb
cuda-runtime-11-6_11.6.0-1_amd64.deb
cuda-samples-11-6_11.6.101-1_amd64.deb
cuda-sanitizer-11-6_11.6.55-1_amd64.deb
cuda-toolkit-11-6_11.6.0-1_amd64.deb
cuda-toolkit-11-6-config-common_11.6.55-1_all.deb
cuda-tools-11-6_11.6.0-1_amd64.deb
cuda-visual-tools-11-6_11.6.0-1_amd64.deb
gds-tools-11-6_1.2.0.100-1_amd64.deb
libcublas-11-6_11.8.1.74-1_amd64.deb
libcublas-dev-11-6_11.8.1.74-1_amd64.deb
libcufft-11-6_10.7.0.55-1_amd64.deb
libcufft-11-6_10.7.2.124-1_amd64.deb
libcufft-dev-11-6_10.7.0.55-1_amd64.deb
libcufft-dev-11-6_10.7.2.124-1_amd64.deb
'libcufile-11-6_1.2.0.100-1_amd64 (1).deb'
libcufile-11-6_1.2.0.100-1_amd64.deb
libcufile-dev-11-6_1.2.0.100-1_amd64.deb
libcurand-11-6_10.2.9.55-1_amd64.deb
libcurand-dev-11-6_10.2.9.55-1_amd64.deb
libcusolver-11-6_11.3.2.55-1_amd64.deb
libcusolver-dev-11-6_11.3.2.55-1_amd64.deb
libcusparse-11-6_11.7.1.55-1_amd64.deb
libcusparse-dev-11-6_11.7.1.55-1_amd64.deb
libnpp-11-6_11.6.0.55-1_amd64.deb
libnpp-dev-11-6_11.6.0.55-1_amd64.deb
libnvjpeg-11-6_11.6.0.55-1_amd64.deb
libnvjpeg-dev-11-6_11.6.0.55-1_amd64.deb
nsight-compute-2022.1.0_2022.1.0.12-1_amd64.deb
nsight-systems-2021.5.2_2021.5.2.53-1_amd64.deb
nvidia-modprobe_510.47.03-0ubuntu1_amd64.deb
nvidia-settings_510.47.03-0ubuntu1_amd64.deb

Ubuntu 20.04 is kind of old, so you need to use the python3 instad of python and pip3 instead of pip.
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux

torch is nightly. Version: 1.12.0.dev20220512+cu116

actually when i installed torch nightly using pip3 some dependency also pulled old torch. so I kept uninstalling and upgrading old things like old numpy, scipy and whatever untill they no more pulled the old torch. every time I uninstalled the old torch I installed the nightly torch and tried again to see what else pulls it back in.

[scott-vector]

Thank you for the information. Is this setup training correctly using VITS? As in, not breaking x minutes through, etc.?

[neurlang]

What do you mean breaking x minutes. your training crashes after, say, 5 minutes? That's unlikely.

Either it will crash in the beginning or never.

Acctually if you have a big wav in your dataset it can out of memory in progress.

[scott-vector]

I was just confirming this is a functional setup you proposed, which it looks like it is... I will update this thread to confirm once I've loaded it in. Thank you.