training using the GPU

[thematheusk3]

Hello everyone o/

I am having trouble training using the GPU.

Win 11
RTX 3060
Python 3.10.6
Tensorflow version 2.10.0
Cuda 11.2
Cudnn 8.1

The graphics card is detected using the following code:
print("Num GPUs available", len(tf.config.experimental.list_physical_devices('GPU')))
tf.test.gpu_device_name()

However, during training, the output shows 'GPU available: False, used: False'.

[jiaxinf]

Okay, I figured it out after hours of investigation. You need to first confirm if CUDA is available on your PC, run these commands:

python
import torch
torch.cuda.is_available()

if it returns false, then you have some problems.
My attempt is to:

1. install Anaconda
2. start your Anacoda ternimal, not your regular PC terminal
3. create conda environment conda create -n env_name python=3.10, and activate it conda activate env_name
4. install Pytorch from there conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
5. run the list of 3 commands at the top again, and ensure it returns true.(if you start training now, you may get 'no module name torch' error)
6. install this project with pip install -U so-vits-svc-fork
7. remember to always run your training or hubert commands from Anaconda terminal

Start your Task Manager, go to GPU and toggle 'CUDA' to see if there's any input there.

[jiaxinf]

No problem, glad to help! It takes ~30s for me to complete 1 epoch and I'm on Tesla T4, so you should be expecting something faster. But I heard to achieve the optimal result you may need ~25k steps, and steps needed for 1 epoch = sample size/batch_size, the training time also depends on epoch_number & batch_size in your config file.

[Saaaul]

Faced such an error, tell me, please, how can I solve it? Appears after svc train -t
3080TI + CUDA

RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Saaaul]

It's decided! It was necessary to set the standard settings of the video card. I disabled the configured profile in MSI Afterburner and everything worked!

[Saaaul]

But I still come across this warning: UserWarning: Type d Storage is deprecated
What does it mean and how to solve it?

[jiaxinf]

Warnings do not affect usage for users, you should be fine. If you do dislike them, bring this up to the developers of this project. Probably an old version of the library or class is used in the code.

[onikaii]

@jiaxinf
Thanks, your instructions worked. This is my first time training a model,

I am training with more than 1000 audios,
I have a rxt 3060ti (8gb), "epochs": 400, "batch_size": 8 (if I enter something greater than 8 I get an error message)
each epoch does it takes ~120s.

cpu percentage usage does not exceed 25%, Is there any way to speed up the epochs?
or is there a way/formula to calculate epochs and batch_size for optimal performance?

[jiaxinf]

do you want to increase the learning_rate or perhaps virtual memory in advanced system setting to see if that helps? My batch_size is 16 so it takes fewer steps in one epoch, also I only have 200+ samples, and my GPU memory is 24GB. 1000/8 = 125 steps for 1 single epoch in your case so you may want to reduce the sample size or rent a cloud GPU that supports larger batch_size like 32 or 40.

[LROER]

I don't know why, but my GPU is still not being detected.

but cuda is avalible

Do you have any ideas?"

[evilcrusher2]

So I've gone backwards to geforce 3.26 and driver down to 531.61 back in April 2023. No dice. Same thing and it's not utlizing the whole gpu like it does on my laptop with the 3070ti. I got this card because it's better architecture should make it quicker, and has more vram which means I can have larger batches at a time to speed things up a little. But it appears that it's barely utilizing outside of CUDA and forcing the heavy lifting on my AMD quad core that wasn't meant for this. Heck the only reason I didn't upgrade the CPU was because it's good enough for doing the regular computing and the GPU would be doing most of the heavy lifting as I edit video, graphics, and music mainly.

[jiaxinf]

Was the 3070ti using TPU etc. in training? I wasn't using any of those in my training. Also, what's your batch_size value?

[evilcrusher2]

So I was finally able to get to my laptop with the 3070ti and run a training session with it to check it out. And you are correct in them not being used in training, as they weren't in the training module I just ran. Only the GPU was identified while the script was running. I ran 100 epoch of a batch size of 7 in about 6 minutes from hitting enter on the command until finish. This is the system results from the laptop GPU while the training was running. note that cuda isn't in the drop down selector.

As well, the batch size on the 4070 rig is 16 because I know the ram can handle that. So I don't get why the 3070ti is having 100% of the gpu utilized while using almost nothing of the CPU, and the 4070 is doing the opposite.

I did notice that the firmware versions were from a different earlier date, back in March. I wonder if that is making the difference. I will be getting ready to do a fresh install of Windows on the 4070 rig on a different m2 drive with 1tb of space instead of a smaller 250gb OS drive that came with the rig. The m2 drive is there, just needs to have data moved over to my 3rd or and external drive. I'll try to find earlier drivers and see if maybe that is also causing the issue. The only thing after that would be to update the OS to Win 11 in an attempt to get the two environments almost identical.

[evilcrusher2]

Was the 3070ti using TPU etc. in training? I wasn't using any of those in my training. Also, what's your batch_size value?

And this is on a fresh install of windows with an install not using the batch file to auto install everything. Yes, from what I can gather the batch size is in a way determining how much gpu gets utilized. I can't tell if its smaller or bigger batches that would be better when within the ram parameters. It's almost like when it gets geared up to start processing, the next batch has to be done and it's having to start from scratch and slows down.

I'm at a loss other than the possibility of older drivers worked better and this card is too new with too new of drivers? I don't get why the 3070ti is utilizing itself more than the 4070. Or could it literally be the CPU bottlenecking it?

[evilcrusher2]

I have noticed that 11.8 shouldn't be required given that the google colab version is running on cards with v12.* of CUDA. Unless there is a major difference between what is running on those notebooks and what's publicly released, this shouldn't be the issue.