RuntimeError: CUDA out of memory. #1005
-
|
Greetings! I'm very new to this TTS area. Therefore, I have a question regarding the usage of GPU memory. I started training a tts model from a pre-trained model. Below is my system specifications, my
And then, to start training, I typed into the PowerShell the following: And below is the output which is followed by an error message which is the problem that I want to understand: Look that there is a message saying: I solved this problem by reducing the And here are my questions:
Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 5 replies
-
Yes, the error was due to lack of GPU memory.
4GB is really not enough for training TTS models, and it also largely depends on the model you are in interest. The glow-tts is memory hungry. Besides, the sentences in your dataset are long (max 823, avg 238 phones). The longer sentence consumes larger memory. You might be able to train a Tacotron2 by max_seq_len=150 and batch_size=10, but you may lose a lot of your sentences. The smaller batch_size is ok, but it is good to keep it above 10.
A GPU with memory size > 8G is enough, 11G is better, and >16GB is preferable. I have trained many of my models on Nvidia RTX1080ti (11GB). Recently I've got a RTX3090 (24GB) and it is about 1.8x faster training and less headache on batch_size. RTX3060ti(16G, upcoming), 2080ti (12G), a5000 (24GB), a6000 (48G) are also good, I think.
you are correct. The memory size is another bottleneck of "poor" GPU for training a large deep learning model (with millions of parameters) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for answering! |
Beta Was this translation helpful? Give feedback.
-
|
a4000(16gb) is also good for the price compared to other cards carrying the same amount of memory.
… On Dec 9, 2021, at 1:56 PM, tekinek ***@***.***> wrote:
Is these errors above, are really due to a GPU memory issue?
Yes, the error was due to lack of GPU memory.
s my GPU able to handle training of TTS models? Or do I need to get a better machine with a better GPU to achieve good results in a TTS task?
4GB is really not enough for training TTS models, and it also largely depends on the model you are in interest. The glow-tts is memory hungry. Besides, the sentences in your dataset are long (max 823, avg 238 phones). The longer sentence consumes larger memory. You might be able to train a Tacotron2 by max_seq_len=150 and batch_size=10, but you may lose a lot of your sentences. The smaller batch_size is ok, but it is good to keep it above 10.
If my GPU is not able to handle this kind of task, what kind of GPU or machine can handle this? What does the community normally use?
A GPU with memory size > 8G is enough, 11G is better, and >16GB is preferable. I have trained many of my models on Nvidia RTX1080ti (11GB). Recently I've got a RTX3090 (24GB) and it is about 1.8x faster training and less headache on batch_size. RTX3060ti(16G, upcoming), 2080ti (12G), a5000 (24GB), a6000 (48G) are also good, I think.
I thought a poor GPU could handle this kind of task, the only difference is that, compared with better GPUs, the time a poor GPU takes to complete the task is higher than a good GPU. Am I correct on this affirmation?
you are correct. The memory size is another bottleneck of "poor" GPU for training a large deep learning model (with millions of parameters)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Beta Was this translation helpful? Give feedback.
-
|
@loganhart420 how is your experiece with a4000? looking at the specs and price, it seems a good card for the price and 140W power consumption. |
Beta Was this translation helpful? Give feedback.
-
|
Before I bought it I looked at the lambda labs gpu benchmarks and they say it trains transformers faster than 3080 but is about 10% on tacotron. From my personal experience I'd say that's about right. the a4000 goes slower than my 3090 when training the transformer based tts models but not really by much. For the price and the extra 8 gigs you get compared to the 3080 I'd say it is worth it. you will lose out on some speed having those 8 extra gigs of vram really helps especially if you mostly plan to train from scratch or have some big datasets for fine tuning, plus you can really only get a 2080Ti for the price they are going for and it's a pretty big upgrade to that. |
Beta Was this translation helpful? Give feedback.
-
|
https://lambdalabs.com/gpu-benchmarks here is the lambda labs link so you can check it for yourself |
Beta Was this translation helpful? Give feedback.
-
|
my bad I meant 6 extra gigs of vram, I was thinking of the 3070 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your sharing. I've never tried A series cards. My next buy should toward a4000 (16gb) or a5000 (24gb). My turbo 3090 is very noisy, and with Tacotron I can't say it noticeably faster than my 4 years old 1080ti. It is about 1.8x faster with GAN based vocoders. |
Beta Was this translation helpful? Give feedback.
-
|
Just an extra comment to the above discussion ... many of the TTS training recipes include the I don't think this was the OP's issue, as I don't see them using |
Beta Was this translation helpful? Give feedback.
Yes, the error was due to lack of GPU memory.
4GB is really not enough for training TTS models, and it also largely depends on the model you are in interest. The glow-tts is memory hungry. Besides, the sentences in your dataset are long (max 823, avg 238 phones). The longer sentence consumes larger memory. You might be able to train a Tacotron2 by max_seq_len=150 and batch_size=10, but you may lose a lot of your sentences. The smaller batch_size is ok, but it is good to keep it above 10.