Is 10GB enough?

[hacksider]

Is the 10gb version of 3080 enough for this?

[tommcg]

Is the 10gb version of 3080 enough for this?

I'm dying. I have a 3080 10gb and I have to believe it will work soon.

[LePrau]

I was able to train a model in dreambooth, but had to tinker with some settings and live with some restrictions:

• as command line arguments I use set COMMANDLINE_ARGS=--xformers --precision autocast - without the precision argument, it fails horribly when using the FP16 setting.
• I made sure that I had as little VRAM usage as possible - around ~0.5GB when idle on desktop with one browser open (check in task manager)
• "train text encoder" always crashes with OOM
• checkpoints / preview images every N steps crash with OOM afterwards. Instead, train N steps with each training and simply resume training with another N steps of training. Means you have to babysit dreambooth, but at least it works reliably.
• use of classification images (directory) and amount of classification images can be used. In my case, 18 classification images for class "cat" were used.
• using text2img at any point leads to OOM error with dreambooth training afterwards. so to train dreambooth after txt2img, you have to restart webui-user.bat completely

Specific settings I used:

Steps: 100 (or after how many steps you want a new checkpoint for finetuning)
Resolution: 512 (lower resolution had barely any effect on Vram usage)
[ leave empty! ] Save a checkpoint every N steps, 0 to disable
[ leave empty! ] Generate a preview image every N steps, 0 to disable

[ ] Use CPU Only (SLOW)
[x] Don't cache latents
[ ]Train Text Encoder
[x] Use 8bit Adam
[-] Center Crop (does not matter)
[x] Gradient Checkpointing
[ ] Scale Learning Rate (not sure if it has any effect
Mixed Precision: [ fp16 ]

With all that I got ~1.5it/s on a 3080 10GB with ~9.8GB VRAM used, resulting model checkpoints were 2gb in size.

Hope that helps!

[dikasterion]

Thank you for sharing your setting:)
but for me this doesn't work...with this error message below...
RuntimeError: expected scalar type Half but found Float

if I change Mixed Precision to 'no', then this error message comes out.
RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

if I uncheck "use 8bit adam" then I get OOM...

I have no choice but waiting until the 3080 10gb is supported....but thanks!

[tommcg]

Thank you for sharing your setting:) but for me this doesn't work...with this error message below... RuntimeError: expected scalar type Half but found Float

if I change Mixed Precision to 'no', then this error message comes out. RuntimeError: CUDA error: invalid argument CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

if I uncheck "use 8bit adam" then I get OOM...

I have no choice but waiting until the 3080 10gb is supported....but thanks!

I posted below and have also added a link to a video that explains how to get this working on a 3080 10gb card. There are also some other tools in the video that may give you more insight into how your VRAM is being used. Good luck! https://youtu.be/cP44JLtXIeI

[ghost]

No luck with 3080 10GB, with the above settings. Idle desktop VRAM usage about 450-500MB on Linux Mint. Dreambooth won't even begin training anymore, a couple days ago i got it running for a little while but it OOM crashed before it finished. Now it won't even start.

[tommcg]

I just managed to get this running on my 3080 10gb. It took a few try's but I can now successfully reproduce it. I'll make a video soon and post the link here.

[tommcg]

Ok, I bashed together a little video showing my workflow to get Dreambooth working on a 10gb card. The tips above totally work. Hope some of the tips in this video will help others to ensure a successful training!

https://youtu.be/cP44JLtXIeI

-T

[LePrau]

Ok, I bashed together a little video showing my workflow to get Dreambooth working on a 10gb card. The tips above totally work. Hope some of the tips in this video will help others to ensure a successful training!

https://youtu.be/cP44JLtXIeI

-T

FIY: I am spreading your video whenever someone asks how to get dreambooth up and running with 10GB of vram. You did a nice job there, calm and clear explanation, no video-editing-shortcuts, I like that!
Especially the fact you had a non-person training item makes it stand out :)

[tommcg]

I'm glad it's useful! Things are moving so fast. I wonder how long it'll be relevant!?