Accelerate Config issues? #616
Unanswered
mediocreatmybest
asked this question in
Q&A
Replies: 1 comment
-
Ok,Yep. Looks like its generating black images in the samples. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi!
I've been trialing different settings on my GPU (12GB vRAM) to try reduce the ram usage as much as possible.
I was also getting some Accelerate warnings on startup so I ran through the "Accelerate Config"
Does anyone know any ideal settings to help reduce RAM usage?
I've trialed DeepSpeed but that seems to raise the ram usage regardless of settings.
One thing I have noticed if I set the floating point to FP16 in "Accelerate Config" the ram usage drops down to less than 5GB reported usage while it is training on SD2.1 Non EMA model (It is still running so I can't confirm if it works or not) I've noticed the Loss is instantly lists as NaN.
So I suspect this isn't working correctly, any thoughts or ideas?
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions