oom

[Tomsentable]

i used 8*A100 to fintine the llama-7b weight,when iteration>=3000,it save weight oom,how to fix it?

[thuqinyj16]

Lower the batch size per GPU. You can also restart training from an existing checkpoint (e.g., trained for 3000 steps). Deepspeed has some bugs unfixed.