Repository navigation
Memory issues when fine-tuning Qwen2-VL with LoRA on 16GB M1 Max - any optimization tips? #932
|
Hello everyone, I've been trying to fine-tune the Qwen2-VL-7B model on my M1 Max (16GB unified memory) using the LoRA setup from the docs, but keep hitting OOM errors during training. I'm working with a custom dataset of around 800 image-text pairs for a document classification task. Here's my current command: The model loads fine and inference works perfectly, but training crashes about 50 iterations in with a memory error. I've already tried:
A few specific questions:
For context, I'm targeting a research paper submission so I need this to actually work, but I don't have access to beefier hardware right now. Any advice from folks who've successfully fine-tuned VLMs on similar setups would be super helpful. |
Replies: 2 comments 2 replies
|
hii @BittuChan, I've previously used fine-tuning VLMs on a 16GB M2 Pro. Here's what worked for me: Quick answer: Yes, 7B VLMs are tight on 16GB, but totally doable with the right settings. Memory Optimization Strategies
For document classification (which sounds like a focused task), 8 layers should be plenty. I've seen good results with as few as 4 layers for similar use cases. The quality difference isn't massive unless you're doing something really complex.
|
|
Also turns out my dataset was preprocessing images at full resolution (some were 4K+) which was definitely contributing to the OOM issues. Capped them at 768x768 and that helped a ton. Marking this as resolved. Really appreciate the detailed breakdown, this saved me days of trial and error! |
hii @BittuChan, I've previously used fine-tuning VLMs on a 16GB M2 Pro. Here's what worked for me:
Quick answer: Yes, 7B VLMs are tight on 16GB, but totally doable with the right settings.
Memory Optimization Strategies
This is your biggest lever. Dropping from 16 to 8 layers cuts your trainable parameters roughly in half:
For document classification (which sounds like a focused task), 8 layers should be plenty. I've seen good results with as few as 4 layers for similar use cases. The quality d…