Skip to content
Discussion options

You must be logged in to vote

hii @BittuChan, I've previously used fine-tuning VLMs on a 16GB M2 Pro. Here's what worked for me:

Quick answer: Yes, 7B VLMs are tight on 16GB, but totally doable with the right settings.

Memory Optimization Strategies

  1. Reduce LoRA Layers
    This is your biggest lever. Dropping from 16 to 8 layers cuts your trainable parameters roughly in half:
mlx_vlm.lora --model mlx-community/Qwen2-VL-7B-Instruct-4bit \
  --data ./data \
  --batch-size 1 \
  --lora-layers 8 \  # Try 8 first, then 4 if still crashing
  --iters 1000

For document classification (which sounds like a focused task), 8 layers should be plenty. I've seen good results with as few as 4 layers for similar use cases. The quality d…

Replies: 2 comments 2 replies

Comment options

You must be logged in to vote
2 replies
@Shiven0504
Comment options

@BittuChan
Comment options

Answer selected by BittuChan
Comment options

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants