Feature Request
Currently, mmproj (multimodal projection) files for vision-language models like Qwen2-VL are only available in f16 and f32 formats. It would be valuable to support quantized mmproj files (Q8_0, Q4_0, etc.) to reduce model size for mobile/embedded deployment.
Use Case
Deploying Qwen2-VL-2B on Android using llama.cpp. The main model quantizes well (Q4_K_M is ~1GB), but the mmproj at 1.33GB (f16) or 2.66GB (f32) is a significant bottleneck for mobile deployment where storage and bandwidth are constrained.
Proposed Solution
Add support for quantized mmproj files in the conversion pipeline, similar to how the main model can be quantized. Estimated sizes:
- mmproj Q8_0: ~665MB (50% reduction from f16)
- mmproj Q4_0: ~333MB (75% reduction from f16)
Impact
This would make vision-language models much more practical for on-device mobile inference while likely having minimal impact on vision encoding quality.
Related
Feature Request
Currently, mmproj (multimodal projection) files for vision-language models like Qwen2-VL are only available in f16 and f32 formats. It would be valuable to support quantized mmproj files (Q8_0, Q4_0, etc.) to reduce model size for mobile/embedded deployment.
Use Case
Deploying Qwen2-VL-2B on Android using llama.cpp. The main model quantizes well (Q4_K_M is ~1GB), but the mmproj at 1.33GB (f16) or 2.66GB (f32) is a significant bottleneck for mobile deployment where storage and bandwidth are constrained.
Proposed Solution
Add support for quantized mmproj files in the conversion pipeline, similar to how the main model can be quantized. Estimated sizes:
Impact
This would make vision-language models much more practical for on-device mobile inference while likely having minimal impact on vision encoding quality.
Related