[Bug]: Qwen2-VL-2B quantization model has no improvement in reasoning speed compared to the original model

[Eduiskss]

Your current environment

I performed GPTQ-int8 quantization using the fine-tuned Qwen2-VL-2B model, and there was no improvement in model inference speed after quantization

🐛 Describe the bug

I performed GPTQ-int8 quantization using the fine-tuned Qwen2-VL-2B model, and there was no improvement in model inference speed after quantization

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Are you able to profile the code and see the cause? I suspect that the main bottleneck for Qwen2-VL series models is in the image preprocessing step which isn't improved via quantizing the model.

[Eduiskss]

Are you able to profile the code and see the cause? I suspect that the main bottleneck for Qwen2-VL series models is in the image preprocessing step which isn't improved via quantizing the model.

I have thought about this, I now try to go to the image processing part first, because I have a warning about slow processors while reasoning, I try to find the file to set use_fast=True, but I encountered the error reporting problem

[Eduiskss]

Are you able to profile the code and see the cause? I suspect that the main bottleneck for Qwen2-VL series models is in the image preprocessing step which isn't improved via quantizing the model.

I tried to modify here but it doesn't work

[DarkLight1337]

What error did you get?

[Eduiskss]

What error did you get?

[DarkLight1337]

Can you try editing the preprocessor_config.json in your downloaded HF repo directly instead to use the fast class?

[Eduiskss]

Can you try editing the in your downloaded HF repo directly instead to use the fast class?preprocessor_config.json

I solved the problem after trying to add size, it runs without the slow processor warning, but it doesn't seem to have much effect on inference efficiency, and it does seem to be on inference according to tokens processing speeds

[DarkLight1337]

Probably you should run a profiler on the code. e.g. use https://github.com/benfred/py-spy

[Eduiskss]

Probably you should run a profiler on the code. e.g. use https://github.com/benfred/py-spy

It seems that the preprocessing part also has some impact, the largest forward takes up 44%, does this make sense

[DarkLight1337]

Yes, it's a known issue that HF's Qwen2-VL preprocessing is very slow.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!