Fast captioning

[rahimentezari]

I would like to use CogVLM for captioning images in large scale. Currently using Chat() function takes about 2.5 sec for captioning one image on an A100 GPU which makes it almost impossible to use for million-scale. What do you recommend to speed up? I see that quantization is not yet supported for example #9.

[eezywu]

same problem

[1049451037]

Quantization can be beneficial. However, that portion of the code was not written by me. As far as I know, there are no plans to open-source the code.

[1049451037]

We now support 4-bit quantization! See README for more details.

[ProGamerGov]

@rahimentezari Have you tested captioning speed on the A100 GPU for 4-bit quantization yet?

[chakrabortyrajatsubhra]

Hey can anyone say how to caption a folder of images with the sat demo model? I will have a fixed query like describe the image. Any suggestions would be greatly helpful.