Implementation of Image-to-Text (Captioning)

[bryanwong17]

Hi, I was wondering if CONCH is able to directly convert an image to text? From the code, it seems like CONCH is only available for "image-to-text retrieval," meaning that given an image and several texts, it will check which text is most similar to the given image. However, in the paper, there is also an example of CONCH doing captioning and a comparison between predicted and corrected captions. If so, could you please provide the code for doing captioning? Thanks!

[Weiqin-Zhao]

I am also looking for this amazing function of this excellent work, hope the authors can release the corresponding code and weights in the future.

[fedshyvana]

We don't have the code for image captioning at this time since the repo is meant to only provide inference capabilities to keep the code base clean. But feel free to use the open_clip (https://github.com/mlfoundations/open_clip) as reference if you want to implement this functionality!