3. Vision‐Language Models for Image Classification

Vision-Language Models

Vision-Language models are trained on large image-text pairs which allows them to learn a general correlation between image and text semantics. Such models typically learn a representation for image and text in a shared space using methods like contrastive learning. These representations can then be used for downstream tasks like classification, object detection, and also generative tasks like visual-question answering and caption generation.

CLIP (2021)

• CLIP is a multimodal model that embeds a pair of image and text in a shared latent space and learns the correlation between these embeddings using contrastive learning.

  • positive correlations result in embeddings that are close in the representation space while pairs that are negatively correlated are farther in the representation space.

• A pretrained CLIP model that is trained on a large vocabulary of text can be used to perform zero-shot classification.

  • compute the CLIP embeddings of the target image to be classified
  • compute the CLIP embeddings of all the class labels (text)
  • measure the cosine similarity between the image embedding and each of the class embeddings
  • the class with the highest similarity is used to label the image

PaliGemma, GPT-4o etc.

• These more recent VLM models take an image input and extract image encodings using pre-trained models like CLIP or ViT.
• A text prompt or instruction is also encoded using a text encoder.
• A multimodal projector maps the image representation into text space. The image representation is then concatenated with text encoding and input to a pretrained LLM decoder model like LLama or QWen.
• the decoder outputs the text. For e.g. if the text instruction asks "is there a cat in the image", the text decoder answers the question, which can then be used to classify the image.