3. Vision‐Language Models for Image Classification

Vision-Language Models

Vision-Language models are trained on large image-text pairs which allows them to learn a general correlation between image and text semantics. Such models typically learn a representation for image and text in a shared space using methods like contrastive learning. These representations can then be used for downstream tasks like classification, object detection, and also generative tasks like visual-question answering and caption generation.

CLIP

• CLIP is a multimodal model that embeds a pair of image and text in a shared latent space and learns the correlation between these embeddings using contrastive learning.

  • positive correlations result in embeddings that are close in the representation space while pairs that are negatively correlated are farther in the representation space.

• A pretrained CLIP model that is trained on a large vocabulary of text can be used to perform zero-shot classification.

  • compute the CLIP embeddings of the target image to be classified
  • compute the CLIP embeddings of all the class labels (text)
  • measure the cosine similarity between the image embedding and each of the class embeddings
  • the class with the highest similarity is used to classify the image

ViLBERT

PaliGemma