3. Vision‐Language Models for Image Classification

Vision-Language Models

CLIP

• CLIP is a multimodal model that embeds a pair of image and text in a shared latent space and learns the correlation between these embeddings using contrastive learning.

  • positive correlations result in embeddings that are close in the representation space while pairs that are negatively correlated are farther in the representation space.

• A pretrained CLIP model that is trained on a large vocabulary of text can be used to perform zero-shot classification.

  • compute the CLIP embeddings of the target image to be classified
  • compute the CLIP embeddings of all the class labels (text)
  • measure the cosine similarity between the image embedding and each of the class embeddings
  • the class with the highest similarity is used to classify the image

ViLBERT

PaliGemma