3. Vision‐Language Models for Image Classification

Vision-Language Models

CLIP is a multimodal model that embeds a pair of image and text in a shared latent space and learns the correlation between these embeddings using contrastive learning.
- positive correlations result in embeddings that are close in the representation space while pairs that are negatively correlated are farther in the representation space.
A pretrained CLIP model that is trained on a large vocabulary of text can be used to perform zero-shot classification.
- compute the CLIP embeddings of the target image to be classified
- compute the CLIP embeddings of all the class labels (text)
- measure the cosine similarity between the image embedding and each of the class embeddings
- the class with the highest similarity is used to classify the image