CLIPForImageClassification

[Andron00e]

Model description

CLIPForImageClassification is an add-on CLIPModel from hub. With one linear classifier on top of the CLIP it performs in Image Classification task as good as ViTForImageClassification does. Moreover, there are several options how to produce a classification with this model.
The first one is to use "image_embeds" output of CLIP and then produce a classification with [projection_dim, num_classes] Linear Layer.
The second one is to use "logits_per_image" output of CLIP and then do classification.

Currently, the model implementation is partially available. With weights of it, retrained on CIFAR10 dataset. My next goal is to expand the range of labels and implement the second variant of this model.

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Repo with implementation
Pre-trained model on hub

[NielsRogge]

Thanks for the suggestion, I've review the PR above

[NielsRogge]

This is now supported by #28952.