VisionTextDualEncoder

[patil-suraj]

What does this PR do?

This PR adds VisionTextDualEncoder model in PyTorch and Flax to be able to load any pre-trained vision (ViT, DeiT, BeiT, CLIP's vision model) and text (BERT, ROBERTA) model in the library for vision-text tasks like CLIP.

This model pairs a vision and text encoder and adds projection layers to project the embeddings to another embeddings space with similar dimensions. which can then be used to align the two modalities.

The API to load the config and model is similar to the API of EncoderDecoder and VisionEncoderDecoder models.

• load vit-bert model from config

config_vision = ViTConfig()
config_text = BertConfig()

config = VisionTextDualEncoderConfig.from_vision_text_configs(config_vision, config_text, projection_dim=512)
# Initializing a BERT and ViT model
 model = VisionTextDualEncoderModel(config=config)

• load using pre-trained vision and text model

model = VisionTextDualEncoderModel.from_vision_text_pretrained(
    "google/vit-base-patch16-224", "bert-base-uncased"
)

Since this is a multi-modal model, this PR also adds a generic VisionTextDualEncoderProcessor, which wraps any feature extractor and tokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
processor = VisionTextDualEncoderProcessor(feature_extractor, tokenizer)

[patrickvonplaten]

Let's also update the README no?

[NielsRogge]

I believe @sgugger was against this.

[sgugger]

Yes we removed generic classes like the Vision Encoder Decoder, especially since they did not have a research article coming with it. This one has an article as an example, so you can add it if you really want it.

[patil-suraj]

Since there is no official research implementation and pre-trained checkpoint I would also prefer to not add it.

[patrickvonplaten]

Ok for me

[patrickvonplaten]

Looks great! Think the only thing left to do is to update the README to include the model class there as well :-)