New feature: text/text contrastive

[rom1504]

Can we adapt openclip to be able to train text/text contrastive models?

And beyond that maybe, text/test/image models ?

use case:

• train pure contrastive text models either for multilingual pairs, or mono lingual
• use similarity between text pairs and also with images to have a better text understanding while having a good text,image understanding

options:

• single tower for text1 and text2
• two towers

It would be nice to find a way to do this without making the code overly complicated.

It goes in a direction of supporting more modalities combination in openclip

A motivation is there are few good models for text,text surprisingly, even though the community on this is quite active

a related idea is image/image as inspired by https://arxiv.org/abs/2212.08045

reference of private models to beat:

• https://twitter.com/Nils_Reimers/status/1602355249297358849
• https://twitter.com/OpenAI/status/1603466863370854401

[jn2clark]

Hi @rom1504, I am very interested in this, particularly text/text/image using a single tower for text. If you have started anything I would love to see it, otherwise I will probably try and get something going in the next couple of weeks.

[rom1504]

Check #323

[jn2clark]

Amazing! Thank you!