Image similarity?

[youssefavx]

Incredible work as always you guys! In looking at the Colab, it seems it's possible to do image-to-text similarity but I'm curious if it's possible to compare image similarity as well.

For instance, if I just replace 'text_features' with 'image_features' would that work / be the best way to do this?

image_features /= image_features.norm(dim=-1, keepdim=True)
image_2_features /= image_2_features.norm(dim=-1, keepdim=True)
similarity = image_2_features.cpu().numpy() @ image_features.cpu().numpy().T

[jongwook]

Thanks!

I believe image similarities will work, although we haven't experimented/evaluated tasks using image similarity measures specifically.

[youssefavx]

Thank you! I'll try it out and see

[rom1504]

I tried it and it seems to work pretty well. (actually better than using a pretrained on classification efficientnet for example)

[thoppe]

I've also tried it and it works well. Once you have an image and sentence embedding, all that CLIP does is a dot product. This essentially means that both the image latents and the text latents are embedded in the same space (and that space is a good one!).

[woctezuma]

I tried it and it seems to work pretty well. (actually better than using a pretrained on classification efficientnet for example)

Just to be clear: when you say that it works well (and even "better" than another embedding), you mean that you have this feeling based on visual inspection of a few examples, or is it based on an image dataset which you had built beforehand to assess the different embeddings? I would not have any issue with any of the evaluation methods, but it is just out of curiosity (and clarity).

By the way, I am reading your nice blog post: https://rom1504.medium.com/image-embeddings-ed1b194d113e

[rom1504]

Visual inspection for now (putting a few thousands image embeddings from clip in a faiss index and trying a few queries), but I also intend to do a more extensive evaluation later on.

I know there are also image retrieval research dataset and tasks where evaluating this model would give the most reliable and comparable results.

[woctezuma]

Results are interesting, but one has to be careful because the model likes to pick text from images.

For instance, here, I suspect that images are similar because the word "story" appears.
The query is the top left picture.
In the second to last picture, there is the word "stonies".
It is only in the last picture that there is a visually similar planet shape in the same position as in the query.

Similarly, here, it is the word "empire" which seems to drive the matching between these images.

Here, the word "forest". Etc.

It is both impressive... and a bit disappointing if I want to use the model for retrieving similar images.

[scorpionsky]

Maybe that is because the words (e.g., "story") appear in the text..

[woctezuma]

Maybe that is because the words (e.g., "story") appear in the text..

There is zero text. I use the pre-trained model, and I only show the images to it.

By the way, if you want to see more results by yourself, you can refresh this page:
https://damp-brushlands-51855.herokuapp.com/render/
Be careful though, as these are Steam games, and some of them can be more aimed at a mature/adult audience.

If you find some interesting results, feel free to post about it here.

[scorpionsky]

Thanks for the clarification. Then the model might have learned attentions to the embedded text regions during the training. It is very interesting, in both of the cases.

[woctezuma]

I have added a search engine to look for specific Steam games:
https://woctezuma.github.io/steam-svelte-autocomplete/index.html

I am still having fun with this.

[thoppe]

I made an app to explore how image similarity works in CLIP by tying it to the closest images in Unsplash. Like @woctezuma noted, it seems to latch onto images with matching text, but in generally there is a lot more going on:

[htoyryla]

Anyone has any experience on using CLIP image encoding to compare two images in a loss function (instead pixel loss like MSE, perceptional loss like LPIPS or structural similar like SSIM) to guide image generation (for instance, using generating images matching both text and a given image) or, for example, VAE training. My experiments with this have not been successful, it either does not converge, or does not result in anything like the given image. I wonder if it is the case that CLIP image encoding is strictly semantic and does not contain enough visual features to properly guide training to produce images which resemble, for instance, compositionally to a given image.

EDITED to clarify that comparing two image embedding is meant.

[thoppe]

@htoyryla there is an active community on twitter trying to do just that! I found an interesting point about the text vectors vs image vectors -- they aren't colocated!

https://twitter.com/metasemantic/status/1356406256802607112