define unknown category?

[batrlatom]

Hi,
would it be possible to construct a prompt in the way, that it recognizes that the image does not belong to the selected categories?
For example, my prompt will have classes 'dog' and 'cat' and I will predict images of car. In this case, it will predict the category 'unknown' ( or any other category ) ?

[jongwook]

Hi! CLIP does not explicitly support "unknown" category out-of-the-box, but I can imaging two alternative approaches

1. You can set a threshold in terms of cosine similarity and classify an image as "unknown" if its feature is not within the threshold of either the "dog" or "cat" vector. This will effectively create a high-dimensional cone in the embedding space for each category, and classify anything falls out of the two cones as "unknown".

To make it a bit more robust, you could also try having a collection of "cat" and "dog" vectors, by converting cat and dog images and prompt-engineering cat- and dog-describing texts, and make them vote, like in a k-NN classifier.

2. Another option is to make a collection of words NOT corresponding to cats and dogs, e.g. from all the other class names from ImageNet, and see if enough probability mass moves away to those categories.

I can also imagine using a combination of these two approaches, but in any case, it'll require some manual tuning of thresholds.

[batrlatom]

Thank you, I will try it.