Doubt on the prompts used for zero-shot evaluation in the notebook

[sayakpaul]

@andsteing

The OpenAI notebook has got 80 templates: https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb.

I wanted to know why all the prompts were not used in the Colab? Is it just for keeping the runtime low or are there any other reasons?

[andsteing]

Hi Sayak

Yes, using "7 best" instead of 80 prompts makes the Colab quite a bit faster.

The Colab you mention states

After the 80 templates were "locked" for the paper, we ran sequential forward
selection over the list of 80 templates. The search terminated after ensembling
7 templates and selected them in the order below.

itap of a {}.
a bad photo of the {}.
a origami {}.
a photo of the large {}.
a {} in a video game.
art of the {}.
a photo of the small {}.

Do you think we should state this more clearly in the Colab?

Best, Andreas

[sayakpaul]

Yes, I think. So. That will be definitely close the knowledge gap a bit.

Closing this thread.

[andsteing]

Added the PROMPTS80 as well, updated the comments, and added code for evaluation on imagenet2012 (requires manual download though).

There is a small increase in performance when using 80 prompts as compared to only 7, but it can make the handling a bit more complicated (e.g. with 1k class names, device memory becomes an issue, and one has to iterate through the tokenized texts to embed them, which makes the code more complicated).

Results in the Colab are:

• B16B i1k 7 prompts: 71.654%
• B16B i1k 80 prompts: 71.886%
• B16B c100 7 prompts: 79.220%
• B16B c100 80 prompts: 79.245%
• L16L c100 7 prompts: 79.780%
• L16L c100 80 prompts: 80.073%

[sayakpaul]

Thanks, Andreas!