GTSRB dataset issue

[pj-ms]

According to the official website, there are two versions of GTSRB:

Version	num train images	num test images
IJCNN 2011 competition	26640	12569
Official	39209	12630

The dataset stats (Table 9, Page 39) seem to suggest it is using train set from IJCNN 2011 version but test set from official version.

Given that Official-Train = IJCNN-Train + IJCNN-Test (Source), is CLIP using IJCNN Train as train set, IJCNN Test set as val set to tune hyper-param, Official Test set as test set? Thanks!

[jongwook]

Thanks for catching this. We were not aware of the distinction and seems like we happened to use the training images from the competition and the testing images from the "official" version.

We'd expect the linear probes (for all models compared) would perform slightly better when it's trained on more images in the official version, so I assume the reported accuracies can still serve as a "baseline" in future studies. We also note that the same set of training images are used across all models, so the comparisons in the paper can still be considered "fair". The zero-shot evaluations are not affected by this, since it only uses the test split.

[pj-ms]

Thanks for the prompt response! Yeah, it is good enough to know the numbers from the competition train + official test.

Btw, as zero-shot is mentioned here, any chance I could know what Alec eventually used for those traffic signs :-D

[jongwook]

We recently uploaded the labels at https://github.com/openai/CLIP/blob/main/data/prompts.md#gtsrb

[pj-ms]

Thanks! Just found it there as well :-D