Is there certain configuration on ImageNet-A?

[kmbmjn]

Describe the bug
Thank you for the awesome library.

When I tried "validate.py" on the validation set of ImageNet with resnet50d.ra2_in1k, I obtained appropriate performance such as the following.

• Acc@1 81.362 (18.638) Acc@5 95.742 (4.258)

However, when I tried the same validation on the "ImageNet-A" dataset with resnet50d.ra2_in1k, I obtained top-1 accuracy of 0.08, which is significantly lower than your results reported in results-imagenet-a.csv.

Similarly, deit_tiny_patch16_224.fb_in1k exhibits top-1 accuracy of 0.160. I expected 7.213% as described in your results.

I just downloaded ImageNet-A dataset, and unzipped it as is, which has 200 folders that correspond to each class.

Is there a certain configuration required on ImageNet-A to obtain your results?

I attach my result below.

python validate.py --data-dir ~/imagenet-a/ --model resnet50d --pretrained
Validating in torch.float32. AMP not enabled.
Loading pretrained weights from Hugging Face hub (timm/resnet50d.ra2_in1k)
Model resnet50d created, param count: 25576264
Data processing configuration for current model + dataset:
input_size: (3, 288, 288)
interpolation: bicubic
mean: (0.485, 0.456, 0.406)
std: (0.229, 0.224, 0.225)
crop_pct: 0.95
crop_mode: center
Test: [ 0/30] Time: 3.142s (3.142s, 81.48/s) Loss: 7.2620 (7.2620) Acc@1: 0.000 ( 0.000) Acc@5: 1.172 ( 1.172)
Test: [ 10/30] Time: 0.390s (0.815s, 314.25/s) Loss: 7.4709 (7.2446) Acc@1: 0.000 ( 0.178) Acc@5: 0.000 ( 0.994)
Test: [ 20/30] Time: 0.391s (0.722s, 354.43/s) Loss: 7.7462 (7.3696) Acc@1: 0.000 ( 0.112) Acc@5: 0.000 ( 0.670)

• Acc@1 0.080 (99.920) Acc@5 0.493 (99.507)
  --result
  {
  "model": "resnet50d",
  "top1": 0.08,
  "top1_err": 99.92,
  "top5": 0.4933,
  "top5_err": 99.5067,
  "param_count": 25.58,
  "img_size": 288,
  "crop_pct": 0.95,
  "interpolation": "bicubic"
  }

To Reproduce
Steps to reproduce the behavior:
See the above.

Expected behavior
See the above.

[rwightman]

ImageNet-A is only a subset of ImageNet-1k classes so you need to remap. Also need to ensure the class labeling matches, that they are in folders with same synset names as original. In my imagenet-a.tar file that I use for eval all of the folder names are synsets... I think some versions out there might have them named 1, 2, 3 (numbers).

Run on ImageNet-A

validate.py imagenet-a.tar --class-map timm/data/_info/imagenet_a_synsets.txt --valid-labels timm/data/_info/imagenet_a_indices.txt

Run on clean ImageNet-1k but same subset of classes as ImageNet-A

validate.py /imagenet/validation --class-map timm/data/_info/imagenet_a_synsets.txt --valid-labels timm/data/_info/imagenet_a_indices.txt