acc not match(72.03 vs 75.4)

[HappyAIWalker]

It's same that the release model is not the best one.
python validate.py "MYDATA/ImageNet" --model "van_tiny" --checkpoint "weights/van_tiny_754.pth.tar" -b 200
The result is following: Acc@1 72.032 (27.968) Acc@5 91.126 (8.874)

I check the state_dict of checkpoint and found that ema version is not exist, and I think the ema version's accuracy is same with paper.

so will you check it ?

[MenghaoGuo]

Amazing.

I download the model from google drive and can get the 75.4 % Acc.

Here is my eval command

I also load the model and print key. It is same with you.

[HappyAIWalker]

it's so strange！ I tried the tiny and small, all not match. I also tried other model (vip and volo) and can got same acc as paper.
the model i tried was download from tsinghua cloud, i will try another one from google cloud later.

[MenghaoGuo]

I try to use the model downloaded from tsinghua cloud. However, i also get the 75.4 % acc.

The version of my softwares:

Pytorch == 1.7.1 (I heard it is important)
Timm == 0.4.12

[HappyAIWalker]

I try it again and find that the version of timm influence so much.
when version of timm is 0.4.9, the acc is 72.03; when I change it the latest one, the acc is 75.376(matched with your result).
so, maybe there are some difference in the preprocessing part.

[thunder95]

I have the same problem, with Pytorch == 1.7.1 and Timm == 0.4.12

Loaded state_dict from checkpoint 'van_tiny_754.pth.tar'
Model van_tiny created, param count: 4105800
Data processing configuration for current model + dataset:
input_size: (3, 224, 224)
interpolation: bicubic
mean: (0.5, 0.5, 0.5)
std: (0.5, 0.5, 0.5)
crop_pct: 0.9
Test: [ 0/20] Time: 1.760s (1.760s, 113.65/s) Loss: 0.8368 (0.8368) Acc@1: 82.500 ( 82.500) Acc@5: 98.500 ( 98.500)
Test: [ 10/20] Time: 0.253s (0.469s, 426.49/s) Loss: 1.1970 (1.1323) Acc@1: 72.500 ( 76.636) Acc@5: 93.500 ( 94.364)

• Acc@1 74.586 (25.414) Acc@5 92.786 (7.214)