Very low performance within the first 10 epochs

[PkuRainBow]

@yuanli2333 Really impressive results with fully transformer architecture!

I have tried to reproduce the results of T2t_vit_t_14, T2t_vit_t_19, and T2t_vit_t_24 while finding their top-1 accuracy is very low within the first few epochs:

# results based on T2t_vit_t_14
epoch,train_loss,eval_loss,eval_top1,eval_top5
0,6.9310056154544535,6.97243375,0.09599999996185303,0.4740000003051758
1,6.575615681134737,6.9578225,0.104,0.512
2,6.16587602175199,6.94662625,0.116,0.5079999998855591
3,5.808463848554171,6.9398025,0.114,0.572
4,5.42472545000223,6.93104625,0.156,0.632
5,5.137583054029024,6.92024125,0.142,0.77
6,4.931810901715205,6.90476,0.194,0.8419999999809266
7,4.7973018517861,6.874345,0.246,1.044
8,4.646611140324519,6.82100625,0.358,1.554

where we can see that the top1 accuracy is only 0.358 at the 8-th epoch. I am wondering whether this result is reasonable?

Thanks!

[yuanli2333]

It is normal, we use EMA test, so it has a very low accuracy within the first several epochs.
You can see that the results without EMA test are high and normal:

[PkuRainBow]

@yuanli2333 Great thanks for your so quick reply!

I am wondering about the influence of the weights EMA (during testing) as I can not find the related comparisons in the arXiv version.

[yuanli2333]

The EMA has few effects on training (0.0%-0.3% improvement). We haven't given it in our arxiv version but will add it in the next version.