About UCF101 and HMDB51 results

[airsplay]

Dear Authors,

Thanks for this great repo for reproducing the results in TimesFormer. I just want to have a quick check whether you have experimented with the two small video classification datasets (i.e., UCF101 and HMDB51) and have some initial results.

[gberta]

We haven't experimented with UCF or HMDB datasets. However, if you do so please let me know the numbers (both from Imagenet and from Kinetics pretrained checkpoints). It would be great to know how TimeSformer performs on these smaller datasets.

My intuition is that it might not work as well as some prior methods. Our model has a large number of parameters, so these datasets might be too small for the model to learn meaningful patterns. Nevertheless, it would quite interesting to see some results in this setting.

[airsplay]

Thanks for the detailed answers! We will let you know if we have the results.

[PeihaoChen]

Hi,

I have tried timesformer on UCF101. With ImageNet pretrained, I only get 42.85% accuracy.

The timesformer args are as followed: "transformer_args": { "depth": 12, "dim_head": 64, "embed_dim": 768, "heads": 12, "patch_size": 16 }. I trained the model for 30 epoch with learning rate of 0.05 using a cosine annealing schedule and SGD optimizer. The input containing 16 frames with 1/4 sampling rate for 25fps video. The image resolution is 224*224. I trained on 7 GPUs with batch size 2 for each GPU.

I know that the Transformer0based model is not easy to train on small datasets. I am not sure whether this result is normal. Do I miss some important training details? On VidTr paper, the authors achieve 96.7 on UCF101 with Transformer-based model.

Looking for an insightful response that helps me to get a reasonable result on UCF101. Many thanks!

[h030162]

https://aistudio.baidu.com/aistudio/projectdetail/2291410
avg_acc1= 0.9243986010551453, avg_acc5= 0.9894263744354248