can not test the reported accuracy on UCF-101

[xiaojieli0903]

Hi, Thanks for sharing the code! I have tested the model on UCF-101 val dataset (split-1) you provided in https://drive.google.com/file/d/1MSyon6fPpKz7oqD6WDGPFK4k_Rbyb6fw/view?usp=sharing. But the accuracy is less than 70%. How can I get the 91.3 accuracy you reported in model_zoo?

[xiaojieli0903]

this is my test log. I didn't use the multi-scale crop because it get the same low accuracy

[yztongzhan]

Hi @xiaojieli0903! Please post your eval script here.

[xiaojieli0903]

Hi @xiaojieli0903! Please post your eval script here.

Hi, here is my script. I didn't use '--enable_deepspeed' because of the environmental problem. Also I change the 'pretrain' to 'resume'.

OUTPUT_DIR='/home/gao2/disk/work_dirs/video_mae/ucf_videomae_pretrain_base_patch16_224_frame_16x4_tube_mask_0.75_videos_e3200/eval_lr_5e-4_epoch_100_bs1-16' DATA_PATH='/home/gao2/code/mmaction2/data/ucf101/' MODEL_PATH='/home/gao2/disk/work_dirs/video_mae/ucf_videomae_pretrain_base_patch16_224_frame_16x4_tube_mask_0.75_videos_e3200/checkpoint.pth' OMP_NUM_THREADS=1 python3 -m torch.distributed.launch --nproc_per_node=1 --master_port 12322 run_class_finetuning.py --model vit_base_patch16_224 --data_path ${DATA_PATH} --resume ${MODEL_PATH} --log_dir ${OUTPUT_DIR} --output_dir ${OUTPUT_DI --data_set UCF101 --nb_classes 101 --batch_size 16 --input_size 224 --short_side_size 224 --save_ckpt_freq 50 --num_frames 16 --sampling_rate 4 --num_sample 2 --opt adamw --lr 5e-4 --warmup_lr 1e-8 --min_lr 1e-5 --layer_decay 0.7 --opt_betas 0.9 0.999 --weight_decay 0.05 --epochs 100 --test_num_segment 5 --test_num_crop 3 --fc_drop_rate 0.5 --drop_path 0.2 --use_checkpoint --dist_eval --eval

[yztongzhan]

I test the checkpoint and it works fine.

[yztongzhan]

According to your script, you used--test_num_segment 5 --test_num_crop 3 and --nproc_per_node=1. The total iteration should be 3783x5x3/16 = 3546. However, your log shows the total iteration 887.

[yztongzhan]

Hi, here is my script. I didn't use '--enable_deepspeed' because of the environmental problem. Also I change the 'pretrain' to 'resume'.

I remove the '--enable_deepspeed' and it also works fine.

[yztongzhan]

The input is raw avi videos. Please make sure your input is consistent with ours.

[xiaojieli0903]

Thanks for your help! I have gotten the correct test accuracy after changing the following items:

1. The script is here. --resume is the model path download from your model_zoo. --no_auto_resume is set.

OUTPUT_DIR='/home/gao2/disk/work_dirs/video_mae/ucf_videomae_pretrain_base_patch16_224_frame_16x4_tube_mask_0.75_videos_e3200/eval_lr_5e-4_epoch_100_bs1-16'
DATA_PATH='/home/gao2/code/mmaction2/data/ucf101/'
MODEL_PATH='/home/gao2/disk/work_dirs/video_mae/ucf_videomae_pretrain_base_patch16_224_frame_16x4_tube_mask_0.75_videos_e3200/checkpoint.pth'

OMP_NUM_THREADS=1 python3 -m torch.distributed.launch --nproc_per_node=1
--master_port 12322 run_class_finetuning.py
--model vit_base_patch16_224
--data_path ${DATA_PATH}
--log_dir ${OUTPUT_DIR}
--output_dir ${OUTPUT_DIR}
--data_set UCF101
--nb_classes 101
--batch_size 16
--input_size 224
--short_side_size 224
--save_ckpt_freq 50
--num_frames 16
--sampling_rate 4
--num_sample 2
--opt adamw
--lr 5e-4
--warmup_lr 1e-8
--min_lr 1e-5
--layer_decay 0.7
--opt_betas 0.9 0.999
--weight_decay 0.05
--epochs 100
--test_num_segment 5
--test_num_crop 3
--fc_drop_rate 0.5
--drop_path 0.2
--use_checkpoint
--dist_eval
--enable_deepspeed
--eval
--resume ${MODEL_PATH}
--no_auto_resume

2. Because there are not enough parameters in the checkpoint, it will get wrong in the line util.py Therefore, I add following codes to load the parameters from the checkpoint in line 534 of util.py.

   if args.auto_resume:
       ...
   else:
       ckpt = torch.load(args.resume)['module']
       checkpoint_load = {}
       for key in ckpt:
           checkpoint_load['module.' + key] = ckpt[key]
       model.load_state_dict(checkpoint_load)
   

[YTHmamba]

hello,Shouldn't the number of test videos be 3783, why 56745

[YTHmamba]

Hi, here is my script. I didn't use '--enable_deepspeed' because of the environmental problem. Also I change the 'pretrain' to 'resume'.

I remove the '--enable_deepspeed' and it also works fine.

hello,Shouldn't the number of test videos be 3783, why 56745

[xiaojieli0903]

According to your script, you used--test_num_segment 5 --test_num_crop 3 and --nproc_per_node=1. The total iteration should be 3783x5x3/16 = 3546. However, your log shows the total iteration 887.

@YTHmamba Hi, please see this reply. The author use the multi-crop and multi segment testing.