i have some problem do a test

[lee2h]

I would like to test network on UCF101 after finetune in UCF101 using pretrained kinetics you provided.

i use resnet18.
I want to get the accuracy of a video unit, not clip.

i use instruction below

python main.py --root_path UCF101 --video_path jpg --annotation_path ucf101_01.json --result_path test --dataset ucf101 --model resnet --model_depth 18 --n_classes 101 --batch_size 64 --n_threads 4 --pretrain_path 18result1s/save_200.pth --no_train --no_val --test --test_subset val --n_finetune_classes 101

jpg is the same as your data directory.
18results1s/save_200.pth is pretrained on Kinetics, finetune in UCF101's network.

it make a error.

run
dataset loading [0/3783]
dataset loading [1000/3783]
dataset loading [2000/3783]
dataset loading [3000/3783]
test
test.py:42: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
inputs = Variable(inputs, volatile=True)
test.py:45: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
outputs = F.softmax(outputs)
Traceback (most recent call last):
File "main.py", line 162, in
test.test(test_loader, model, opt, test_data.class_names)
File "test.py", line 50, in test
test_results, class_names)
File "test.py", line 20, in calculate_video_results
'label': class_names[locs[i]],
KeyError: tensor(12)

KeyError : tensor(12) , i change my instruction, the number in parentheses is change, maybe.

can you help me?

[kenshohara]

Did you use pytorch 0.4?
My codes currently do not work on pytorch 0.4.
I would like to update my codes for 0.4,
but it takes more time.

[hareeshdevarakonda]

@lee2h Hope this should help you similar kind of error i got please try in this way that is zero tensor error

video_results = []
for i in range(sorted_scores.size(0)):
video_results.append({
'label': class_names[torch.Tensor.item(locs[i])],
'score': float(sorted_scores[i])
})

[lee2h]

@kenshohara @hareeshdevarakonda

i use pytorch 0.5.

maybe i found solution like yours.
i change the code to below

video_results.append({
'label': class_names[locs[i].item()],
'score': sorted_scores[i].item()
})

it maybe same as your solution.

and also, from val.json to video accuracy,

in utils/eval_ucf101.py
line 56 and 92,
i change iteritems() to items().

and i run below code

import eval_ucf101 as U
ucf=U.UCFclassification('ucf101_01.json','val.json',subset='validation',top_k=1)
ucf.evaluate()
print(ucf.hit_at_k)

the printed number is 0.015xxxx
i think it means 1.5% accuracy.
what's the problem?
i want 84.4% accuracy.

below is my val.json

val.zip

my model is resnet18, pretrained in Kinetics and finetune on UCF101 at 200 epoch in split 1.
during training and validating, validation result is 76.9% (clip) in 200 epoch.

python3 main.py --root_path UCF101 --video_path jpg --annotation_path ucf101_01.json --result_path test --dataset ucf101 --model resnet --model_depth 18 --n_classes 101 --n_threads 4 --pretrain_path 18result1s/save_200.pth --no_train --no_val --test --test_subset val --n_finetune_classes 101

i get val.json from above command.

[lee2h]

@kenshohara

i downgrade pytorch 0.5 to 0.3, restore code, and run your code.

pretraining on kinetics (you provided), finetuning on UCF101.

python3 main.py --root_path UCF101 --video_path jpg --annotation_path ucf101_01.json --result_path 18result0.3 --dataset ucf101 --model resnet --model_depth 18 --n_classes 400 --n_finetune_classes 101 --pretrain_path resnet-18-kinetics.pth --ft_begin_index 4 --resnet_shortcut A --batch_size 64 --n_threads 4 --checkpoint 10

and, i evaluate on UCF101 split 1.

python3 main.py --root_path UCF101 --video_path jpg --annotation_path ucf101_01.json --result_path result0.3 --dataset ucf101 --model resnet --model_depth 18 --n_classes 101 --n_threads 4 --pretrain_path 18result0.3/save_200.pth --no_train --no_val --test --test_subset val --n_finetune_classes 101 --resnet_shortcut A

i get val.json.
val.zip

and i run below code.

import eval_ucf101 as U
ucf=U.UCFclassification('ucf101_01.json','val.json',subset='validation',top_k=1)
ucf.evaluate()
print(ucf.hit_at_k)

printed result is 0.005286809410520751

i think, it means 0.5% accuracy.
the result is same as run on pytorch 0.5. meaningless.

the validation accuracy on 200epoch is 0.7668 (76.68%). (clip)
i want test accuracy 84.4%(video), but result accuracy is 0.5%.
what's the problem?

[lee2h]

in command, i change pretrain_path to resume_path, it works well.
i get UCF101 split 1, 82.2% video accuracy.

i will train and evaluate for another split and pytorch 0.5.
if there is no problem, i will close this issue.

thank you for your awesome work.

[zhujingsong]

If I want to train and evaluate the model on the whole UCF101 dataset. What should I do ?
I find that the split1 , split2, split3 have overlaps.
Should I continue train the model on split2 using the pretrained model on split1?
Or I continue to train the model on split2 using pretrained kinetics model, and average the performance?

[lee2h]

there is no problem in another split and in pytorch 0.4 (need some code change in above).

@zhujingsong
the latter.

thanks.

[ZHUXUHAN]

@kenshohara @hareeshdevarakonda

i use pytorch 0.5.

maybe i found solution like yours.
i change the code to below

video_results.append({
'label': class_names[locs[i].item()],
'score': sorted_scores[i].item()
})

it maybe same as your solution.

and also, from val.json to video accuracy,

in utils/eval_ucf101.py
line 56 and 92,
i change iteritems() to items().

and i run below code

import eval_ucf101 as U
ucf=U.UCFclassification('ucf101_01.json','val.json',subset='validation',top_k=1)
ucf.evaluate()
print(ucf.hit_at_k)

the printed number is 0.015xxxx
i think it means 1.5% accuracy.
what's the problem?
i want 84.4% accuracy.

below is my val.json

val.zip

my model is resnet18, pretrained in Kinetics and finetune on UCF101 at 200 epoch in split 1.
during training and validating, validation result is 76.9% (clip) in 200 epoch.

python3 main.py --root_path UCF101 --video_path jpg --annotation_path ucf101_01.json --result_path test --dataset ucf101 --model resnet --model_depth 18 --n_classes 101 --n_threads 4 --pretrain_path 18result1s/save_200.pth --no_train --no_val --test --test_subset val --n_finetune_classes 101

i get val.json from above command.

thank your answers

[Fazlik995]

Hi @kenshohara ,@lee2h

I use the same setting as you used:

python3 main.py --root_path UCF101 --video_path jpg --annotation_path ucf101_01.json --result_path test --dataset ucf101 --model resnext --model_depth 101 --n_classes 101 --n_threads 4 --resume_path results/save_200.pth --no_train --no_val --test --test_subset val --n_finetune_classes 101

my val accuracy is ~85%, video acc. ~88%. However, I supposed to get ~94%

I can not understand, what is wrong?

Could you help me, please?

Thanks