Why my reference flow downloaded model doesn't produce the right result.

[yonghenglh6]

Hi:
When I test the reference model download from website a. The flow model produce very low accuracy.
Here is my config:

args.dataset='ucf101'
args.split=1
args.modality='flow'
args.frame_path='XXXXXXX/datas/UCF101/img_output'
args.net_proto='models/ucf101/tsn_bn_inception_flow_deploy.prototxt'
args.net_weights='models/ucf101_split_1_tsn_flow_reference_bn_inception.caffemodel'
args.save_scores='flow_score2'

It produced the Final accuracy 13.278916% on UCF101 split 1.
While the only-rgb model produced the Final accuracy 86.037706%.
Are there any mistake for my config? Or any other body get the same accuracy number as mine?
Seeking for help~~

[yjxiong]

Your configs look fine. Have you followed the steps in the README, including the optical flow extraction, and assured every step succeeded?

[yonghenglh6]

Thanks for your reply.
I checked all steps, ran again and get the same wrong result again.
I'm going to rearrange all in another environment to find the reason.

[yonghenglh6]

Hi yjxiong,
Sorry to bother you again.
My optical flow data was extracted by your dense flow program. But when I look into the result directory, the images contains data with cyclical interval of one image. Just as the picture says:

Some of the images contains data, while some of them show nothing. These two kinds appear periodically.
Forgive my ignorance of optical flow methods.
Is this normal? Or may some of my environment have faults?

[yjxiong]

Are you using the script scripts/extraction_optical_flow.sh? Would you please also provide the video name of frames in your question?

[yonghenglh6]

Yes, I used the script you mentioned, "scripts/extraction_optical_flow.sh".
The first image sequence I posted comes from v_ApplyEyeMakeup_g01_c01, but the second losts the source name.
I changed to a new environment,rearranged all, and ran the program again, but still got the same wrong result.
Thank you anyway. I'll try another database.

[yjxiong]

Your flow frames indeed look different from what I got. Here I have the same first sequence for your references. Just to note that the flow model is trained with TVL1 optical flow, which is provided by the extract_flow_gpu tool in DenseFlow. It needs the OpenCV to be compiled with GPU support.

Please have a look at whether you are using the TVL1 algorithm for flow extraction.

[yonghenglh6]

Maybe that is the reason.
I modified the script to "extract_flow_cpu" before. Because I found using cpu with 24 num_worker is much faster than using gpu, and by my common sense the extracted result would not be changed.
Now I changed back. And when it is done, I will report again.

[yjxiong]

Good to know the reason.

Yes, you can train your own models using other optical flow algorithms while the performance might be similar. But the reference model is trained with TVL1, so using other algorithms in testing will yield incompetent results.

[yonghenglh6]

I retrain the flow model with the UCF101 frames extracted by extract_flow_cpu. But it converges hardly.
The train loss is about 1.2 with 20000 iterations, and is still 0.8 with 80000 iterations.
The 20000-iteration flow model I trained got a 56% test accuracy, and the 80000-iteration flow model got a 72.01% test accuracy.
It seems the frame extracted by extract_flow_cpu is very different from that by "extracted_flow_gpu".
I wondered why you used extract_flow_gpu instead of extract_flow_cpu. The speed of extract_flow_gpu is so slow that it may take several days, while extract_flow_cpu with 24 threads takes only three hours.
Thank you. When I have more experiments, I will report again.

[yjxiong]

extract_flow_cpu has multiple flow algorithms, and the parameters in TSN project are set to work on the TVL1 algorithm.
I don't know which algorithm you are using. To our experiences, using 4 TitanX GPUs the flow extraction of UCF101 is done in 4 hours.

[yjxiong]

@yonghenglh6
Any luck with your experiments?

[yonghenglh6]

I can't get the extracted results with gpu.
I have two titan X, but when running the script, the gpu-util from nvidia-smi is zero all the time.
By the way, even using both tvl1, the results from gpu and cpu is really different.


The first picture above is by cpu, while the second is by gpu.
Thank you.
All in all, the wrong accuracy is from the difference of extracting methods.

[yjxiong]

@yonghenglh6
I just checked the DenseFlow codebase. The CPU version of extract_flow only implemented Farneback flow, which is quite different from TVL1. So the difference in extracted flow images you experienced might be caused by this.