Loss changes little (~4.5) and Prec@1 keeps extremely low (~2%) when training with flow images

[yjzhux]

Hi. The code work well with rgb frames after I made some changes. However, it met some mistakes when training with flow imges. The loss and Prec@1 seemd to keep unchanged.

I ran this code on 4 GPUs and the batch size is 224. I set the learning rate to the initial LR (0.005) decayed by 10 every 30 epochs. The new_length was set as 5 and in_channels was change to be 10 (5*2). The flow images were computed with OpenCV and saved as '.jpg':
flow = cv2.calcOpticalFlowFarneback(prevGray, nextGray, None, 0.5, 3, 15, 3, 5, 1.2, 0)
flow_x = cv2.normalize(flow[..., 0],None,0,255,cv2.NORM_MINMAX)
flow_y = cv2.normalize(flow[..., 1],None,0,255,cv2.NORM_MINMAX)

The following figure shows the results at the beginning.

And the following figure shows the results after 30 epochs.

I tried vgg16 and inception_v3 (pre-trained model / training from scratch). Also I tried different initial LR, from 0.005 to 0.0001. Same issue. It's weird. Does anyone have comments about this?

[bryanyzhu]

Hi, thanks for bringing up the issue. Actually I think it is because the pre-processing of optical flow images.

The default preprocessing of PyTorch for natural color images is:

clip_mean = [0.485, 0.456, 0.406] * args.new_length
clip_std = [0.229, 0.224, 0.225] * args.new_length

But for optical flow, I haven't found an effective way to normalize it in order to use the pre-trained models. Currently, i use 0.5 because most people use 0.5 for gray scale images. But it doesn't work.

I will debug into it later. Or you can make progress on this issue. Thanks.

[yjzhux]

@bryanyzhu Oh, I just saw your reply. Sorry about this.
I tried different mean and std values, but it didn't help. Do you make any progress?
Can you share your results (spatial and temporal) of this code on UCF101?

[bryanyzhu]

@yanjunzhu Hi, i just updated the codebase and report the accuracy on both spatial and temporal model on UCF101. Hopefully you can train your flow models now. But as I mentioned in README, my flow model's accuracy is still 5% lower than it reported in the paper. So maybe there still exist some subtle problems. Welcome any comments and pull request. Thank you.

[yjzhux]

@bryanyzhu Thanks. I will have a try later.