A question about the computation of loss

[liuyuan-pal]

Hi Jiahui, I'm trying to reproduce the results on YFCC.
I have a question about the computation of loss.
I find that the essential loss is used after 20k steps and only those essential loss less than 0.1 is used in the backward. (https://github.com/zjhthu/OANet/blob/master/core/loss.py#L49 and https://github.com/zjhthu/OANet/blob/master/core/loss.py#L84) I am wondering what is the motivation behind this implementation and what will happen if we use all essential losses all the time.
Thank you for the excellent work.

[zjhthu]

It is a common practice in the robust loss function. As said in the DFE, Clamping the residuals ensures that hard problem instances in the training set do not dominate the training loss. I remember that we have tested different thresholds, 0.1 works best. Larger or smaller thresholds gives a little worse results.

[liuyuan-pal]

Thanks for your answer!

[zjhthu]

As for using essential loss only after 20k steps, this is inherited from learning-corr.