Using different loss

[vinevix]

Hi, I'm doing experiments with your code and I need to use different loss than CrossEntropy. Since you added -1 to the targets (during padding), you use ignore_index=-1 flag, how can I prevent to compute gradient on those -1 using different losses? (Of course they can't offer the ignore_index flag)

[buxiangzhiren]

You could find the index of each token which denotes ignore_index!=-1 and then keep the corresponding loss in your sentence. In other words, it is not necessary to sum the losses of all tokens in a sentence, you just sum the losses of some tokens whose ignore_index!=-1.