Incorrect use of labels for GPT2?

[soumyasanyal]

Hi,

As far as I understand the model and the usage of GPT2, shouldn't the get_dummy_token function return torch.ones() * -100 instead of torch.zeros()? This is because we should be ignoring the outputs of GPT2 for these prefix inputs. Currently, it's forcing the model to predict token 0 which is the exclamation mark ("!").

Reference lines: https://github.com/rmokady/CLIP_prefix_caption/blob/main/train.py#L222-L223

Thanks!

[robertodessi]

This line sets the loss to ignore 0-index tokens, it is equivalent to the default -100 behavior of pytorch cross entropy https://github.com/rmokady/CLIP_prefix_caption/blob/main/train.py#L316

[soumyasanyal]

I see. Forgot to check there. Thanks!