Replace pad_token with -100 for LM loss calculation

[setu4993]

The docs for both GPT and GPT2 specify that labels that are not -100 will be used for the calculation of the loss. So, the padding for the labels should be -100, not tokenizer.pad_token_id.

[codecov-commenter]

Codecov Report

Merging #4718 into master will decrease coverage by 0.64%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4718      +/-   ##
==========================================
- Coverage   77.10%   76.45%   -0.65%     
==========================================
  Files         128      128              
  Lines       21723    21725       +2     
==========================================
- Hits        16749    16610     -139     
- Misses       4974     5115     +141     

Impacted Files	Coverage Δ
src/transformers/data/data_collator.py	89.55% <100.00%> (+0.32%)	⬆️
src/transformers/modeling_tf_pytorch_utils.py	8.72% <0.00%> (-81.21%)	⬇️
src/transformers/modeling_ctrl.py	96.56% <0.00%> (-2.58%)	⬇️
src/transformers/modeling_openai.py	79.51% <0.00%> (-1.39%)	⬇️
src/transformers/trainer.py	38.08% <0.00%> (-1.17%)	⬇️
src/transformers/modeling_tf_utils.py	86.49% <0.00%> (-0.78%)	⬇️
src/transformers/benchmark/benchmark_utils.py	72.80% <0.00%> (-0.30%)	⬇️
src/transformers/file_utils.py	73.79% <0.00%> (ø)
src/transformers/modeling_utils.py	90.61% <0.00%> (+0.11%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e80d6c6...6cceabd. Read the comment docs.

[LysandreJik]

This is correct! Thanks @setu4993!

[setu4993]

Closed the PR by mistake... Re-opening it.

[julien-c]

Just a quick question for @mfuntowicz, is copy.deepcopy a performant way to clone a tensor? (given that this is called at each training step)

[setu4993]

I saw deepcopy being used elsewhere in the code, so added it as is. Just looking at the new tensor documentation, and they recommend using .clone().detach(). Happy to change it to that.

[setu4993]

Bumping this since I haven't seen any activity in a few days.

[julien-c]

Yes .clone().detach() sounds good.

[julien-c]

LGTM but let's let @LysandreJik have a last check and merge this

[setu4993]

Thanks @julien-c!

[setu4993]

Hey @LysandreJik, can you please review when you have a chance? Thanks!

[LysandreJik]

Thanks @setu4993!