Guide to fixed-length model perplexity evaluation

[joeddav]

This post / guide is inspired by this recent Twitter discussion and this gist on the different ways that perplexity can be evaluated and the optimal strategy of a strided "sliding window".

Interested in feedback both on the guide/writing component as well as the theoretical discussion on PPL. Right now my understanding is that our language modeling script uses non-overlapping segments rather than the sliding window.

Relevant to #4415, #4219.

[codecov]

Codecov Report

Merging #5449 into master will increase coverage by 0.86%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5449      +/-   ##
==========================================
+ Coverage   77.82%   78.68%   +0.86%     
==========================================
  Files         141      141              
  Lines       24608    24608              
==========================================
+ Hits        19150    19364     +214     
+ Misses       5458     5244     -214     

Impacted Files	Coverage Δ
src/transformers/generation_utils.py	97.10% <0.00%> (+0.28%)	⬆️
src/transformers/generation_tf_utils.py	86.68% <0.00%> (+0.50%)	⬆️
src/transformers/modeling_openai.py	81.09% <0.00%> (+1.37%)	⬆️
src/transformers/modeling_tf_openai.py	94.98% <0.00%> (+74.19%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d16e36c...b3dae20. Read the comment docs.

[sgugger]

Left some comments but I really like this, and the Research section is exactly right for that kind of docs!