LAMBADA metric_fn inaccurate for multi-token responses

[sdtblck]

we need to check if the answer is split across multiple tokens rather than just calculating accuracy from the very last token.

should just be able to parse whether the token has a space at the beginning of the word and if not, take the previous token as well.

https://github.com/EleutherAI/GPTNeo/blob/master/model_fns.py#L255

[kevinwatkins]

My understanding was that that was how OpenAI had done it (last token rather than all the tokens of the last word), based on this remark at openai/gpt-2#131 (comment). But of course that could be my misinterpretation, or you might not want to follow them in that; it does seem rather odd.

Simplifying the procedure to test accuracy by comparing for equality of last BPE token instead of last word the accuracy is up to 46.89

[sdtblck]

Hm, we had "WutheFwasthat" from OpenAI from the discord in our server the other day, and he seemed to concur with my statement above. I think the other guy in that thread is not OpenAI affiliated.

[kevinwatkins]

Cool... I should catch up with the Discord, sorry about that

[leogao2]

We probably want this fixed asap so we can run LAMBADA on the Pile ablation

[StellaAthena]

@leogao2 @sdtblck Is this still a concern?