check out the implementation of ppl with strided sliding window strategy

[qiufengyuyi]

hi,i check the implementation of ppl for llm from https://huggingface.co/docs/transformers/perplexity. it uses a strided sliding window strategy to calculate ppl. But in opencompass, no windows are applied. Is that because you set de windows size as the max length, and all the input size is no more than that length?

[Ezra-Yu]

Yes, we do not use windows, we use the tire inputs.

[tonysy]

Would you like to give an example on how to evaluate a generation task with PPL?

[qiufengyuyi]

examples from huggingface ,or writing a fiction/poet/story when there's no completely "right" reference

[tonysy]

So you just want to calculate the PPL without 'real' answer matching process?

[qiufengyuyi]

yes, there should be some conditions that we do not have answers ,just need to evaluate the quality of a generation from a llm , but maybe the ppl is not the best choice for this test. just check the implementation of ppl when analyzing your code and examples from huggingface.

[tonysy]

Get it. The current implementation is designed for evaluation the Multi-Choice-Quesitons, we will add this feature into our backlog. If you are interested, welcome to contribute such an implementation into OpenCompass.