How to get chatgpt scores on ppl tasks #462
-
|
I think we can not get the logprobs from models like gpt-3.5-turbo/GPT4, so how do you eval those ppl tasks. |
Beta Was this translation helpful? Give feedback.
Answered by
Leymore
Oct 9, 2023
Replies: 1 comment 1 reply
-
|
In fact, we indeed cannot evaluate the PPL task on API models like ChatGPT. For API models, we always use the Gen evaluation method. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
thanks a lot! |
Beta Was this translation helpful? Give feedback.
Answer selected by
RanchiZhao
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In fact, we indeed cannot evaluate the PPL task on API models like ChatGPT. For API models, we always use the Gen evaluation method.