-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error when running request generate_until
#1310
Comments
I have the same problem with |
That aside, I've made a fix in #1315 so that you don't have to copy utils functions so you can remove the lines with |
Also, is |
But, from Open LLM Leaderboard, gsm8k is measured using |
it is not? But I got it from |
Yeah, nvm, I forgot where it was located. Anyway, the Open LLM Leaderboard uses an older version of lm-eval before the big-refactor. Currently, for |
I guess polemo2_in can't be exact_match... |
polemo2_in has accuracy which I think can also be used for gsm8k and is probably the same as acc. Lm-eval supports using metrics from Huggingface's evaluate library. |
So, in short |
@djstrong could you check if this patch works? @fahadh4ilyas |
@lintangsutawika Sorry, it works without the patch :) I haven't checked with the patch. @fahadh4ilyas Try with |
Perhaps to minimize confusion we should simply add a |
Here is my config
And here is the error
EDIT: The problem is from gsm8k task. When I tried to do that task itself, I got the same error. Here is the config
The text was updated successfully, but these errors were encountered: