-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem of oaieval.py #243
Comments
I'm currently having the same problem. I'm unsure of how to clear the cache to get it to "see" my new jsonl file. I've tried to change the name of the test to no avail EDIT: After running with --debug, I found it was loading a |
The behaviour is explained in custom-eval.md:
This will be a difficult spot for people looking through custom-eval I think. I've also noticed that the flag Lines 173 to 175 in 19bfdba
Though of course @andrew-openai will know a lot more than I do! |
Thanks, it helps.
|
The oaieval.py still preformed the very first version of the dataset after I updated the jsonl file as the program only execute one item while I have three in my dataset.
It looks like the first version had been cached, cause the content in log file created by record.py also didn't the match the dataset, but the content of the first version of my dataset (which means the log file contain content various from dataset).
How can I help with this?
The text was updated successfully, but these errors were encountered: