Run evals with gpt-4?

[garyrob]

The evals I am creating won't get good results on gpt-3.5. They need gpt-4. I have tried them manually with ChatGPT with the two models, and 3.5 can't do it while 4 can.

But I don't see how to run evals with gpt-4. oaieval gpt-3.5-turbo test-match works fine on my machine, but oaieval gpt-4 test-match fails with the error, "ValueError: Couldn't find model: gpt-4".

I assume the issue is that "GPT-4 is currently in a limited beta and only accessible to those who have been granted access", and I'm still on the waitlist for that.

I further assume that, even though the evals will only get good results on gpt-4 and I can't run them on gpt-4, you will still find my evals useful. (Or, you could give me gpt-4 access... :) :) )

Let me know if I'm wrong.

[garyrob]

Thinking it over, it seems pretty clear that the fact that my prompts only get good responses with GPT-4 won't be a problem for you when YOU run the evals, even if I can't use GPT-4 to run them myself. So I'll submit them anyway.

If I'm not misunderstanding something, feel free to simply close this issue. Thanks.

[vasfvitor]

Hi, I guess this answer your question

evals/.github/PULL_REQUEST_TEMPLATE.md

Line 1 in b928cd4

# Thank you for contributing an eval! ♥️

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.

[andrew-openai]

Yep, the workflow is adding PRs with just gpt-3.5-turbo performance, and we review the PR generally for quality of the prompts/evaluation logic. We'll share GPT-4 performance in that review.

[garyrob]

Thanks for all that info!