New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

alpaca_eval_gpt4_turbo as evaluator quality #163

Closed

nlpcat opened this issue Nov 14, 2023 · 1 comment

nlpcat commented Nov 14, 2023

have we run any study on the human agreement if we use alpaca_eval_gpt4_turbo as the evaluator config? thanks

Collaborator

YannDubs commented Nov 14, 2023

Yes, you can see alpaca_eval_gpt4_turbo_fn here: https://github.com/tatsu-lab/alpaca_eval/tree/main/src/alpaca_eval/evaluators_configs

Note that:

I only evaluated it once per example so I don't have the estimates of bias and variance
I haven't tuned the prompt

You can see that it performs pretty well, it's cheaper, and has fewer biases than GPT4 (length and list preference)

YannDubs closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment