Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use chatGPT as baseline? #101

Closed
YannDubs opened this issue Jul 27, 2023 · 3 comments
Closed

Use chatGPT as baseline? #101

YannDubs opened this issue Jul 27, 2023 · 3 comments

Comments

@YannDubs
Copy link
Collaborator

the number are getting close to 100% win rate we should consider recalibrating win rates by comparing to chatGPT
@yuchenlin
Copy link

+1

@yuchenlin
Copy link

yuchenlin commented Aug 30, 2023 •
edited
Loading

although it seems that we can do that by ourselves via customizing the output json file. It would be better to have an official release of the ChatGPT references though

The references using ChatGPT: https://github.com/tatsu-lab/alpaca_eval/blob/main/results/chatgpt/model_outputs.json
So I guess we can directly set the reference to this file for using ChatGPT as the baseline

@rtaori
Copy link
Collaborator

rtaori commented Sep 4, 2023

Yup this is supported as a setting. We currently don't have plans to make chatGPT as the default, so closing this for now.

@rtaori rtaori closed this as completed Sep 4, 2023
@YannDubs YannDubs mentioned this issue Nov 13, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rtaori @yuchenlin @YannDubs