Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

log prob error when using Azure API #206

Closed
chengjl19 opened this issue Jan 12, 2024 · 3 comments
Closed

log prob error when using Azure API #206

chengjl19 opened this issue Jan 12, 2024 · 3 comments

Comments

@chengjl19
Copy link

image
I don't know if there is any bug with amazon API ？
@chengjl19 chengjl19 changed the title log prob error when using Amazon API log prob error when using Azure API Jan 12, 2024
@chengjl19
Copy link
Author

Sorry, I mean Azure API

@chengjl19
Copy link
Author

When I set logprobs=1, I found this "This model doesn't support the 'logprobs' paramter". Can you please switch the default evaluation method to others everyone can access?

@YannDubs
Copy link
Collaborator

YannDubs commented Jan 12, 2024 •
edited
Loading

Hi @chengjl19, unfortunately, Azure does not support logprobs yet but that will hopefully be fixed soon (as described at the bottom of this section) Anyone can use the new evaluator using the OpenAI API instead. If you can't use the OpenAI API, then I would suggest using AlpacaEval1 for now. You can do so by setting IS_ALPACA_EVAL_2=False in your environment. Sorry for that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@YannDubs @chengjl19