Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for other models in AutoEval #59

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Divij97
Copy link

@Divij97 Divij97 commented Aug 4, 2023

This PR targets targets this feature request

@Divij97 Divij97 changed the title Add framework for adding new model evaluators Add support for other models in AutoEval Aug 4, 2023
@steventkrawczyk
Copy link
Contributor

Hey @Divij97 this looks great! Very elegant way to support Anthropic + OpenAI as evaluators. I'm guessing Claude and GPT will need different eval prompts, but this is definitely headed in the right direction. Let me know when this is ready for a full review

Copy link
Collaborator

@NivekT NivekT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,

Thanks for opening this PR!

We refactored how experiment.evaluate() work. The TL;DR is that .evaluate() will apply the evaluation function on a row of results at a time (plus any keyword args for the evaluation function).

At a glance, I don't think it should impact this PR but please rebase and let me know if there is any issue.

Copy link
Collaborator

@NivekT NivekT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Divij97, I have ran the CI and there are some import errors. Will you be able to rebase and have a look?

After that we should be able to merge quickly. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants