Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add llm as judge in metrics #146

Merged
merged 14 commits into from
Apr 11, 2024
Merged

add llm as judge in metrics #146

merged 14 commits into from
Apr 11, 2024

Conversation

NathanHB
Copy link
Member

@NathanHB NathanHB commented Apr 4, 2024

No description provided.

@NathanHB NathanHB linked an issue Apr 10, 2024 that may be closed by this pull request
Add LLM as a judge as a metric #141
Closed
clefourrier
clefourrier reviewed Apr 10, 2024
View reviewed changes
src/lighteval/metrics/metrics.py
use_case=MetricUseCase.SUMMARIZATION,
sample_level_fn=JudgeLLM(
judge_model_name="gpt-3.5-turbo",
template_path="src/lighteval/tasks/extended/mt_bench/judge_prompts.jsonl",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to put the prompts somewhere else? maybe have a folder for the prompt examples in the future?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes probably, we are gonna move mt-bench to lighteval tasks anyway

clefourrier
clefourrier approved these changes Apr 10, 2024
View reviewed changes
Copy link
Member

@clefourrier clefourrier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM, super nice!
Looking forward to having a more modular system :)

@clefourrier clefourrier mentioned this pull request Apr 10, 2024
Open
@clefourrier clefourrier merged commit 7f6f807 into main Apr 11, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clefourrier clefourrier clefourrier approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add LLM as a judge as a metric
2 participants
@NathanHB @clefourrier