Skip to content

v0.21.0: fix: provide system prompt to MMLU evaluators

Choose a tag to compare

View all tags
@relyt0925 relyt0925 released this 12 Dec 16:31
v0.21.0
0e986fe 
When we run MMLU evaluations today we do not pass the prompt templates to MMLU
which causes the models to be evaluated without the proper chat templates.
This can lead to models acheiving results that are suboptimal from what they
can truly be. This commit addresses that issue by providing a system prompt
argument to the MMLU evaluator objects.

Signed-off-by: Oleg S <97077423+RobotSail@users.noreply.github.com>
Assets 2
Loading