Skip to content

v0.21.2

@RobotSail RobotSail tagged this 13 Dec 22:22
When we run MMLU evaluations today we do not pass the prompt templates to MMLU
which causes the models to be evaluated without the proper chat templates.
This can lead to models acheiving results that are suboptimal from what they
can truly be. This commit addresses that issue by providing a system prompt
argument to the MMLU evaluator objects.

Signed-off-by: Oleg S <97077423+RobotSail@users.noreply.github.com>
Assets 2
Loading