Repository navigation
v0.21.2
When we run MMLU evaluations today we do not pass the prompt templates to MMLU which causes the models to be evaluated without the proper chat templates. This can lead to models acheiving results that are suboptimal from what they can truly be. This commit addresses that issue by providing a system prompt argument to the MMLU evaluator objects. Signed-off-by: Oleg S <97077423+RobotSail@users.noreply.github.com>