Open
Description
When the eval metric returns something other than a number for the score, the summarizer for the eval results shows NaN:
<html>
<body>
<!--StartFragment-->
model | tests | tests positive | accuracy with eval
-- | -- | -- | --
qwen2.5:3b | 20 | 20 | 100
llama3.2:1b | 20 | 20 | NaN
<!--EndFragment-->
</body>
</html>Eval metric:
<html>
<body>
<!--StartFragment-->
llama3.2:1b | Her voice resonated throughout the hall. Word: Her. | Incorporating the data .... | ❓ | The output must not provide explanations, descriptions, or reasoning and must consist solely of the applicable part-of-speech tag, "Unknown," or "CantAnswer."
-- | -- | -- | -- | --
<!--EndFragment-->
</body>
</html>