CAR scenarios are currently too brittle in evaluation. The scorer should evaluate whether the model selected the correct response mode (response, clarification, or abstain) and whether the answer includes the required grounding terms, instead of relying on full exact string matching.
There is also an answer parsing issue where explanatory parenthesized prose before the final JSON can be parsed instead of the actual JSON answer.
Expected behavior:
- Load CAR-specific evaluation metadata from
groundtruth_eval.json.
- Pass CAR scenarios when the model returns exactly one valid mode key, the mode matches the expected mode, and at least one required term is present.
- Keep optional terms diagnostic only.
- Prefer final JSON objects/lists over earlier parenthesized prose during parsing.
CAR scenarios are currently too brittle in evaluation. The scorer should evaluate whether the model selected the correct response mode (
response,clarification, orabstain) and whether the answer includes the required grounding terms, instead of relying on full exact string matching.There is also an answer parsing issue where explanatory parenthesized prose before the final JSON can be parsed instead of the actual JSON answer.
Expected behavior:
groundtruth_eval.json.