Question Description
The LLM response parsing logic in rf/task_id/tests/evaluate_rubrics.py does not fully account for Markdown code fences appearing inside JSON string values.
The parser currently uses logic like:
if "```json" in text:
after = text[text.find("```json") + 7:]
end = after.find("```")
if end != -1:
text = after[:end].strip()
This assumes that the first closing-code-fence sequence after the outer ```json marker terminates the JSON block. However, an LLM may include a fenced code example inside a JSON string, especially in a justification field.
For example:
```json
{
"ratings": [
{
"status": "YES",
"justification": "The diff shows that a new `CannotSetMailbox` exception class is created in `app/user_settings.py` with a `msg` attribute, following the exact same pattern as the existing `CannotSetAlias` class. The code added is:\n\n```python\nclass CannotSetMailbox(Exception):\n def **init**(self, msg: str):\n self.msg = msg\n```\n\nThis mirrors the existing `CannotSetAlias` class structure which has the same `**init**` method signature and `msg` attribute assignment."
}
]
}```
In this case, after.find("```") finds the ```python fence inside the justification value instead of the closing fence surrounding the JSON response. The extracted JSON is therefore truncated and cannot be parsed.
Impact
The JSON parsing fails even though the LLM response contains a usable ratings object. As a result, the response may be skipped and the evaluation is not performed.
Suggested fix
The simplest fix is to use end = after.rfind("```") so that the outermost closing code fence is used to extract the complete response.
Question Description
The LLM response parsing logic in
rf/task_id/tests/evaluate_rubrics.pydoes not fully account for Markdown code fences appearing inside JSON string values.The parser currently uses logic like:
This assumes that the first closing-code-fence sequence after the outer ```json marker terminates the JSON block. However, an LLM may include a fenced code example inside a JSON string, especially in a justification field.
For example:
In this case, after.find("```") finds the ```python fence inside the justification value instead of the closing fence surrounding the JSON response. The extracted JSON is therefore truncated and cannot be parsed.
Impact
The JSON parsing fails even though the LLM response contains a usable ratings object. As a result, the response may be skipped and the evaluation is not performed.
Suggested fix
The simplest fix is to use end = after.rfind("```") so that the outermost closing code fence is used to extract the complete response.