Skip to content

evaluate_rubrics.py fails to parse valid JSON when string values contain Markdown code fences #15

Description

@Refrain-NJU

Question Description

The LLM response parsing logic in rf/task_id/tests/evaluate_rubrics.py does not fully account for Markdown code fences appearing inside JSON string values.

The parser currently uses logic like:

    if "```json" in text:
        after = text[text.find("```json") + 7:]
        end = after.find("```")
        if end != -1:
            text = after[:end].strip()

This assumes that the first closing-code-fence sequence after the outer ```json marker terminates the JSON block. However, an LLM may include a fenced code example inside a JSON string, especially in a justification field.
For example:

```json
{
  "ratings": [
    {
      "status": "YES",
      "justification": "The diff shows that a new `CannotSetMailbox` exception class is created in `app/user_settings.py` with a `msg` attribute, following the exact same pattern as the existing `CannotSetAlias` class. The code added is:\n\n```python\nclass CannotSetMailbox(Exception):\n    def **init**(self, msg: str):\n        self.msg = msg\n```\n\nThis mirrors the existing `CannotSetAlias` class structure which has the same `**init**` method signature and `msg` attribute assignment."
    }
  ]
}```

In this case, after.find("```") finds the ```python fence inside the justification value instead of the closing fence surrounding the JSON response. The extracted JSON is therefore truncated and cannot be parsed.

Impact

The JSON parsing fails even though the LLM response contains a usable ratings object. As a result, the response may be skipped and the evaluation is not performed.

Suggested fix

The simplest fix is to use end = after.rfind("```") so that the outermost closing code fence is used to extract the complete response.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions