[BUG] apply_diff uses too aggressive HTML unescaping preventing searching and changing some code

[rkfg]

Problem (one or two sentences)

Sequences like &, <, > are transparently replaced with &, >, <, both in search and replace. Because of that changing or writing HTML-escaping code becomes impossible.

Context (who is affected and when)

Simple HTML escaping in Go becomes impossible to write, strings.ReplaceAll(htmlText, "&", "&") becomes strings.ReplaceAll(htmlText, "&", "&") which is useless.

Reproduction steps

Here's the tool use stanza:

{
        "type": "tool_use",
        "id": "E2nCKFzolIQ5bNGNWFmwc9b7aSZdQmgI",
        "name": "apply_diff",
        "input": {
          "path": "search/search.go",
          "diff": "<<<<<<< SEARCH\n:start_line:336\n-------\n\tif r.Text != \"\" {\n\t\ttruncated := truncate(r.Text, 200)\n\t\tfmt.Fprintf(&textLine, \"\\n%s\\n\", truncated)\n\t\t// Escape HTML special chars\n\t\thtmlText := truncated\n\t\thtmlText = strings.ReplaceAll(htmlText, \"&\", \"&amp;\")\n\t\thtmlText = strings.ReplaceAll(htmlText, \"<\", \"&lt;\")\n\t\thtmlText = strings.ReplaceAll(htmlText, \">\", \"&gt;\")\n\t\tfmt.Fprintf(&htmlLine, \"<br>%s<br>\", htmlText)\n\t}\n\tif r.ImageDesc != \"\" {\n\t\ttruncated := truncate(r.ImageDesc, 200)\n\t\tfmt.Fprintf(&textLine, \"\\n\\U0001f574\\ufe0f %s\", truncated)\n\t\tfmt.Fprintf(&htmlLine, \"<br>\\U0001f574\\ufe0f %s\", truncated)\n\t}\n=======\n\tif r.Text != \"\" {\n\t\ttruncated := truncate(r.Text, 200)\n\t\tfmt.Fprintf(&textLine, \"\\n%s\\n\", truncated)\n\t\t// Escape HTML special chars and convert newlines to <br>\n\t\thtmlText := truncated\n\t\thtmlText = strings.ReplaceAll(htmlText, \"&\", \"&amp;\")\n\t\thtmlText = strings.ReplaceAll(htmlText, \"<\", \"&lt;\")\n\t\thtmlText = strings.ReplaceAll(htmlText, \">\", \"&gt;\")\n\t\thtmlText = strings.ReplaceAll(htmlText, \"\\n\", \"<br>\")\n\t\tfmt.Fprintf(&htmlLine, \"<br>%s<br>\", htmlText)\n\t}\n\tif r.ImageDesc != \"\" {\n\t\ttruncated := truncate(r.ImageDesc, 200)\n\t\tfmt.Fprintf(&textLine, \"\\n\\U0001f574\\ufe0f %s\", truncated)\n\t\t// Escape HTML special chars and convert newlines to <br>\n\t\thtmlDesc := truncated\n\t\thtmlDesc = strings.ReplaceAll(htmlDesc, \"&\", \"&amp;\")\n\t\thtmlDesc = strings.ReplaceAll(htmlDesc, \"<\", \"&lt;\")\n\t\thtmlDesc = strings.ReplaceAll(htmlDesc, \">\", \"&gt;\")\n\t\thtmlDesc = strings.ReplaceAll(htmlDesc, \"\\n\", \"<br>\")\n\t\tfmt.Fprintf(&htmlLine, \"<br>\\U0001f574\\ufe0f %s\", htmlDesc)\n\t}\n>>>>>>> REPLACE"
        }
      }

The model uses correct code here as HTML entities should not be unescaped in JSON context. However, Roo errors out with:

{
        "type": "tool_result",
        "tool_use_id": "E2nCKFzolIQ5bNGNWFmwc9b7aSZdQmgI",
        "content": "<error_details>\nNo sufficiently similar match found at line: 336 (97% similar, needs 100%)\n\nDebug Info:\n- Similarity Score: 97%\n- Required Threshold: 100%\n- Search Range: starting at line 336\n- Tried both standard and aggressive line number stripping\n- Tip: Use the read_file tool to get the latest content of the file before attempting to use the apply_diff tool again, as the file content may have changed\n\nSearch Content:\n\tif r.Text != \"\" {\n\t\ttruncated := truncate(r.Text, 200)\n\t\tfmt.Fprintf(&textLine, \"\\n%s\\n\", truncated)\n\t\t// Escape HTML special chars\n\t\thtmlText := truncated\n\t\thtmlText = strings.ReplaceAll(htmlText, \"&\", \"&\")\n\t\thtmlText = strings.ReplaceAll(htmlText, \"<\", \"<\")\n\t\thtmlText = strings.ReplaceAll(htmlText, \">\", \">\")\n\t\tfmt.Fprintf(&htmlLine, \"<br>%s<br>\", htmlText)\n\t}\n\tif r.ImageDesc != \"\" {\n\t\ttruncated := truncate(r.ImageDesc, 200)\n\t\tfmt.Fprintf(&textLine, \"\\n\\U0001f574\\ufe0f %s\", truncated)\n\t\tfmt.Fprintf(&htmlLine, \"<br>\\U0001f574\\ufe0f %s\", truncated)\n\t}\n\nBest Match Found:\n336 | \tif r.Text != \"\" {\n337 | \t\ttruncated := truncate(r.Text, 200)\n338 | \t\tfmt.Fprintf(&textLine, \"\\n%s\\n\", truncated)\n339 | \t\t// Escape HTML special chars\n340 | \t\thtmlText := truncated\n341 | \t\thtmlText = strings.ReplaceAll(htmlText, \"&\", \"&amp;\")\n342 | \t\thtmlText = strings.ReplaceAll(htmlText, \"<\", \"&lt;\")\n343 | \t\thtmlText = strings.ReplaceAll(htmlText, \">\", \"&gt;\")\n344 | \t\tfmt.Fprintf(&htmlLine, \"<br>%s<br>\", htmlText)\n345 | \t}\n346 | \tif r.ImageDesc != \"\" {\n347 | \t\ttruncated := truncate(r.ImageDesc, 200)

Clearly it escaped the request and couldn't find a match. Same happens if the model patches or writes a file from scratch, in both cases HTML entities are escaped when they shouldn't.

Expected result

apply_diff works as it should

Actual result

No match could be found, invocation fails

Variations tried (optional)

After that the model tried rewriting the whole file and the bug manifested itself here as well, the entities came out unescaped:

Instead of adding just one line it mangled the other lines too and I now have to change them back manually.

App Version

Version: 3.53.0 (ad25634) (plus a couple of other apply_diff related PRs from here)

API Provider (optional)

OpenAI Compatible

Model Used (optional)

Qwen 3.6 35B APEX

Roo Code Task Links (optional)

No response

Relevant logs or errors (optional)

[roomote-v0]

I've analyzed this issue and here's my proposed implementation plan:

Root cause: The unescapeHtmlEntities() function in src/utils/text-normalization.ts is called unconditionally on all file content for non-Claude models in ApplyDiffTool.ts (line 32), WriteToFileTool.ts (line 85), and ExecuteCommandTool.ts (line 56). This was originally added to handle XML-based tool calls where HTML entities were XML escaping artifacts. However, now that Roo Code exclusively uses native JSON tool calls (XML tool calls are explicitly rejected), these HTML entities in JSON arguments are almost always intentional literal strings in the code being edited -- not encoding artifacts.

The transformation is lossy and ambiguous: there is no way to distinguish & that the model intended as the literal string & (e.g. Go code doing HTML escaping) from & produced as an encoding artifact for &.

Proposed fix:

1. Remove unescapeHtmlEntities from ApplyDiffTool.ts and WriteToFileTool.ts -- these tools deal with file content where HTML entities can legitimately appear as part of the source code. Since all tool calls now go through native JSON parsing (JSON.parse in NativeToolCallParser.ts line 701), the content is already properly unescaped by the JSON parser. Any HTML entities in the parsed content are intentional.

2. Remove unescapeHtmlEntities from ExecuteCommandTool.ts -- same reasoning applies; commands are parsed from JSON and should be used as-is. Edge cases where a model incorrectly HTML-encodes a command are better handled by letting the command fail and the model retry, rather than silently mangling content.

3. Keep unescapeHtmlEntities available in text-normalization.ts (no deletion) since it may still be useful for other contexts, but remove the automatic application to tool content.

4. Update existing tests in writeToFileTool.spec.ts, executeCommandTool.spec.ts, and add/update tests for ApplyDiffTool to reflect the new behavior.

This approach fixes the reported bug where code containing HTML entities (like strings.ReplaceAll(htmlText, "&", "&")) gets mangled, while accepting that rare model-level encoding bugs should be handled at the model/provider level rather than with lossy post-processing.

Please mention @roomote in a comment saying that you approve this plan, and I'll proceed with the implementation.

[rkfg]

@roomote approved!

[roomote-v0]

I've seen the approval and will proceed with the implementation plan. I'll remove the unescapeHtmlEntities calls from ApplyDiffTool.ts, WriteToFileTool.ts, and ExecuteCommandTool.ts since native JSON tool calls already handle unescaping properly. I'll open a PR to share progress shortly.

[roomote-v0]

Opened PR #12265 with my attempt at addressing this. The fix removes unescapeHtmlEntities() calls from ApplyDiffTool, WriteToFileTool, and ExecuteCommandTool since native JSON tool calls already handle unescaping -- any HTML entities in the parsed content are intentional. Feedback welcome!

#12265