Never publish a leaked tool call as a review

[tarekziade]

What happened

This review was published on #79 with a body of nothing but Kimi's raw tool-call markup and zero inline comments:

🤗 **Serge** says:

<|tool_calls_section_begin|><|tool_call_begin|>functions.read_file:6<|tool_call_argument_begin|>

<|tool_call_end|><|tool_calls_section_end|>

Root cause — three defects compounding

1. The provider leaked a tool call as text. On turn 8, moonshotai/Kimi-K2.6 via the HF Router serialized its tool call into message.content using chat-template special tokens, left message.tool_calls empty, and returned finish_reason: "stop". _run_agentic_loop took the not chat.tool_calls branch and treated it as the final answer. _needs_final_salvage didn't fire either — the content wasn't blank and the finish reason wasn't length.

2. _extract_json accepted the tool arguments as the review. Its third pass (raw_decode at every {) parsed the leaked call's own argument object, {"path": "reviewbot/reviewer.py", "start_line": 248, "end_line": 280}, so no _UnparseableLLMOutput was raised.

3. The empty-summary salvage published the markup. With summary="" and no comments, the stub-JSON fallback called _prose_outside_json, which stripped the JSON and returned the surrounding special tokens as the summary. publish_review posted it.

Reproduced byte-for-byte against the persisted job row before fixing.

Scope

The serge#79 review is the one confirmed occurrence. A first scan of the prod jobs DB also flagged a /tasks run (Kimi-K2.7-Code, ended no_fix, nothing published), but that row has since been evicted by WEB_JOB_RETENTION=25 — the table is a rolling window of the 25 most recent jobs — so it can no longer be verified. Treat the prevalence as "seen more than once on Kimi, never on another model", not as a measured rate.

The fix

Each defect is fixed at its own layer, so no single one has to hold alone.

llm_client — recover text-serialized tool calls into real ToolCall objects and strip the markup from the content, so the turn stays a tool turn and the model gets the file it asked for. Covers the streaming and buffered paths (they converge on one place). Structured tool_calls still wins whenever the provider sends them. Tolerant of a truncated tail; drops a call whose id isn't recognizable rather than inventing a tool name.

_extract_json — new optional require_any_key. The review caller passes ("summary", "comments", "event") and the task callers pass ("title", "body", "patch"), so incidental JSON can't pass as a result. Default behaviour is unchanged for any caller that doesn't opt in.

publish_review — raises the new EmptyReviewError instead of posting a review with no summary and no inline comments, and a summary that is only leaked markup counts as no summary. A markup-only summary that arrives with real inline comments still publishes, but renders the (no overall summary provided) line rather than the tokens. All three publish paths report the failure rather than swallowing it:

• webhook auto-publish → marks the job error and posts a failure comment
• run_review → posts a comment in the thread
• the UI publish endpoint → 409 empty_review, so the human can edit the summary and retry

A legitimate review that quotes a special token (<|endoftext|> while discussing a tokenizer) is deliberately left untouched — the check only rejects text with no substance left, it never rewrites text that has some.

Tests

29 new tests; 596 pass, make format clean.

• TextToolCallRecoveryTests — the exact prod payload, multiple calls, prose around the markup, truncated tail, unrecognizable id, both HTTP paths, structured-wins
• LeakedTextToolCallLoopTests — end-to-end through the real client with mocked HTTP: the leaked turn executes read_file, the output comes back as a tool message, the loop reaches a proper review
• ExtractJsonRequiredKeysTests — the leaked arguments are rejected as a review but still accepted without the filter
• ModelMarkupOnlyTests, EmptyReviewGateTests — the publish gate, including the quoted-special-token case

🤖 Generated with Claude Code