Skip to content

Fail turns whose model text looks like an unparsed tool call - #119

Merged
github-actions[bot] merged 1 commit into
mainfrom
fix/dangling-tool-call-false-completion
Aug 1, 2026
Merged

Fail turns whose model text looks like an unparsed tool call#119
github-actions[bot] merged 1 commit into
mainfrom
fix/dangling-tool-call-false-completion

Conversation

@mohitranka

Copy link
Copy Markdown
Contributor

Summary

  • Found in a usability audit of the OpenAI-Codex-first / Ollama-fallback golden path: a model with unreliable tool-calling (reproduced with a small local Ollama model, qwen2.5-coder:3b) can emit a JSON-ish blob naming a real tool as plain assistant text instead of a real structured tool call. Since last.tool_calls was empty, the turn fell through to TaskExpectation::ReadOnly, which completes on any non-empty final text — the header reported ✓ Completed while the target file was never touched.
  • apply_model_response now detects this case before evaluation: if the turn issued zero real tool calls and the final text contains a registered tool name in call-shaped syntax (a quoted tool name immediately followed by : or ,), the turn fails explicitly with CompletionReason::DanglingToolCallText instead of silently completing. The detection is deliberately conservative (ordinary prose that just mentions a tool name does not match) and lives in lib.rs rather than completion.rs, since that module documents itself as having no dependency on the model's own text.
  • No new CompletionReason match arms were needed elsewhere — the TUI already humanizes unknown failure categories generically (dangling_tool_call_text → "Dangling tool call text"), confirmed live.

Test plan

  • cargo test --package forge-core — 128/128, including two new regression tests: dangling_tool_call_text_does_not_complete (the repro) and prose_mentioning_a_tool_name_still_completes (false-positive guard).
  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --all-targets --locked — all clean.
  • Re-ran the exact live repro against the real release binary on Ollama/qwen2.5-coder:3b: header changed from ✓ Completed to ✗ Failed · Dangling tool call text, target file confirmed byte-for-byte unchanged on disk both before and after.

A model with unreliable tool-calling (observed with a small local Ollama
model) can emit a JSON-ish blob naming a real tool as plain assistant
text instead of a real structured tool call. Since `last.tool_calls` is
empty, the turn fell through to `TaskExpectation::ReadOnly`, which
completes on any non-empty final text — reporting "Completed" while no
file was actually touched.

Detect this case in `apply_model_response` before it reaches the
evaluator: if the turn issued zero real tool calls and the final text
contains a registered tool name in call-shaped syntax (a quoted name
immediately followed by `:` or `,`), fail the turn explicitly instead.
The detection lives in lib.rs rather than completion.rs, since that
module is documented to have no dependency on the model's own text.

Adds `CompletionReason::DanglingToolCallText` and two regression tests:
the false-completion repro, and a false-positive guard for prose that
merely mentions a tool name in a sentence.
@github-actions
github-actions Bot enabled auto-merge (squash) August 1, 2026 12:40
@github-actions
github-actions Bot merged commit 3bdc70d into main Aug 1, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant