fix: [#652] warn when plan agent model refuses to tool-call#653
fix: [#652] warn when plan agent model refuses to tool-call#653anandgupta42 merged 1 commit intomainfrom
Conversation
Some models (observed: ollama-cloud/qwen3-coder-next, occasionally openai/gpt-5.4) end plan-agent steps with `finish_reason=stop` and never emit tool calls. Users see a text answer, no exploration, and abandon the session — accounting for most of the plan agent's 31% completion rate seen in 2026-04-04 telemetry. When the plan agent finishes a step with `stop` AND no tool calls have been made in the session, fire a one-shot warning: - `log.warn` with session/model/provider context - `plan_no_tool_generation` telemetry event (per-model tracking) - Synthetic text part in the TUI recommending `/model` switch The text part is `synthetic: true` so it's shown to the user but excluded from the transcript sent back to the LLM next turn. Closes #652
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)✅ Unit Test PR creation complete.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Note Unit test generation is a public access feature. Expect some limitations and changes as we gather feedback and continue to improve it. Generating unit tests... This may take up to 20 minutes. |
|
✅ Created PR with unit tests: #654 |
What does this PR do?
Warns the user when the plan agent finishes a step with
finish_reason=stopand zero tool calls in the session — catching models that refuse to tool-call in plan mode (observed onollama-cloud/qwen3-coder-next, occasionallyopenai/gpt-5.4).Signals emitted (one-shot per session):
log.warnwith session/model/provider contextplan_no_tool_generationtelemetry event (new type) — per-model abandonment tracking/modelrecommendationGuards:
planNoToolWarningEmittedflag)agent="plan",finish_reason="stop",sessionToolCallsMade===0tool-calls,length,error) don't triggerbuilder,analyst,explore,general) don't triggerThe text part is
synthetic: true— shown to the user in the TUI but excluded from the transcript replayed to the LLM next turn (existing filter atprompt.ts:648, 795).Type of change
Issue for this PR
Closes #652
How did you verify your code works?
config.tserrors unchanged)altimate_changemarkers around all additions)processor.test.tscovering:finish_reason=tool-callsfinish_reason=length/error/content-filter/unknownChecklist
Summary by cubic
Warns when the
planagent stops without any tool calls so users aren’t stuck and we can track model refusals. Addresses #652.agent="plan",finish_reason=stop, and no session tool calls.log.warn; sendplan_no_tool_generationtelemetry (with model/provider andtokens_output); add a synthetic TUI note suggesting/model.Written for commit 31db2ed. Summary will update on new commits.
Summary by CodeRabbit
New Features