Silent success: failed tool chain still ends with reason.kind = completed and headless exit 0 #4410
Replies: 1 comment
|
Verified against rc.2 (HEAD b150a55) — your framing is right, and it's a distinct layer from #4370. The mechanism is fully traceable, and the root gap is in the turn-end taxonomy itself: a "completed" reason is emitted whenever the model stops calling tools, with no distinction between a genuine answer and a surrender after an error chain. 1. The verified chain (rc.2)
2. The taxonomy gap (the layer worth its own fix)
3. Bounding the fix — don't key on "any isError"A blunt rule ("any failed tool call in the turn → error reason") would false-positive on legitimate recovery turns: a model retries a failed call and succeeds, or routes around a flaky tool — those are successful turns with failed calls. The discriminating signal is the terminal character of the chain: the final step before turn-end produced only Good call separating it from the wire fix — this one is about observability of outcome, not recovery from poison. |
Uh oh!
There was an error while loading. Please reload this page.
After a ToolNotFoundError loop gives up, the turn still ends with reason.kind = "completed" and headless exits 0 — a failed agent run presented as success.
This is independent of the wire-level tool-call bug (thread #4370): the wire fix stops the loop, but does not stop the masking. A tool chain that errored to exhaustion should surface as error (or at least a non-zero headless exit), not completed.
Worth its own fix rather than riding along.
All reactions