fix(agent): degrade images for text-only models - #4566
Conversation
Signed-off-by: LordMelkor <kray@squareup.com> Co-authored-by: Codex <noreply@openai.com> Ai-assisted: true
wesbillman
left a comment
There was a problem hiding this comment.
Reviewing on Wes's behalf.
The underlying failure class is real: buzz-agent currently serializes every ToolResultContent::Image back into subsequent provider requests unconditionally, while buzz-acp requeues ordinary application errors ahead of later channel events. There is also an earlier independently documented/manual reproduction of the text-only media rejection in #3300/#3223 (that path reported 422 rather than this PR's exact 400 wording).
I don't think this implementation is safe to merge yet because the learned capability is not always keyed by the model that actually rejected the image. For OpenAI relay-mesh auto, effective_model remains auto, while openai_request() resolves each request to mesh or ordinary auto from the live catalog (resolve_openai_model). This PR caches the rejection under the outer effective_model, so one text-only routed backend permanently strips images from all later auto requests in this process—even after routing changes to an image-capable backend. That contradicts the stated model isolation and can silently degrade capable models.
Please key the negative capability to the actual request model/route that returned the rejection (or explicitly define/test safe virtual-model behavior), and add regression coverage for auto changing routes after one backend rejects an image. The reactive retry plus serialization-time placeholder is otherwise a reasonable general approach; inference from model names would be worse.
Co-authored-by: Codex <noreply@openai.com> Ai-assisted: true Signed-off-by: LordMelkor <kray@squareup.com>
|
🤖 @wesbillman Thanks — addressed in |
Summary
Related issue
Fixes #4563.
Related prior work: #3300, #3223, #4477, and #4127.
Testing
cargo test -p buzz-agent -p buzz-acp— 1,153 passed across 14 suitescargo fmt --all -- --checkcargo clippy -p buzz-agent -p buzz-acp --all-targets -- -D warnings -A clippy::question_markStrict Clippy is currently blocked by three pre-existing
clippy::question_markfindings in untouchedcrates/buzz-agent/src/hints.rsandcrates/buzz-acp/src/queue.rs.