Skip to content

fix(agent): degrade images for text-only models - #4566

Open
LordMelkor wants to merge 2 commits into
block:mainfrom
LordMelkor:fix/text-only-agent-image-fallback
Open

fix(agent): degrade images for text-only models#4566
LordMelkor wants to merge 2 commits into
block:mainfrom
LordMelkor:fix/text-only-agent-image-fallback

Conversation

@LordMelkor

@LordMelkor LordMelkor commented Aug 3, 2026

Copy link
Copy Markdown

Summary

  • retry once with text placeholders when an endpoint returns the narrow unsupported-image HTTP 400 signature
  • cache that negative capability by exact resolved request model/route for the process lifetime, while preserving images in session history and leaving other routes unchanged
  • immediately dead-letter and reset the ACP session if the deterministic error escapes the built-in fallback, so later text events are not blocked behind retries
  • add provider serialization, resolved-route cache isolation, classifier, route-change, and queue-release regression coverage

Related issue

Fixes #4563.

Related prior work: #3300, #3223, #4477, and #4127.

Testing

  • cargo test -p buzz-agent -p buzz-acp — 1,153 passed across 14 suites
  • cargo fmt --all -- --check
  • cargo clippy -p buzz-agent -p buzz-acp --all-targets -- -D warnings -A clippy::question_mark

Strict Clippy is currently blocked by three pre-existing clippy::question_mark findings in untouched crates/buzz-agent/src/hints.rs and crates/buzz-acp/src/queue.rs.

Signed-off-by: LordMelkor <kray@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Ai-assisted: true
@LordMelkor
LordMelkor requested a review from a team as a code owner August 3, 2026 17:31

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing on Wes's behalf.

The underlying failure class is real: buzz-agent currently serializes every ToolResultContent::Image back into subsequent provider requests unconditionally, while buzz-acp requeues ordinary application errors ahead of later channel events. There is also an earlier independently documented/manual reproduction of the text-only media rejection in #3300/#3223 (that path reported 422 rather than this PR's exact 400 wording).

I don't think this implementation is safe to merge yet because the learned capability is not always keyed by the model that actually rejected the image. For OpenAI relay-mesh auto, effective_model remains auto, while openai_request() resolves each request to mesh or ordinary auto from the live catalog (resolve_openai_model). This PR caches the rejection under the outer effective_model, so one text-only routed backend permanently strips images from all later auto requests in this process—even after routing changes to an image-capable backend. That contradicts the stated model isolation and can silently degrade capable models.

Please key the negative capability to the actual request model/route that returned the rejection (or explicitly define/test safe virtual-model behavior), and add regression coverage for auto changing routes after one backend rejects an image. The reactive retry plus serialization-time placeholder is otherwise a reasonable general approach; inference from model names would be worse.

Co-authored-by: Codex <noreply@openai.com>
Ai-assisted: true
Signed-off-by: LordMelkor <kray@squareup.com>
@LordMelkor

LordMelkor commented Aug 4, 2026

Copy link
Copy Markdown
Author

🤖 @wesbillman Thanks — addressed in 3700126e. The negative image capability is now learned at the OpenAI dispatch boundary against the concrete resolved request model (mesh, ordinary auto, or an explicit model), including the internal meshauto fallback path. A new regression changes auto routes after mesh rejects an image and verifies that the later auto route still receives the image. Full buzz-agent/buzz-acp suite: 1,153 passed; format and scoped Clippy pass. Could you please take another look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Text-only model image rejection can poison managed-agent channel queue

2 participants