Skip to content

Text-only model image rejection can poison managed-agent channel queue#4563

Description

@LordMelkor

馃 A managed agent using a text-only endpoint can get stuck when its conversation history contains an image returned by a tool.

Reproduction

  1. Configure buzz-agent with a text-only OpenAI-compatible model endpoint.
  2. Run a tool that returns ToolResultContent::Image so the image is retained in the session history.
  3. Send the next model request.
  4. The endpoint rejects the request with HTTP 400, for example: Image input is not supported for this endpoint.

Current behavior

  • Provider serializers replay the tool-result image as binary/base64 input.
  • The failed turn remains associated with the ACP session.
  • buzz-acp treats the application error as retryable, requeues the batch at the front of the channel queue, and applies channel-wide backoff.
  • Later text-only events queue behind the deterministic failure and can share its retry/dead-letter fate.

Expected behavior

  • Preserve the image and accompanying text/metadata in session history.
  • When an endpoint proves it cannot accept images, retry the request once with a textual image placeholder instead of binary image data.
  • Remember that negative capability for the exact effective model for the process lifetime, without persisting it or inferring it from model names.
  • Do not let the deterministic unsupported-image error consume the queue retry budget or block later text events.

Proposed scope

  • Add a narrow provider-error classifier for the observed HTTP 400 signature.
  • Degrade images only while serializing requests for a model that has returned that signature.
  • Keep multimodal behavior unchanged for unknown and image-capable models.
  • Add an ACP-side non-retryable fallback with session invalidation and a user-visible notice for defense in depth.

Related work: #3300, #3223, #4477, and #4127.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions