Skip to content

get_image fails with OpenAI-compatible Chat Completions endpoints #242

Description

@RussellLuo

When querying an image with an OpenAI-compatible model through LiteLLM, get_image fails after the tool call.

Reproduction

uv run openkb query "What is shown in this image?"

Configuration (.openkb/config.yaml):

model: qwen3.6-plus

Error

uv run openkb query "What is shown in this image?"

    · read_file({"path": "index.md"})
    · read_file({"path": "summaries/example-doc"})
    · read_file({"path": "summaries/example-doc.md"})
    · read_file({"path": "sources/example-doc.md"})
    · get_image({"image_path": "sources/images/example-doc/img_001.p…

[ERROR] Query failed: litellm.BadRequestError: OpenAIException - data: {"error":{"code":"invalid_parameter_error","param":null,"message":"if content is list. item must be dict and key[type] should in dict","type":"invalid_request_error"},"id":"chatcmpl-6f17a300-0a94-937e-90e2-f5ebb9274f31"}

Cause

ToolOutputImage is converted into an image content part inside a role: "tool" message. Chat Completions only supports text content for tool messages, although images are supported in user messages.

Proposed solution

For Chat Completions, OpenKB could return a text tool result and move the image into a following multimodal user message. This would preserve image understanding while remaining compatible with OpenAI-compatible providers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions