When querying an image with an OpenAI-compatible model through LiteLLM, get_image fails after the tool call.
Reproduction
uv run openkb query "What is shown in this image?"
Configuration (.openkb/config.yaml):
Error
uv run openkb query "What is shown in this image?"
· read_file({"path": "index.md"})
· read_file({"path": "summaries/example-doc"})
· read_file({"path": "summaries/example-doc.md"})
· read_file({"path": "sources/example-doc.md"})
· get_image({"image_path": "sources/images/example-doc/img_001.p…
[ERROR] Query failed: litellm.BadRequestError: OpenAIException - data: {"error":{"code":"invalid_parameter_error","param":null,"message":"if content is list. item must be dict and key[type] should in dict","type":"invalid_request_error"},"id":"chatcmpl-6f17a300-0a94-937e-90e2-f5ebb9274f31"}
Cause
ToolOutputImage is converted into an image content part inside a role: "tool" message. Chat Completions only supports text content for tool messages, although images are supported in user messages.
Proposed solution
For Chat Completions, OpenKB could return a text tool result and move the image into a following multimodal user message. This would preserve image understanding while remaining compatible with OpenAI-compatible providers.
When querying an image with an OpenAI-compatible model through LiteLLM,
get_imagefails after the tool call.Reproduction
uv run openkb query "What is shown in this image?"Configuration (
.openkb/config.yaml):Error
Cause
ToolOutputImageis converted into an image content part inside arole: "tool"message. Chat Completions only supports text content for tool messages, although images are supported in user messages.Proposed solution
For Chat Completions, OpenKB could return a text tool result and move the image into a following multimodal user message. This would preserve image understanding while remaining compatible with OpenAI-compatible providers.