view tool does not work for Anthropic/BYOM

[nirmalkumarr12]

We just integrated with the 0.2.2 package and tested view tool, it works for open ai byom models but not for anthropic byom ?

ANTHROPIC (Sonnet46) — view tool result in LLM request:

Plain Text

{
"type": "tool_result",
"tool_use_id": "toolu_01U5dgfBwh9z78k3TTc3WhVm",
"content": "Viewed image file successfully."
}
OPENAI (GPT5Reasoning) — view tool result in LLM request:

Plain Text

{
"role": "tool",
"tool_call_id": "call_j9Jg22DdmchwzBPxaw9f3EgI",
"content": [
"Viewed image file successfully.",
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgo..."
}
}
]

[github-actions]

Bug Investigation: view tool image data missing for Anthropic BYOM

Verdict: ✅ Confirmed Bug

What I Investigated

I reviewed the following code paths:

• nodejs/src/types.ts — ToolResultObject / ToolBinaryResult / convertMcpCallToolResult
• nodejs/src/client.ts — normalizeToolResultV2 (how the SDK packages tool results before sending to CLI)
• dotnet/src/Types.cs — .NET ToolBinaryResult / BinaryResultsForLlm
• dotnet/test/ToolsTests.cs — existing binary result tests
• go/types.go / go/definetool.go — Go SDK binary result handling

What I Found

The SDK correctly supports binary/image results via binaryResultsForLlm on ToolResultObject. When a tool returns image data, the SDK serializes it over JSON-RPC to the Copilot CLI server, which is then responsible for formatting the tool result according to the target LLM provider's native API.

The discrepancy you're seeing lies in the CLI server, not the SDK itself:

Provider	Tool result format sent to LLM	Image included?
OpenAI	`{"role":"tool","content":[text, {"type":"image_url","image_url":{"url":"(redacted)	✅ Yes
Anthropic	{"type":"tool_result","tool_use_id":"...","content":"Viewed image file successfully."}	❌ No

For Anthropic, tool results containing images should use a structured content array:

{
  "type": "tool_result",
  "tool_use_id": "...",
  "content": [
    { "type": "text", "text": "Viewed image file successfully." },
    { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "..." } }
  ]
}

Instead, the CLI is collapsing the result to a plain string, discarding the image data entirely.

Corroborating Evidence

There is also a skipped test in the .NET SDK (dotnet/test/ToolsTests.cs:220) named Can_Return_Binary_Result, explicitly skipped with the comment:

"Behaves as if no content was in the result. Likely that binary results aren't fully implemented yet."

This confirms the same underlying problem: the CLI server is not properly forwarding binaryResultsForLlm to Anthropic models, regardless of whether the image comes from the built-in view tool or from a custom user tool.

Root Cause

The Copilot CLI server is not applying the Anthropic-specific image content formatting when constructing tool_result messages for the LLM. The fix needs to happen in the CLI server to produce an array-form content field (matching Anthropic's API spec) when binary results are present, mirroring what it already does correctly for OpenAI.

Labeling this as bug.

Generated by Bug Handler for issue #1070 · ● 564K · ◷