[Bug] MCP tool results drop structuredContent — the official computer-use plugin can't reach its own element_token

[gezi-wen]

The new experimental computer-use provider (@deepseek-ai/dsh-experimental-computer-use-cua-driver-native) ships a system prompt section that tells the model:

Use element_token from that snapshot, or coordinates from its screenshot. A new snapshot of that window invalidates its earlier element tokens.

I spent a while trying to follow that instruction and couldn't — the token never arrives. After pulling the chain apart, I think the tool-result bridge is throwing away exactly the half of the result the tool wants the model to use.

Here's what I found.

The provider calls the native driver and hands the raw MCP result to createMcpToolDefinition from @deepseek-ai/dsh-mcp-client:

const result = await activeDriver.callTool(tool.name, JSON.stringify(args), { signal: combined });
return JSON.parse(result.rawJson);

dsh-mcp-client keeps both halves in createExecutor:

const value = {
  content,
  ...result.structuredContent !== void 0 ? { structuredContent: result.structuredContent } : {}
};

but the renderer only projects content:

render(_args, value) {
  return [{ type: "text", text: extractText(value.content, rawName) }];
}

So anything an upstream tool puts in structuredContent stays invisible to the model.

That matters here, because cua-driver puts everything useful in there. I asked the native runtime directly (listToolsJson) — out of 56 tools, 34 declare an outputSchema, and get_window_state is the only one whose schema mentions tokens:

• elements[].element_token
• snapshot_id
• elements[].frame — {x, y, w, h}

Then I called it and read the raw JSON. content carries the markdown tree and the screenshot. structuredContent carries elements with 239 entries, and 239 of 239 have both element_token and frame. None of it reaches the model — in my session log the string structuredContent appears exactly zero times.

Two things this actually cost me:

1. The guidance says to use element_token, so the only route left is pixel coordinates. For an Electron window under 175% display scaling, I couldn't get a synthetic click to land at all — three attempts, no error, nothing happened. Pressing the very same button through UIA InvokePattern (from a plugin of mine, gui_uia) works on the first try, so the accessibility route itself is fine. It's just unreachable from the official provider.

2. frame is the authoritative screen rectangle. Dropping it forces coordinate math across DPI scaling and multi-monitor layouts — which is where my mis-clicks came from.

I genuinely don't know whether render() ignoring structuredContent is deliberate, maybe it's meant to be UI-only. If so, the computer-use provider has no way to opt in, and its prompt ends up asking for something the runtime can't deliver.

Would something along these lines be reasonable: when a tool declares an outputSchema, either serialize structuredContent into the model-visible text, or let a tool mark which structured fields the model should see?

Happy to test a patch if someone points me at the intended direction. Reproducible on Windows 11 with DSH 0.1.7-rc.2, and I still have the raw outputs if they'd help.

[PerryLink]

我核了桥接层——它其实保留了 structuredContent，所以丢失点在更下游

1. 桥接层的契约与实现都写着"保留"

packages/mcp/mcp-client/README.md:134（原文）：

Canonical success is { content: JsonValue[], structuredContent? }, preserving valid MCP JSON blocks for programmatic and PTC mode callers.

实现侧也对得上：

// packages/mcp/mcp-client/src/tools.ts
structuredContent?: Structured          // :41   结果类型里有这个字段
structuredContent: structuredSchema ?? {}   // :256  出参 schema 里会带上它
required: structuredSchema === undefined ? ['content'] : ['content', 'structuredContent']   // :258

⇒ 也就是说：MCP 客户端这一层是把 structuredContent 当作一等字段保留并在 schema 里声明的，并不是"拍平成一个 content 文本块"。

2. 所以你的定位需要往下一层挪

你的假设是"tool-result bridge 把工具想给模型用的那一半丢掉了"。按上面的代码，桥接层不太可能是丢弃点。更可能的是：

• 工具结果投影到模型可见内容的那一步（即决定"模型最终看到什么"的那层）；或
• 该 provider 自己在把 MCP 结果转成 ToolResult 时只取了 content（structuredContent 在它那一侧被丢），而桥接层收到的就已经是残缺的了。

⇒ 建议做一个能一刀切开的实验：在同一台机器上，用 dsh-mcp-mgr 或任意一个会回声的 MCP 服务器，让工具返回同时含 content 与 structuredContent 的最小结果，然后看：

• 模型侧能看到的工具结果里有没有那段结构；
• 会话日志里该 tool/result 事件的原始载荷里有没有。

日志里有而模型侧没有 ⇒ 丢失在"投影给模型"这一步；日志里就没有 ⇒ 丢失在 provider/桥接之前。这一步能把范围从三层压到一层，比继续读代码快得多。

3. 你这条报告的真正价值

element_token 这个例子非常好：系统提示明确要求模型使用 element_token，而它拿不到——于是模型会反复尝试（这正好解释了"卡住/重试"的观感）。建议把这一点写成影响面：不是"少了个字段"，而是"提示词让模型去用一样它收不到的东西"，这会让模型行为看起来像故障。

4. 版本提醒

⚠️ 当前最新是 0.1.7-rc.2（tag dsh-v0.1.7-rc.2）。别装 latest（npm 上它仍停在 0.1.5-rc.3，更旧）：

npm i -g @deepseek-ai/dsh@next      # 当前 = 0.1.7-rc.2

一条边界

我确认的是桥接层的契约与出参 schema 都保留 structuredContent（这足以说明"桥接层丢弃"这个归因需要修正）。实际丢在哪一层要看你第 2 项那个最小实验的结果——我没有跑 computer-use provider。