Skip to content

dsh-win-computer-use 0.1.1

Latest

Choose a tag to compare

@wwwort wwwort released this 19 Sep 12:56

Windows-native computer use for DeepSeek Harness. 0.1.1 closes the element-tree gap
on both sides: a whole chat round now runs with no element tree at all.

Reading is cheaper than looking

Text content is read as text, not captured as pixels:

{"op": "read", "window": "ChatGPT", "tail": 2000}                     // newest 2000 chars
{"op": "wait", "window": "ChatGPT", "state": "text_stable", "tail": 2000,
 "absent": "正在回应", "contains": "ChatGPT 说"}

Measured on the ChatGPT desktop app: one long reply came back as 2,200 characters of
quotable text in a single ~280 ms call
; the whole page is 25,722 characters through the
same path. Document reading order puts the newest content last and the app's own chrome
first, so a tail read is clean by construction — and it never touches the sidebar.

It is also more reliable than element search in browsers. At one point the ChatGPT
window's UIA element tree had collapsed to 13 nodes (caption buttons and empty panes,
the entire page gone) while read still returned the full page text. A find returning
nothing in a Chromium app does not mean the control is absent.

Writing without the element tree

type and key take an x/y point and click it first, because activating a window
does not give its input box keyboard focus — measured: keystrokes after a plain activation
were dropped, Ctrl+A selected nothing, and Enter went to whatever the app itself had
focused. That last one is how a "send" silently does nothing.

{"steps": [
  {"op": "type", "window": "ChatGPT", "mode": "physical", "x": 1436, "y": 1205,
   "text": "...", "verify": true},
  {"op": "key", "keys": "enter", "window": "ChatGPT"},
  {"op": "wait", "window": "ChatGPT", "state": "text_stable", "absent": "正在回应", "tail": 2000}
]}

Long text can go "via": "clipboard" — one clipboard write plus Ctrl+V instead of
thousands of SendInput records (the previous clipboard text is restored).

Five traps, each one measured

  1. focus activates a window but does not focus its control.
  2. The foreground is now returned once per call, not per step: restoring it between
    steps of the same batch steals the target app's focus and the next Enter lands nowhere
    (reported as foregroundRestored; restore: false opts out).
  3. Pure "the text stopped changing" settles on a "still working" placeholder — hence the
    absent / contains guards.
  4. Chromium's element tree can collapse; text keeps working.
  5. contenteditable text updates lazily, so an immediate read-back can show a stale value.

Also in this release

  • One batch tool runs a whole find / click / type / key / wait / read / shot / … sequence
    in one model request; a screenshot as the last step returns the picture inline.
  • Measured cost of a round trip: an 8-action task is 8+ requests or 1; the per-request
    context averaged 437,542 tokens in a long session, so one batch call saves ~3.06M context
    tokens (88%) on that shape.
  • Routing by complexity, measured both ways: a task the agent can specify without looking
    should not be delegated (−15%), while a cold multi-turn task saves 61% delegated
    (86% at ten turns). The protocol ships as the computer-operator skill.
  • Focus-free by default: UI Automation patterns, Win32 messages, offscreen PrintWindow
    capture. mode: "background" refuses rather than silently taking your screen.
  • Warm engine over loopback TCP: Add-Type is paid once (~800 ms/call → 40–100 ms/action).

Install

dsh plugin --profile web add github:wwwort/dsh-win-computer-use

Requires Windows and Windows PowerShell 5.1.