Repository navigation
Windows-native computer use for DeepSeek Harness. 0.1.1 closes the element-tree gap
on both sides: a whole chat round now runs with no element tree at all.
Reading is cheaper than looking
Text content is read as text, not captured as pixels:
Measured on the ChatGPT desktop app: one long reply came back as 2,200 characters of
quotable text in a single ~280 ms call; the whole page is 25,722 characters through the
same path. Document reading order puts the newest content last and the app's own chrome
first, so a tail read is clean by construction — and it never touches the sidebar.
It is also more reliable than element search in browsers. At one point the ChatGPT
window's UIA element tree had collapsed to 13 nodes (caption buttons and empty panes,
the entire page gone) while read still returned the full page text. A find returning
nothing in a Chromium app does not mean the control is absent.
Writing without the element tree
type and key take an x/y point and click it first, because activating a window
does not give its input box keyboard focus — measured: keystrokes after a plain activation
were dropped, Ctrl+A selected nothing, and Enter went to whatever the app itself had
focused. That last one is how a "send" silently does nothing.
{"steps": [
{"op": "type", "window": "ChatGPT", "mode": "physical", "x": 1436, "y": 1205,
"text": "...", "verify": true},
{"op": "key", "keys": "enter", "window": "ChatGPT"},
{"op": "wait", "window": "ChatGPT", "state": "text_stable", "absent": "正在回应", "tail": 2000}
]}Long text can go "via": "clipboard" — one clipboard write plus Ctrl+V instead of
thousands of SendInput records (the previous clipboard text is restored).
Five traps, each one measured
focusactivates a window but does not focus its control.- The foreground is now returned once per call, not per step: restoring it between
steps of the same batch steals the target app's focus and the next Enter lands nowhere
(reported asforegroundRestored;restore: falseopts out). - Pure "the text stopped changing" settles on a "still working" placeholder — hence the
absent/containsguards. - Chromium's element tree can collapse; text keeps working.
- contenteditable text updates lazily, so an immediate read-back can show a stale value.
Also in this release
- One batch tool runs a whole
find / click / type / key / wait / read / shot / …sequence
in one model request; a screenshot as the last step returns the picture inline. - Measured cost of a round trip: an 8-action task is 8+ requests or 1; the per-request
context averaged 437,542 tokens in a long session, so one batch call saves ~3.06M context
tokens (88%) on that shape. - Routing by complexity, measured both ways: a task the agent can specify without looking
should not be delegated (−15%), while a cold multi-turn task saves 61% delegated
(86% at ten turns). The protocol ships as thecomputer-operatorskill. - Focus-free by default: UI Automation patterns, Win32 messages, offscreen
PrintWindow
capture.mode: "background"refuses rather than silently taking your screen. - Warm engine over loopback TCP:
Add-Typeis paid once (~800 ms/call → 40–100 ms/action).
Install
dsh plugin --profile web add github:wwwort/dsh-win-computer-useRequires Windows and Windows PowerShell 5.1.
{"op": "read", "window": "ChatGPT", "tail": 2000} // newest 2000 chars {"op": "wait", "window": "ChatGPT", "state": "text_stable", "tail": 2000, "absent": "正在回应", "contains": "ChatGPT 说"}