Repository navigation
Releases: momasiku/dsh-pilot
Release list
dsh-pilot v0.6.5 - the takeover ring lands on the pointer, and stays long enough to read
Desktop control for the whole Windows machine. screen_view hands the model a real PNG of the live screen; desktop_control drives the real mouse, keyboard and windows against that same pixel space; desktop_sequence runs a whole batch of actions in one call and looks once at the end.
What changed in 0.6.5
The takeover prompt is where the pointer is. The overlay process is DPI-unaware — powershell.exe's manifest settles awareness before the script runs, so asking for per-monitor v2 from inside it returns false. On a 150% display the panel is 2560×1600 while WPF laid the window out for 1707×1067, and GetCursorPos still answered in physical pixels: the halo was drawn half again away from the pointer. The script now reads the real display mode through EnumDisplaySettings (WMI's Win32_VideoController reports 0×0 on current NVIDIA drivers) and divides the cursor position by the quotient. The caption was never wrong — laying it out in logical units is what makes the OS scale it.
The hide delay is a setting. takeoverLingerMs, default 8 seconds, replaces a hard-coded two. A single action is long enough to read now, and Esc still cancels while the ring is up.
Verified two ways: the overlay run standalone with a frozen cursor logs physical 2560x1600 (EnumDisplaySettings), scale 1.5 and places the halo at 853,533 logical = 1280,800 physical; run against the real pointer, a 1:1 crop shows the ring centred on the cursor (centre 908,740 vs pointer 908,739).
Install
# the repository
dsh plugin --profile <profile> add github:momasiku/dsh-pilot
# or this release's prebuilt tarball
dsh plugin --profile <profile> add https://github.com/momasiku/dsh-pilot/releases/latest/download/dsh-pilot.tgzRestart the desktop app once afterwards, then start a new conversation.
Requirements: Windows, PowerShell 5.1 or 7, Node ^22.19.0 || >=24.0.0, DSH 0.1.5-rc.2 through 0.2.x, and a model route that declares image input (deepseek-flash does).
Assets
| Asset | Why |
|---|---|
dsh-pilot.tgz |
version-free name, so releases/latest/download/dsh-pilot.tgz keeps pointing at the newest build |
dsh-pilot-0.6.5.tgz |
the same tarball, for URLs pinned to this tag |
v0.6.4 - the prompt is held per round
接管提示改成按回合显示
- 一个回合只显示一次，最后一次动作后 2 秒隐藏；新动作取消隐藏。
- 修掉启动竞态（首个动作记两次 show）。
Test-Path短暂失败不再立即隐藏（连续三次读不到才隐藏）。- 单实例互斥：DSH 重启留下的孤儿进程会自己退出。
- 新增
tools/overlay-smoke.ps1，四项测量。
已知代价（按你的要求不预热）：首次出画约 0.7 秒。
v0.6.3 - controls reaches the model
controls 的两处真问题
- 渲染层丢字段：
controls/windows在结构化输出里有、模型可见文本里没有。现在渲染成<controls>/<windows>块。 $name未赋值：$null.Length -eq 0恒真，ListItem / TreeItem / MenuItem / DataItem 四类无论有没有名字都被丢。Pane不在白名单：原生#32770对话框的所有子控件都是 Pane，导致满按钮的对话框过滤后为空。命名 Pane 现在算控件。- 过滤不再无声：新增
controlTotal/filteredOut。实测运行对话框 10 → 保留 6。 - 回归测试：
script-smoke真开一次 Win+R 断言非空。
v0.6.2 - verified on DSH 0.2.1-rc.3
兼容性声明落到能验证的地方
- 在 DSH 0.2.1-rc.3 上实测通过：
screen_view、windows、controls、desktop_sequence全部正常。 check-peers --asar：直接读已安装桌面端的app.asar，按它真实携带的 dsh-* 版本判定，而不是只看开发副本。- 两个细节：Electron 的 fs 会拦截
.asar路径（回退临时副本）；asar 头部有 NUL 填充（先剥离再JSON.parse）。
v0.6.1 - 简介和 README 都用中文先说
简介和 README 都用中文先说
- 仓库简介和 npm 简介换成同一句中文：能做和人一样的事，包括且不仅限于 CAD、WPS、浏览器操作、发微信；安全、快速、可靠、聪明、人性化。
- 中文说明移到 README 顶部，英文段落后置并加上
# English标题，顶部语言链接同步调换。 - 没有代码改动。
v0.6.0 - a takeover line and a lit pointer
让桌面看得出谁在操作
动作运行期间，屏幕顶部一行大号半透明字写明操作者与出口 ——
DSH 正在操作你的电脑 · 按 Esc 取消 —— 光标处跟着一个蓝色光晕。
没有屏幕边框：第一版画过，2560x1600 上要么在蓝壁纸里看不见，要么重得像相框。
Esc 中止的是整个批次，不是当前一步。 光晕盖章的取消令牌本来就由动作循环轮询，
现在它属于整个 desktop_sequence，所以中间的 wait 也会被打断，结果里会写明原因：
the person pressed Esc; the batch stopped where it was。实测：12 秒的批次，
第 4.5 秒按 Esc，7.1 秒返回。
光晕是画出来的，从不写入光标：不调用 SetCursorPos，不装输入钩子。
Windows 永远把真实光标画在所有窗口之上，所以光晕垫在它下面 —— 看起来更大更亮，
但没有替换它，也就没有别人那种让鼠标发卡的问题。
顺带修的三个
- 标题比较前先剥零宽字符。 Edge 自己的标题里
Microsoft Edge中间藏了个 U+200B，
于是focus "Microsoft Edge"报 "no visible top-level window matches" —— 而那个窗口人眼看得清清楚楚。 focus允许只给hwnd，和它的说明一致；之前步骤校验强制要求title。- 每个脚本必须保留 UTF-8 BOM。 PowerShell 5.1 把无 BOM 的文件按 ANSI 读，
中文标题就成乱码。script-smoke现在会因此失败 —— 它当场抓出另外三个已经丢了 BOM 的脚本。
覆盖层还必须不抢前台：它在第一次 Show 之前就设好 WS_EX_NOACTIVATE，
否则一个置顶窗口会抢走焦点，反而破坏它自己要服务的那些动作。
v0.5.17 - a one-control window, and no guessed preview size
Two fixes.
A window that publishes exactly one control no longer fails its own schema.
controls on a LibreOffice window returned
tool "desktop_control" returned invalid output: "value.controls" must be an array
— PowerShell unrolls a one-element array on return, so the listing came back as a
bare string after the tree had already been read. Fixed at the source with the unary
comma, and again in the host half, which wraps a stray string back into an array.
The contract no longer predicts where the request pipeline will put a frame.
0.5.14 predicted it from attachments.imageLimits.maxImagePixels and called it 1:1
when the capture fit under that budget. Measured on a 1920x1080 display: the plugin
read 64,000,000 px and said "one image pixel is one real screen pixel", while the
message carrying the frame said request preview 1708x961 — already shrunk 0.89x.
64,000,000 is the deployment's admission limit; the projection uses the model route's
image budget (~1,640,000 px here), which the plugin cannot read. Predicting from the
wrong number produces a confident false promise, so the prediction is gone. The
contract states the capture size, says plainly that a step it cannot see may shrink
the frame, and names the size the message itself reports as the authoritative one,
with the arithmetic spelled out: multiply by 1920 / preview_width.
Verified: controls on a one-element list returns System.String[]; five suites
green; the new contract rendered through output.render for both branches.
v0.5.14 - predict the request preview
The multiplier is a number now, not a division.
A frame is captured at physical size, stored, and then shrunk again by the request
pipeline to fit the model's image budget — a step this plugin never sees. On a
2560x1600 display the model was handed a 1610x1006 preview while the contract beside
it said "delivered at capture size … do not rescale". Every reading taken off that
preview and passed through unchanged lands 1.59x off.
The deployment publishes the same budget the pipeline uses
(attachments.imageLimits.maxImagePixels), and the projection is a pure function of
it, so the pipeline can be predicted instead of described. The contract now names
the size you will actually see:
2560x1600 capture -> 1609x1006 preview, multiply every reading by 1.5911
1920x1080 capture -> 1696x954 preview, multiply by 1.1321
3840x2160 capture -> 1696x954 preview, multiply by 2.2642
1024x768 capture -> 1024x768 preview, exactly 1:1
1365x960 capture -> 1365x960 preview, exactly 1:1
Nothing is hardcoded, so it holds at any resolution: a frame whose area fits the
budget arrives untouched, which is why framing the window or a region is now the
advice for a true 1:1 frame.
The prediction mirrors requestImageDimensions() locally, and every contract still
ends by deferring to the preview size the message itself states — a drift between the
two costs accuracy, never correctness.
Verified across 1024x768, 1365x960, 1920x1080, 2560x1600, 3840x2160 and 5120x2880;
script-smoke 15/15, sequence-smoke 23/23, smoke, render-smoke, check-peers
all green.
v0.5.12 - a cold Electron tree, and an empty list that explains itself
Two things the first run on a real desktop turned up.
Chromium builds its accessibility tree lazily. Measured on the harness window
right after a restart: 3 elements and 1.2 s, then 77 elements and 0.3 s on the next
call. A caller that trusts the first answer concludes the window has no tree and falls
back to pixels for nothing. controls now retries once, after 300 ms, when the worker
has not yet seen a real tree and the answer is suspiciously small. A window that
genuinely has few elements pays one extra read per worker process and no more.
An empty filtered list now explains itself. meaningfulOnly can legitimately match
nothing — Notepad on this machine publishes two anonymous panes and no operable
control at all — and an empty list reads as "this window has no tree", which is the one
wrong conclusion to leave a caller with:
(no operable controls here; the window has 2 elements - pass meaningfulOnly:false to see them)
Verified on a real desktop after a restart: controls returns 77 elements / 3415 chars
/ about 854 tokens in 309 ms with no capture; target 15 resolved to its exact
rectangle centre 26,20 in 259 ms; move by target 259 ms against 24 ms by x,y and
1114 ms by x,y with a frame; the Notepad hint above. script-smoke 15/15,
sequence-smoke, smoke, render-smoke, check-peers all green.
v0.5.11 - read the control tree, click a control by index
Read the control tree, and click a control by index.
A screenshot costs a guess: the model finds a button in pixels, estimates its centre,
clicks, and takes another frame to see whether the guess was right. controls returns
the same controls as role, name and rectangle, so the click becomes the centre of that
rectangle and nothing has to be looked at.
desktop_control({ action: 'controls', title: 'Notepad' })
// 4 [Button] '最小化' @ 887,1 45x39
desktop_control({ action: 'click', target: 4, title: 'Notepad' })Measured on a 673-element window: 0.65 s and ~850 tokens filtered (77 operable
elements) against ~9k unfiltered, and ~150 ms / ~1k tokens for the frame it replaces.
controls takes no screenshot, so it pays no settle.
An index is resolved against a fresh read at click time, and the worker remembers the
listing it handed out: a row that changed identity fails loudly instead of being
clicked by position.
It does not always exist. A canvas, a drawing surface, a custom-painted panel and an
Electron window that publishes nothing give an empty or useless tree - the screenshot
path is the fallback, and the README says when to use which.
Also in this release: the settle is no longer paid when nothing is captured (0.5.9,
814 ms -> 12 ms), a refused focus now fails instead of being reported as success
(0.5.9, tolerant: true restores the old behaviour), and the peer range accepts the
0.2.1 host generation (0.5.10).
Verified: script-smoke 15/15, sequence-smoke 23/23, smoke, render-smoke,
check-peers all green.