Repository navigation
Partner v0.2.2
Partner v0.2.2
A hardening release for desktop control. v0.2.1 fixed a bug in Memory Services and left the desktop as it was; this release is the other half — the parts of "let the model press buttons in my apps" that were true in the code but not proven on screen. Four things changed, and they change what an action reports, not what it can do:
- the UI Automation client is a COM client on its own thread, with timeouts, instead of the managed one;
- password fields never report their value, and system shortcuts ask on their own;
- every action says whether it was delivered and whether retrying is safe, instead of leaving that to be read out of prose;
- an observation says why it is incomplete, and a capture or comparison that spans a window moving is thrown away rather than judged.
The session, memory, settings and workspace formats are untouched. This is a drop-in replacement for v0.2.1, with one caveat: a consumer that parsed the first line of a desktop_act result sees a JSON object there now (see Upgrading).
The result of an action now says what happened
A model that only reads prose had to guess whether "点击完成" meant "the click landed, the screen changed" or "SendInput accepted it, nothing else". It also could not tell a refusal from a partial delivery, so it retried clicks that may already have happened.
Every desktop_act result now begins with one line:
{"code":"OK","delivery":"posted_unverified","retry":"unsafe"}code—OK,STALE_REF,AMBIGUOUS_TARGET,PERM_DENIED,POLICY_DENIED,TIMEOUT,APP_UNRESPONSIVE,NOT_FOUND,INVALID_ARGS.delivery—not_delivered,delivery_uncertain,posted_unverified,delivered_verified. "Posted" means Windows accepted the whole input batch, which is not the same as the application receiving it: UIPI silently drops input aimed at a higher-integrity window whileSendInputstill reports success.retry—safeonly when nothing was injected; everything that may have landed isunsafe.
The prose after that line keeps the detail, and "注入成功" became "已发出(Windows 已接收输入,投递结果以下面的校验为准)" so the promise matches the evidence. The same three fields are in the HTTP answer of the desktop endpoint.
A partial input batch releases what it pressed
SendInput can insert a prefix of a batch and fail the rest. That leaves a modifier held — Shift down, no Shift up — and the user's keyboard is broken for reasons nothing in the result explains. A failed batch now releases exactly the keys and buttons it pressed (never a modifier the user was already holding), then reads the key state back and says whether the release was confirmed.
What a keystroke is allowed to reach
Password fields never report their value
A password field is identified three ways: the UI Automation IsPassword flag, a page input with type=password, or an autocomplete that contains password (so current-password and new-password are both covered). Such an element is marked secret, its value is null, and it never serves as the control's display name — using an unlabelled password field's value as its name put the plaintext into the observation, which is the one place it must not appear. After a write into it, the state comparison only asks empty or filled.
System shortcuts ask on their own, and never get a task lease
Win+R opens a run dialog, Win+L locks the session, Alt+F4 closes the application, Ctrl+Shift+Esc opens Task Manager. These leave the window a human approved, so "continue working in this window for 10 more steps" must not widen into them. Any Win chord, Alt+F4, Alt+Tab, every Ctrl+Alt+* and Ctrl+Shift+Esc is now named by a guard, asks under every gear except No approval, and does not spend or receive the 10-step lease an ordinary in-window action does.
A keystroke goes to the named control, not to whatever had the caret
Typing into a form whose focus still sits in the previous field writes into that field, reports success, and leaves the model to discover the misplaced text later. A KeyPress or TypeText aimed at an elementId now calls SetFocus and confirms HasKeyboardFocus before injecting; if the caret is somewhere else the action is refused and the current focus is named. The window also has to be in front first, because focusing a control in a background window can raise that window — and then the keystroke lands in a window the user was not looking at.
An incomplete observation says why
A window whose walk stopped was previously reported with a single boolean. It now carries elementsTruncated with the reason — Depth, Produced, Visited, Time, Fault, Refused, Unresponsive, Gone — because they lead to different next steps, and the note the model gets is worded for the next step rather than for the cause the code guessed:
- unresponsive → wait or let it recover, do not re-observe in a loop;
- refused (permissions, UIPI) → screenshots and coordinates only;
- budget exhausted → narrow with
nameContains/roles/offset, do not treat the list as complete; - nothing but a webview surface → it may be a Chromium/Electron app, and that is now a possibility instead of an assertion about Electron.
A window that shares an identity with another control in the same window is marked ambiguous; acting on it by elementId returns AMBIGUOUS_TARGET and asks for coordinates or a narrower observation, rather than flipping a coin between two equally plausible controls.
The UI Automation client
The managed client is replaced by the COM API (CUIAutomation8) on a dedicated MTA thread: a 2-second connection timeout and a 5-second transaction timeout, so a provider that stops answering is a named refusal rather than a stalled turn. Waiting for a Chromium tree to settle happens on the caller, not the UIA queue.
A provider in another process is read through a CacheRequest in AutomationElementMode_Full; a provider in this process is walked without caching. The mode matters: the default None returned 63 of 129 nodes on an Edge page, while Full returned 127 of 127.
Measured on this machine, the COM client returns the same node counts as the managed one (Notepad 43/43, Explorer 7/7, a Chromium window 87/87, an Edge page 127/127) — so the reason for the change is the timeouts, the thread and the cache, not a claim that the managed client could not see. Single-window observe p50: 6.9ms (Explorer), 61.7ms (Chromium, at the 40-control cap), 327.0ms (Edge page).
Two more guards came with it:
- A window that was captured is identified before and after (handle, process start time, rectangle). A frame taken while the window moved, resized, closed or was replaced is discarded, and a comparison across such a move answers
Indeterminate— comparing two coordinate spaces would otherwise be reported as "the screen changed". - A hung window is probed with
SendMessageTimeout(WM_NULL, 2s)first and reported asAPP_UNRESPONSIVEwithout entering UI Automation at all.
Window enumeration also stopped listing windows that are not on the screen the model is looking at: cloaked windows (another virtual desktop, a suspended UWP app) and unowned tool windows are excluded, and a UWP app hosted in ApplicationFrameHost.exe is reported as the process inside the frame rather than as the host.
Upgrading
- Install v0.2.2 directly, or let v0.2.1 update itself.
- No data or format changes: settings, sessions, the memory database and downloaded engines are untouched.
- If you have a tool, script or prompt that reads the first line of a
desktop_actresult, it now sees{"code","delivery","retry"}. The prose follows on the next line, and "已发出" replaced "注入成功". - System shortcuts (Win chords, Alt+F4, Alt+Tab, Ctrl+Alt+*, Ctrl+Shift+Esc) ask even under Approve risky actions, and approving one does not grant a lease.
- Nothing listens on a new port and no new permission is requested; the debugging port keeps its existing rule (only browsers and Electron apps started by
desktop_launch).
Limits
- Measured on one 2560×1440 screen at 96 DPI (100%), virtual-desktop origin (0,0). Dual monitors, a secondary screen with negative coordinates and 125%/150% scaling are still untested;
probes/windows/run.ps1 -Command monitorsprints screen identity and DPI. - The "first Chromium read is incomplete, then settles" behaviour was not reproduced here. The code still re-reads at 250ms until the count stops growing, but a real cold start has not been proven.
- The page-control path has still not been verified on real Electron applications.
- No OCR: interfaces UI Automation cannot see are left to a vision model reading screenshots.
Engineering
DesktopSafetyTests(new, offline) covers the partial-batch release, the shortcut rules and the governance hook, the password and ambiguity markings, and the delivery classification.DesktopLiveSafetyTests(new,PARTNER_DESKTOP_LIVE=1) covers the same guarantees against a real window: a password field's value never reaching the walk, typing going to the named control rather than the one that had focus, a capture whose window moves being discarded, and a hung window being reported without waiting for UIA.probes/windows/UiaClientProbeis a standalone probe (not product code) that prints one line per measurement so a row can be copied into the ledger inprobes/windows/FINDINGS.md;probe-page.htmlandrun.ps1are its fixtures. The ledger records both confirmations and contradictions, including the two claims in the earlier notes that this machine did not reproduce.- The smoke's
desktop_actchecks matched the result body at its start; they now match anywhere in the body, because the contract line comes first.
The voice engines are downloaded from the engines-v1 release on first use; their URLs and SHA-256 are pinned in this build. The app code is MIT licensed; the engine runtimes and model weights keep their own licences, see THIRD-PARTY-NOTICES.md in the release assets.