PointHand: a Windows computer-use MCP server that borrowed a few ideas from agent-desktop

[ayzekhdawy]

Hi agent-desktop team, and thanks for publishing such a carefully documented project.

I maintain PointHand, an MIT-licensed computer-use MCP server for Windows 10/11 (TypeScript MCP front end, Rust host on UI Automation and SendInput). It focuses on supervised creative and practical desktop work: it captures covered windows from their own pixels, binds pointer actions to per-monitor screenshot regions, draws freehand strokes and shapes (it painted the Mona Lisa in MS Paint), records speaker loopback, and keeps a visible overlay with a physical-Escape stop and an app allow-list.

Reading your CONCEPTS.md changed how we think about a few things, and the next PointHand release reimplements these ideas in our own host (no code copied; Apache-2.0 credited in our docs):

• Headless actions → background mode. With background: true the agent works in one window while the user keeps working in another: UI Automation actions run without moving focus, and clicks, drags, drawing, scrolling, typing and single keys are posted to the target window even when it is covered. The idle gate is skipped because nothing physical moves; the allow-list and Escape still apply. Like you, we report delivery as posted rather than claiming success.
• find and wait → find_elements (role / name / automation id over up to 3000 nodes) and wait_for (window title or element appears or disappears).
• Window management and protected processes → minimize / maximize / restore, move / resize, and a polite close that refuses session processes such as explorer.exe.
• Secure fields → password values never appear in the tree.
• Integrity boundary → input to a higher-integrity window is refused up front instead of being dropped silently.
• Toggle / Select / Focus semantic actions and get_element.

If any of this is useful back on your side (for example the Windows notes on posted mouse messages and which app families ignore them), I'm happy to share what we measure. Thanks again for the inspiration.

— İshak, Noniux