You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(computer-use): support graphical VMs inside Chrome (#146)
* feat(computer-use): support graphical VMs inside Chrome
Cindro could drive ordinary web pages through DOM refs, but not a VM or
remote desktop rendered inside a tab (noVNC/Proxmox, Guacamole, SPICE,
VMware consoles, canvas/WebRTC desktops): there are no elements to snapshot,
the guest lives in nested cross-origin iframes, and synthetic JS events never
reach it.
Engine (computer-use/):
- browser_visual.py: pure logic — measured image->CSS mapping (absorbs DPR,
browser zoom, debugger infobar, crop/downscale), iframe-chain + css-transform
+ object-fit letterbox composition of graphical surfaces, guest-framebuffer
coordinates, US-layout CDP key/mouse event builders, per-tab inspection
store keyed by inspect_id for concurrent agents.
- tools_browser_visual.py: browser_visual_inspect, browser_click_xy,
browser_move_xy, browser_drag_xy, browser_scroll_xy, browser_mouse_down/up,
browser_key_press, browser_type_trusted, browser_input_release. verify=True
re-captures and reports whether the screen changed. browser_snapshot now
flags graphical pages so agents switch to pixel tools automatically.
- policy: browser_visual_inspect is plan-safe; text typed by
browser_type_trusted goes through the destructive-command scanner.
Extensions (extension/ and computer-use/extension/, shared visual.js):
- page.inspect / page.surfaces / page.focus / input.sequence handlers:
Page.captureScreenshot + OffscreenCanvas crop/scale/grid, per-frame surface
scan (incl. OOPIFs and shadow DOM), allowlisted Input.* dispatch with
release-on-error, stale viewport/scroll refusal, focus emulation. Glow
cursor follows xy input and is hidden during captures. Versions bumped.
API brains: ApiBrain forwards MCP image results (screenshots) to
OpenAI-compatible models as a follow-up user message, keeps only the newest
two, and drops them + retries once if the backend rejects images.
Prompts/docs: co-work guide + capability catalog (v6) + side-panel primer
mention the VM tools; docs/BROWSER_VM_CONTROL.md; AGENTS.md gotchas.
Tests: unit tests for the pure logic, tool tests against a scripted bridge,
and a real headless-Chromium lab that loads BOTH shipped extensions and
drives fake noVNC/Guacamole consoles (cross-origin OOPIF, nested and scaled
iframes, letterbox, 150% zoom, DPR 2, resize/scroll/fullscreen, disconnect
and recovery). linux-ci installs chromium so the lab runs in CI.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uV59CXB81WmJAamwRL5eH
* packaging: ship a real .DirIcon/PNG for the new appimagetool
The appimage job fetches appimagetool's continuous build, and the
2026-10-29 build now fails an AppDir whose .DirIcon is missing, a symlink
or broken ("No valid .DirIcon"), and resolves the desktop Icon= only as
.png/.xpm. linuxdeploy only symlinks the SVG, so the last successful run
(2026-10-04) predates the break rather than this branch's changes.
Install a 256x256 PNG rendering of packaging/cindro.svg as both
AppDir/cindro.png and a regular-file AppDir/.DirIcon before packaging.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uV59CXB81WmJAamwRL5eH
---------
Co-authored-by: Claude <noreply@anthropic.com>