Skip to content

Home

Jump to bottom
github-actions[bot] edited this page Jul 7, 2026 · 3 revisions

go-mcp-computer-use Wiki

go-mcp-computer-use

Go version Release CI Windows MCP Last commit PRs welcome Docs

Built iteratively across AI-assisted development sessions, with v0.1.x covering 70+ bug-fixed Win32/COM tools and v0.2.x (default branch) adding the chained automation pipeline, SQLite memory store, ONNX ML detection, introspection engine, adaptive ML, and the training data pipeline for user-specific model fine-tuning. The AI agent was guided by a curated set of quality-enforcement skills from coff33ninja/ai-skills — anti-hallucination, anti-slop, safe-code-modifications, anti-sycophancy, code-simplification, context-engineering, don't-kill-tokens, os-awareness, anti-tool-sprawl, follow-existing-patterns, no-dead-code-removal, universal-format-lint, self-validate, verify-and-cite, and others.

Status: v0.2.32 — 120 tools including statistical prior model, training pipeline, memory-backed UI element cache, ONNX detection, runtime privacy controls, key hold/release, input recording, set_config, YOLO dataset export, introspection engine, adaptive ML engine, OCR→command training bridge, and ONNX cascade fallback for template matching. See docs/reference/tools.md for the full listing.

Features

Features

  • Screenshot — full screen or region capture (GDI BitBlt → PNG → base64)
  • Mouse — click, move, scroll, drag, hover
  • Keyboard — type, key combos (Ctrl+C, Alt+Tab), type+submit, select all+type
  • OCR — extract text via Windows.Media.Ocr, optional language (en-US, ja-JP, fr-FR...)
  • Template matching — find an image on screen via NCC (normalized cross-correlation)
  • Find & Click — OCR + click: find text on screen and click it
  • Chained tools — find_text_and_click, launch_and_wait, wait_for_text, click_menu_item, select_all_and_type
  • Screen recording — capture frames at interval for a duration
  • Window management — list, focus, move, resize, min/max/restore, close, find, state
  • Audio devices — list playback/recording devices, set default
  • Clipboard — get/set text with retry + timeout
  • System — volume, mute, brightness, battery, disk, DPI, display info, uptime, idle
  • Network — hostname, IPs, DNS, gateway, ping
  • Processes — list, launch, kill
  • Power — shutdown, restart, sleep, hibernate, lock
  • Per-monitor DPI — per-monitor DPI awareness, scale reporting
  • UI Automation — find elements by name/automationID, get text, invoke buttons via native COM UIAutomation (no PowerShell)
  • OCR via native WinRT COM — StorageFile → BitmapDecoder → OcrEngine pipeline, 2-8x faster than PowerShell (falls back to PowerShell on error)
  • UIPI detection — warns when keyboard input targets elevated/admin windows
  • Training data pipeline — persistent screenshot collection with categorized folders (raw/click/, raw/type/, raw/navigate/, watcher/elements_found/, etc.) and SQLite metadata. Auto-saves on every UI action for model fine-tuning.
  • Memory-backed UI element cache — ONNX detections auto-stored as memory facts (ui:{window}:{class}) with TTL. AI reuses cached coordinates across sessions.
  • find_ui_element tool — cascading lookup: memory → ONNX → OCR. Self-learning: saves findings to memory + training store.
  • 120 MCP tools — see docs/reference/tools.md for the full listing

Tools

Auto-generated reference at docs/reference/tools.md — always in sync with internal/server/server.go. Run go run ./scripts/gen-tools-doc.go to regenerate.

Categories: Screenshot & Vision, Mouse, Keyboard, Window Management, Chained / Composite, Chain Automation, UI Automation, Browser Automation, File Explorer, Audio, Memory & Templates, ONNX ML, Priors & Statistics, Training Pipeline, Data Export, Data Logging, Adaptive Agent, Introspection & Debugging, Runtime Config, System, Process Management.

Quick Links

Documentation

Documentation

Clone this wiki locally