Telekinesis v0.9.0

What's Changed

• Medium for Dart/Flutter: annotations + build_runner manifest generator by @egarim in #41
• Locale-proof Medium matching via AutomationId by @egarim in #42

Full Changelog: v0.8.1...v0.9.0

Telekinesis v0.8.1 — relay exit-code fix

Patch on v0.8.0: relayed one-shot commands on Windows always exited 1 even on success — cmd parsed the batch's echo 0> rc as a stream redirect. Fixed with (echo %errorlevel%). Caught (and the fix verified) driving the FormGauntlet claim form end-to-end over SSH on real hardware: fill → rejected → read validation errors via UIA → fix exact fields → accepted on attempt 2.

If you script the CLI's exit codes on Windows over SSH, use 0.8.1, not 0.8.0.

Telekinesis v0.8.0 — headless one-shot CLI + Windows session relay

Drive a desktop over plain SSH — no MCP client (#37, #38)

Headless one-shot CLI: every perception and action is now a single telekinesis <verb> process printing JSON to stdout (exit 0 ok / 1 failed / 2 usage or refused): apps, tree, find, read, focused, snapshot, and — gated per invocation behind --enable-actions — launch, click, click-at, invoke, set-text, type, press. Targets are stateless queries re-resolved per call: "Save" or "Button:Save". snapshot returns the actionable elements with ready-made queries in one round trip. See docs/HEADLESS-CLI.md.

The Windows session trap, solved: a process started over SSH runs in session 0, where UIA sees no windows and input injection reaches no desktop. launch starts GUI apps in the logged-on user's console session via a one-shot Scheduled Task, and every other verb auto-relays itself there transparently — ssh winbox telekinesis apps just works. Live-proven end-to-end on real hardware (Surface, ARM64, over SSH): launch Calculator → snapshot → invoke One/Plus/Two/Equals natively → read back "Display is 3".

Packaging note: the Windows archives are now folder-based (not single-file) — WPF's UI Automation client crashes under single-file publish.

Full diff: v0.7.0...v0.8.0

Telekinesis v0.7.0 — browser control + provider registry

Telekinesis v0.7.0 — the browser is now first-class, and providers are pluggable.

Browsers, first-class (#20)

The web comes through the same accessibility tree — no browser driver, no CDP, no scraping:

• read_page — snapshot a page, reading its text + links/buttons/fields with actionable ids.
• find_elements — scope to page or chrome, so browser UI (the address bar, tabs) stops shadowing page content.
• navigate — load a URL straight from the tree.
• GetSubtreeAsync — the provider-layer sibling that lets a browser expose its page tree lazily (Chromium builds it on demand; read_page warms it).

Provider plugins (#21)

One interface, app-matched fidelity behind it:

• IProviderPlugin — a registry resolves each application to the highest-priority provider that claims it (Handles), around a DelegatingAccessibilityBackend you override only where you upgrade.
• BrowserProvider (page-first) — claims browser processes and reorders the default search: page content before chrome.
• Vision tier as a built-in fallback plugin — the vision tools (screenshot, parse_screen, recall_targets) ride the same registry.
• External plugins via explicit plugins.json only — no directory scanning, no auto-load; external assemblies are flagged by doctor and their tools are gated behind --enable-actions.

Full details in docs/BROWSERS.md and docs/PROVIDERS.md.

Dual-licensed AGPL-3.0 + commercial (see COMMERCIAL.md).

Install (dotnet tool, needs .NET 10 + Windows Desktop runtime on Windows):

dotnet tool install -g Telekinesis

Or a self-contained single file — no .NET required. Unzip/untar and run telekinesis (telekinesis.exe on Windows):

OS	x64	arm64
Windows	telekinesis-0.7.0-win-x64.zip	telekinesis-0.7.0-win-arm64.zip
Linux	telekinesis-0.7.0-linux-x64.tar.gz	telekinesis-0.7.0-linux-arm64.tar.gz
macOS	telekinesis-0.7.0-osx-x64.tar.gz	telekinesis-0.7.0-osx-arm64.tar.gz

Telekinesis 0.6.0

Telekinesis becomes scriptable, safe to expose, and self-teaching.

Scenario runner, assertions, CI (#3)

• telekinesis run demos/<scenario>.json --enable-actions — scripted, self-verifying demos with caption output: pre steps, bind + {{binding.path}} substitution, assert expressions, nonzero exit on first failure. See the Windows-validated demos/calc-add.json.
• assert_element tool + telekinesis assert — poll for exists/visible/enabled with a 0/1 exit contract for shell scripts.

Remote posture (#3)

• telekinesis serve --sse --port N — MCP over HTTP, localhost-only. Read-only by default; --enable-actions required for the action tools. Every action also lands in a JSON-lines audit file. Deployment guidance in docs/REMOTE.md.
• fill_credential — password-manager handoff: the secret is typed by your credential provider (TELEKINESIS_CREDENTIAL_CMD), never passing through the model, the server, or the log.

The tiers teach each other (#16)

• recall_targets(show: true) / probe --recall --show — remembered targets drawn as X-ray boxes with match scores.
• TELEKINESIS_LEARN=1 — successful a11y actions record perceptual-memory anchors (a11y-verified labels), with a visibility guard so covered elements never store wrong pixels.

Local UI brain (#10, experimental)

• telekinesis pilot "<goal>" --app pid:N --enable-actions — a local model (Ollama; default qwen3:4b-instruct, TELEKINESIS_BRAIN_URL for a LAN/GPU box) plans one schema-constrained action per step over ranked candidates + readouts; validated, guarded, and trace-logged.
• telekinesis pilot-eval <trace.jsonl> --model X — replay recorded traces through any model offline; agreement + latency. Honest baseline in docs/PILOT.md: prompted 4B grounds perfectly (100 % schema compliance) but misplans sequencing — the traces this release collects are the path to the tuned pilot (#19).

Dual-licensed AGPL-3.0 + commercial (see COMMERCIAL.md).

Install (dotnet tool, needs .NET 10 + Windows Desktop runtime on Windows):

dotnet tool install -g Telekinesis

Or a self-contained single file — no .NET required. Unzip/untar and run telekinesis (telekinesis.exe on Windows):

OS	x64	arm64
Windows	telekinesis-0.6.0-win-x64.zip	telekinesis-0.6.0-win-arm64.zip
Linux	telekinesis-0.6.0-linux-x64.tar.gz	telekinesis-0.6.0-linux-arm64.tar.gz
macOS	telekinesis-0.6.0-osx-x64.tar.gz	telekinesis-0.6.0-osx-arm64.tar.gz

Telekinesis 0.5.0

Three new capability tiers on top of the accessibility core.

Vision fallback — for when the tree fails

• screenshot — capture the desktop or a region as PNG (read-only-safe); captures include layered windows (raw BitBlt + CAPTUREBLT).
• parse_screen — screenshot → UI elements via an optional Microsoft OmniParser sidecar (official omniparserserver contract, TELEKINESIS_OMNIPARSER_URL); pixel bounds ready for…
• click_at — pointer click at raw screen coordinates.
• Escalation flow, sidecar setup, and coordinate-space rules in docs/VISION.md.

X-ray overlay — see what the AI sees

• highlight — draw a labeled box over an element or region on the real desktop; click-through, never takes focus.
• probe --overlay --app pid:N — live numbered boxes over every element of an app (--for N for timed runs).
• TELEKINESIS_SHOW_INTENT=1 — injected actions flash their target before the input lands: the agent telegraphs its moves.
• docs/XRAY-OVERLAY.md includes a demo filming script.

Perceptual memory — learning from previous runs

• Screens seen before answer instantly from a perceptual-hash parse cache ("source":"memory").
• Targets successfully acted on become anchors; recall_targets re-locates them on the live screen with zero parser calls; misses decay and evict.
• telekinesis memory export --out — the store doubles as a training-ready grounded dataset (crops + captions + normalized boxes + use counts) for a future local grounding-model adapter.
• docs/PERCEPTUAL-MEMORY.md

Also in this release: per-monitor-v2 DPI self-check in doctor + covered-element click guard (#7 partial), minimized-window bounds fix. Windows overlay/vision/memory tiers are opt-in capability interfaces — Linux and macOS report them as not supported until their implementations land. Dual-licensed AGPL-3.0 + commercial (see COMMERCIAL.md).

Install (dotnet tool, needs .NET 10 + Windows Desktop runtime on Windows):

dotnet tool install -g Telekinesis

Or a self-contained single file — no .NET required. Unzip/untar and run telekinesis (telekinesis.exe on Windows):

OS	x64	arm64
Windows	telekinesis-0.5.0-win-x64.zip	telekinesis-0.5.0-win-arm64.zip
Linux	telekinesis-0.5.0-linux-x64.tar.gz	telekinesis-0.5.0-linux-arm64.tar.gz
macOS	telekinesis-0.5.0-osx-x64.tar.gz	telekinesis-0.5.0-osx-arm64.tar.gz

Telekinesis 0.4.0

Adds slider and combo-box control to the MCP surface.

• set_value — set a numeric element (slider, spinner) via the native RangeValue pattern.
• invoke now takes an optional action (expand / collapse / toggle / select) so combo boxes and tree items are operable.

Both were already in the backend and CLI; this exposes them to MCP clients. Dual-licensed AGPL-3.0 + commercial (see COMMERCIAL.md).

Install (dotnet tool, needs .NET 10 + Windows Desktop runtime on Windows):

dotnet tool install -g Telekinesis

Or a self-contained single file — no .NET required. Unzip/untar and run telekinesis (telekinesis.exe on Windows):

OS	x64	arm64
Windows	win-x64.zip	win-arm64.zip
Linux	linux-x64.tar.gz	linux-arm64.tar.gz
macOS	osx-x64.tar.gz	osx-arm64.tar.gz

Telekinesis 0.1.0

First packaged release.

Install as a dotnet tool (needs .NET 10; on Windows also the Desktop runtime):

dotnet tool install -g Telekinesis

Or download a self-contained single file — no .NET required. Unzip/untar and run telekinesis (telekinesis.exe on Windows):

OS	x64	arm64
Windows	win-x64.zip	win-arm64.zip
Linux	linux-x64.tar.gz	linux-arm64.tar.gz
macOS	osx-x64.tar.gz	osx-arm64.tar.gz

MCP config:

{ "mcpServers": { "telekinesis": { "command": "telekinesis" } } }

Run telekinesis doctor first. Prefer --read-only when the agent only needs to see.

🤖 Generated with Claude Code