Skip to content

v1.0.6 — AI eyes: screenshot tool + one-paste Deck setup

Choose a tag to compare

View all tags
@Garthnesss Garthnesss released this 28 Oct 04:30
· 9 commits to main since this release
v1.0.6
654f6ae

What this is

ElfDeckAI gives an AI agent hands and eyes on a Linux machine — built for the Steam Deck. Three zero-dependency daemons create a virtual keyboard (ElfKey), gamepad (ElfPad) and mouse (ElfMouse) through Linux uinput, and a small MCP server exposes them — plus screenshots — to any MCP-capable AI client (Claude Desktop, Claude Code, Codex, …) over stdio, typically via SSH. The AI sees the screen, presses buttons, reads the result, and plays: turn-based games, menus, dialogue, setup flows. Pure Python standard library, no packages to install on the Deck side beyond Python itself.

Install

On a Steam Deck (the one-paste setup) — switch to Desktop Mode, open Konsole, paste:

curl -fsSL https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/install-on-deck.sh | sh

It installs everything, starts the three daemons as user services, verifies the virtual devices exist, and prints the exact MCP config block for your PC — including the Deck's current IP address. Then turn on SSH on the Deck (passwd once, systemctl enable --now sshd) and add that config block to your AI client on your PC. Full walkthrough, written so you can hand it to your AI and let it explain everything: AI_SETUP_GUIDE.md.

Anywhere else (PC side or generic Linux):

pip install elfdeckai

Also on PyPI: https://pypi.org/project/elfdeckai/ — provides the elfkey, elfpad, elfmouse and elfdeckai-mcp commands.

What's new in 1.0.6

  • Screenshots work when the MCP server is launched over SSH. The server now imports the graphical-session environment (WAYLAND_DISPLAY / DISPLAY / session bus) from the systemd user manager when it's missing — previously, capture tools crashed in SSH sessions because they had no display to talk to. Found and fixed during a live session on a real Deck.
  • Installer: detects the Deck's IP address via ip (SteamOS has no hostname command in a minimal shell), so the printed MCP config contains your real address instead of a placeholder.
  • AI setup guide: new troubleshooting entry — if a Steam game ignores the gamepad (Steam Input can hand the controller slot to the Deck's built-in pad), drive the keyboard tools instead; some games treat the keyboard as primary input.

Also new since the last GitHub release (from 1.0.5)

  • screenshot tool (19 MCP tools total). Returns a PNG of the screen of the machine the server runs on, as MCP image content — the AI literally sees your Deck. Capture tries, in order: Spectacle (KDE), grim, scrot, ImageMagick, and the KWin D-Bus interface; if all fail, the error lists what each one said.
  • install-on-deck.sh — the one-paste installer described above. Works even on a stock Deck with no pip: it falls back to downloading the single-file programs directly and wiring up user services.
  • AI_SETUP_GUIDE.md — a setup guide written for an AI reader: hand the file to your assistant and it walks you through both sides (Deck + PC), verification, honest limits, troubleshooting and uninstall.

Verified on real hardware

This release was proven end-to-end on an actual Steam Deck, through the exact path a new user takes: the one-paste installer (including its no-pip fallback — a stock Deck has no pip), MCP-over-SSH from a second machine, a live screenshot of a running game (Dragon Quest VII, Desktop Mode), and inputs driven through the tools — the party was walked across the room and into an NPC conversation, confirming the see → act → see loop. 151 automated tests pass.

Requirements & honest limits

  • Linux with /dev/uinput (the included udev rule handles permissions on non-Deck machines; the Deck allows it out of the box). Python 3.10+, standard library only. 64-bit.
  • Screenshots need a graphical session: on the Deck that means Desktop Mode. Gaming Mode vision is not supported yet.
  • The control loop runs at screenshot speed — a second or two per look-and-act cycle. Perfect for turn-based, menu-driven and dialogue-heavy games; not for twitch action.
  • Everything is local-network by design: the daemons only listen on Unix FIFOs, and the MCP server reaches your PC only through your own SSH connection.

MIT licensed. Issues and PRs welcome.

Assets 2
Loading