Repository navigation
Releases
beta-2.0
VDroid Scripter, beta-2.0
Second beta of VDroid Scripter (Visual Android Scripter). beta-1.0 gave you a library, steps and routes. This release makes them travel: one library now serves phones of different sizes and densities, one route serves any text you pass to it, and typing needs no keyboard setup at all. The AI seat grew the most: two working modes, the ability to see the screen and curate its own icons, and a much smaller bill per turn. Underneath, OCR reads screens it used to miss and the server no longer depends on your distribution's libraries.
✨ Highlights
- ⌨️ Typing with no keyboard setup.
type_textnow reads the open keyboard off the live frame, recognises its layout by the language of the step (QWERTY, AZERTY, QWERTZ, ЙЦУКЕН and the other European layouts) and taps the keys. Capitals go through Shift, digits through the number row, and the localenumerictypes on a numeric keypad or an app's own dial pad. Nothing about keyboards is stored any more, and the key editor is gone. - 🧳 A library that travels between phones. A recorded gesture is scaled to the screen it is replayed on, so it covers the same share of the display. A template image is saved with the screen density of its device and resized to the density of the phone it is searched on. One crop and one recording serve phones of different resolutions, as long as they draw the element the same way.
- 🧮 Routes with arguments. Put
%sin the text a route types or looks for, and pass the values when you run it:args=Молоко&args=Молоко. One route now adds any item to a basket or messages any contact.args_localenames the language of the values, and a wrong number of values is refused before anything runs. - ✂️ Run any slice of a route.
end_idjoinsstart_id, both inclusive. Run a route up to a step, from a step, or just the part in between. - 🧭 Two modes for the AI. The MCP server starts in navigator: it records nothing, treats a request as a goal, and reaches it by chaining the routes you already have, picking the slice and the arguments itself instead of asking which route to run. Switch it to explorer with a route name and it works a goal out on unknown screens, recording every step that succeeded into that route as it goes.
- 👁️ Eyes for the agent.
GET /devices/{serial}/capturereturns the current frame as JPEG or PNG. A vision-capable agent uses the newcapturetool wherescanis not enough, andsave_imageto crop an icon into the library by itself.scanstays first and stays the source of landmark values. - 🪙 A cheaper, faster agent loop.
scanhands the agent a compact table, about three times fewer tokens than the JSON, and leaves out low-confidence noise.queue_stepsandrun_routewait for the batch and answer with the result, so there is no polling. The built-in instructions and tool descriptions were cut down to about 4.6k tokens per turn. Four new tools bring the total to 19:capture,save_image,edit_routeandset_mode. - 🔤 OCR that reads more screens. Text is now read twice, with a global and a local binarization, and the results are merged. White launcher labels over a photo wallpaper, which used to return nothing, are found. Every word carries a confidence, and
ёmatchesе, soПятерочкаfindsПятёрочка. - 📦 A self-contained server.
install.shbuilds OpenCV, FFmpeg, Leptonica and Tesseract from pinned sources as static libraries and links them into the binary. A distribution upgrade to OpenCV 5 or a new FFmpeg can no longer break it, and the Tesseract language files are downloaded for you.
💥 Breaking changes
- The keyboard endpoints are removed.
GET /devices/{serial}/keyboard,POST /devices/{serial}/edit_keyboard,GET /devices/{serial}/reset_keyboardandGET /devices/{serial}/delete_buttonno longer exist, along with theKEYBOARDS_DIRsetting and the keyboard editor in the Android client.type_textneeds the keyboard already open and showing the language of the step. Punctuation and symbols fail the step. - Routes have no
prompt. The field is gone from the model, the API andsave_route. Routes saved by beta-1.0 still load, and the prompt is dropped the next time they are saved. %sin a text landmark is a placeholder. A route that contains one must be run with matchingargs. Saving a route clears thelocaleof every landmark that holds a%s, because the language comes with the run.POST /save_imagecrops the video frame, not an ADB screenshot. Rectangles are in video-frame coordinates, the same onesscan,rectanglesandcaptureuse. Each image now has a smallimages/<name>.jsonnext to it with the density.- MCP tools behave differently.
queue_stepsandrun_routeblock until the batch is done, for up to three minutes, andwait_for_sessionis only for what runs longer.scanreturns a table instead of JSON. Routes are recorded in explorer mode or written withsave_route, and the agent changes modes only when you tell it to. - The server writes no log files. The
LOGSsetting is gone and logs go to stdout. When the MCP server startsvdroid-scripterfor you, its output is appended tologs/server.login the data directory. install.shno longer uses distribution packages for the native libraries. The first run builds them, which takes ten to fifteen minutes. Later runs reuse the result.
🛠 Reliability and fixes
- The phone's screen stays awake while a session is open. Before, scrcpy's stay-awake flag was silently ignored, the screen timed out, and on some phones the stream came back as a frozen black frame.
- The session status turns to
running <step>in the same call that queues the steps, so a poller never reads a staleidle. - OCR reads text inside dark regions with light cards and chat bubbles correctly, including when the open keyboard shortens the background.
- Library images are cut from the same video frame the scan and the matcher see, instead of a separate screenshot with slightly different pixels.
- Typing is twice as fast: the pause between key taps is now 50 to 150 ms.
- Scan results and the devices list carry more: OCR
confidenceon text entries, and the screendensityof every device. - Deleting an image removes its density file too.
- Unused code, fields and settings were removed across the server.
- Dependencies refreshed: Go 1.27.1, MCP Go SDK 1.8.0, Android Gradle plugin 9.4.1, Gradle 9.8.0, Compose BOM 2026.09.00, Ktor 3.6.0 and the smaller AndroidX updates.
📚 Docs
- The README explains how typing works, the new installation path and where the language files live, and the MCP section covers the new tools, both modes, vision and route arguments.
docs/api.mdaddscapture, route arguments,start_idandend_idslices, image density and gesture scaling, and OCR confidence. The keyboard section is removed.
📦 Downloads
This release includes three files.
- vdroid-scripter, the server, built for macOS on Apple Silicon. It is self-contained now and links only system libraries. It still needs ADB on the host, and the Tesseract language files in
/usr/local/share/vdroid_scripter/tessdata(or whereverTESSDATA_PREFIXpoints). On every other platform, and as the easiest path on macOS too, runinstall.shfrom the repo, which builds everything and downloads the language files for you. See the README for details. - vdroid-mcp, the MCP server, built for macOS on Apple Silicon. It is pure Go, so on any other platform one command builds it:
cd mcp_server && go build -o vdroid-mcp .Register it withclaude mcp add vdroid -- <path>/vdroid-mcp. - vdroid-client-beta-2.0.apk, the example Android client, shrunk and obfuscated. Enable "install from unknown sources", install it, then point it at your server's IP and port (default is :8080). If it refuses to install over an older build, uninstall the old one first.
⚠️ Known limitations
- Hardware navigation buttons (Back, Home, Recents) are still not supported. Use on-screen navigation in your flows.
- There is no authentication and no encryption. Use the server on a trusted local network only.
type_texttypes letters, space and number-row digits. Punctuation, symbols and switching the keyboard language are not supported yet.- Scaling an image by density cannot help when another Android version or vendor skin draws the element differently. Save a second image under a variant name for that.
- The Android client cannot pass arguments, so a route with
%splaceholders runs only through the API or the MCP server. - The stay-awake setting holds only while the phone is powered over USB, AC or wireless charging. If the screen is turned off during a session, close the session and open it again.
- Gesture recording keeps only the first finger and assumes the device's natural portrait orientation.
- No YOLO model is bundled. Drop your own
best.onnxandobj.namesinto theyolo/data directory to useyololandmarks. - Wireless screen streaming compatibility varies by device and manufacturer. See the README compatibility notes.
beta-1.0
VDroid Scripter, beta-1.0
First beta of VDroid Scripter (Visual Android Scripter). This release replaces the script and location model of alpha-3.0 with something simpler and far more flexible: a small library of named images and gestures, and steps that find their target on the live screen. On top of that come a real agent loop for AI (scan, routes and a much smarter MCP server), gesture recording straight from the device, a rebuilt Android client, and a lighter server.
✨ Highlights
- 🧩 Library and steps instead of scripts. Scripts tied to locations are gone. You now curate a flat library of named template images and recorded gestures, and a step applies one event to a target it locates at run time. The same library item works on any screen it appears on.
- 🎯 Landmark chains. A step's target is a chain of landmarks: a library image, a piece of text, or a YOLO class. The chain resolves on a single frame, each landmark nearest to the previous one, so "the toggle next to Show refresh rate" is one step. Duplicates without a unique neighbour resolve in reading order.
- 👆 Generated gestures.
tapandlong_tap, which were only groundwork in alpha-3.0, are now real step events with human-like timing. Newswipe_up,swipe_down,swipe_leftandswipe_rightdraw a curved, jittered swipe from the middle of the screen, andtype_texttypes on the CV-detected keyboard. None of them need anything from the library. - 🔎 Scan, the agent's eyes.
GET /devices/{serial}/scanlooks at one frame and returns every landmark it can name: readable text, YOLO detections, and the library images you ask about. An AI agent works from this JSON and never sees a screenshot. - 🗺️ Routes. Save a flow that worked as a named route, together with the prompt it was dictated with. Run it with one call, or start it from any step id to resume after a failure.
- 🤖 A grown-up MCP server. It moved to
mcp_server/as its own Go module and now has 15 tools. It startsvdroid-scripteron demand, so no second terminal is needed, and it can stop it again. Its built-in instructions teach batching, the locale rule, literal execution, and scan-driven recovery. - 🎙️ Record gestures on the device itself.
POST /devices/{serial}/record, or therecord_actionMCP tool, listens to the touch panel for 5 seconds and saves what you did as a library action. No client app required. - 📱 Rebuilt Android client. Two tabs, Devices and Library. The Library shows Images, Actions and Routes, every item can be deleted, and actions and routes can be played on any connected device from a device picker, several devices at once. Opening the stream on a device is the emergency stop for whatever is running on it.
- ⚡ Lighter, leaner server. The CV socket is gone and nothing runs between requests, which keeps small hosts such as a Raspberry Pi idle while you watch the screen. H.264 decoding is now an in-house shim over libavcodec with no FFmpeg version pin, and OCR runs in-process through libtesseract with no CLI and no temporary files.
💥 Breaking changes
- Scripts, locations and the location graph are removed.
POST /run_scripts,POST /edit_scriptand the script endpoints no longer exist. There is no automatic migration from alpha-3.0: crop your images and record your gestures again in the new library. POST /devices/{serial}/queue_stepsreplacesrun_scripts. The body is a JSON array of steps.delayandtimeoutare milliseconds and are taken literally, with no server-side defaults.- Sessions return two ports, not three. Only
video_portandcontrol_portremain. Rectangles are fetched on demand withGET /devices/{serial}/rectangles. - MCP tools changed.
queue_scriptsbecamequeue_steps.get_location_graphandopen_sessionare gone, because sessions now open by themselves. The build command is nowcd mcp_server && go build -o vdroid-mcp . - Android client. Find text and the live rectangle overlay are replaced by the scan button and the on-demand eye button.
🛠 Reliability and fixes
- The server waits for the first video frame before it runs steps or scans, and keeps serving the last frame on a static screen, so a step never fails just because nothing moved.
- Session statuses and errors carry the step id, for example
step 3: unable to find ..., so a failed route can be resumed from exactly that step. - OCR got several rounds of work: multi-word phrases, dark regions, more languages, and better accuracy overall.
- Stopping the server closes every device session first, so no scrcpy process is left running on the phone.
- The session status gained
recording, and queueing steps on a device that is recording is refused with409. - The Android client now shows the server's error text instead of swallowing failed requests.
install.shpins OpenCV 4 explicitly:opencv@4on Homebrew and the AURopencv4package on Arch.- Dependencies refreshed across the server, the MCP server and the Android client.
📚 Docs
- The README is rewritten as the single guide: how it works, installation, device setup, configuration, the Android client, the MCP server and its instructions, and gesture recording.
docs/server.mdanddocs/android_client.mdare folded into the README.docs/api.mdcovers every new endpoint: steps, scan, rectangles, routes, recording and the new session statuses.
📦 Downloads
This release includes three files.
- vdroid-scripter, the server, built for macOS on Apple Silicon. It needs OpenCV 4, FFmpeg, Tesseract OCR and ADB from Homebrew on the host. On every other platform, and as the easiest path on macOS too, run
install.shfrom the repo, which installs everything and builds it for you. See the README for details. - vdroid-mcp, the MCP server, built for macOS on Apple Silicon. It is pure Go, so on any other platform one command builds it:
cd mcp_server && go build -o vdroid-mcp .Register it withclaude mcp add vdroid -- <path>/vdroid-mcp. - vdroid-client-beta-1.0.apk, the example Android client, shrunk and obfuscated. Enable "install from unknown sources", install it, then point it at your server's IP and port (default is :8080). If it refuses to install over an older build, uninstall the old one first.
⚠️ Known limitations
- Hardware navigation buttons (Back, Home, Recents) are still not supported. Use on-screen navigation in your flows.
- There is no authentication and no encryption. Use the server on a trusted local network only.
- Gesture recording keeps only the first finger and assumes the device's natural portrait orientation.
- No YOLO model is bundled. Drop your own
best.onnxandobj.namesinto theyolo/data directory to useyololandmarks. - Wireless screen streaming compatibility varies by device and manufacturer. See the README compatibility notes.
alpha-3.0
Changelog
alpha-3.0 — 2026-08-12
The AI release: VDroid Scripter can now be driven by an AI agent through a
dedicated MCP server, scripts can be queued in batches, and editing a script's
location no longer loses its template images.
MCP server (new)
- New standalone MCP server (
server/mcpserver/, binaryvdroid-mcp) exposing
the HTTP API as tools for AI-driven flow building over stdio:
list_devices,get_location_graph,open_session,close_session,
get_session_status,queue_scripts,wait_for_session. - Built-in agent instructions: plan flows from the location graph, queue whole
chains in one call, and recover from failures by re-running the last
successful script to re-locate the device. - Builds without the CV native libraries:
go build -o vdroid-mcp ./mcpserver.
Configure the target server withVDROID_URL; register with
claude mcp add vdroid -- <path>/vdroid-mcp.
Server
- Breaking:
POST /run_scriptis replaced byPOST /run_scripts. The new
body is{"serial": "...", "scripts": [{"location": "...", "name": "..."}]}—
a whole flow is validated up front and queued in one request; the session
worker executes the scripts in order and a failure clears the remaining queue. - New
POST /edit_scriptendpoint with body{"prev_location": ..., "script": ...}.
When the location changes, the entire script folder —run.jsonand all
template PNGs — is moved to the new location before the script is rewritten. - Groundwork for generated actions: touch event synthesis in
pkg/models
(GenerateTapEvents,GenerateLongTapEvents) builds scrcpy tap pairs with
human-like timing (50–120 ms tap, 700–900 ms long tap). Not yet exposed
through the API.
Android client
- The streaming session is now closed on the server (
DELETE .../session) when
the user leaves the streaming screen. - Editing a script uses the new
edit_scriptrequest, so moving a script to
another location keeps its template images; clearing the name still deletes
the script. - Script running migrated to the new
run_scriptsbatch request. - Fixed
ScriptItemcard layout: short names stay centered, long names fill
the row before wrapping.
Docs
docs/api.md: documentedrun_scriptsandedit_script, corrected the
session status strings and thenext_locationarray type.docs/server.md: added the MCP server build and registration section.
alpha-2.0
VDroid Scripter, alpha-2.0
Second alpha of VDroid Scripter (Visual Android Scripter). This release is a big step up from alpha-1.0, with full keyboard and text typing support, much smarter text recognition, a reworked and more reliable scripting engine, a revamped Android UI, and a one-command server installer.
✨ Highlights
- ⌨️ On-screen keyboard and text typing. Record and replay typing as part of a script. The server detects the on-screen keyboard with computer vision and types character by character, including numbers. You can review and correct detected keys right from the app.
- 🔎 Smarter text recognition (OCR). Text is now found reliably on both light and dark backgrounds thanks to automatic contrast handling, and searches are locale aware.
- 👁️ Visibility steps. Steps can now check that a template image or text is visible on screen, not just tap it, so you can build conditional, more robust flows.
- 🧠 Reworked scripting engine. A new step and flags format, far more reliable multi-step playback, scripts can also run on an already active mirror session.
- 📱 Revamped Android streaming UI. A cleaner action menu, a dedicated keyboard menu, and selection handling that survives screen and orientation changes.
- ⚙️ One-command server install. The new
install.shinstalls every dependency, builds the server, and puts the vdroid-scripter binary on your PATH (macOS, Arch, Debian/Ubuntu, Fedora).
🛠 Reliability and fixes
- Automatically retries the scrcpy connection after a device reboot.
- Fixed a device list serialization error when no devices are connected.
- Numerous fixes for running multi-step scripts end to end.
- Keyboard menu, locale handling, and menu state bugfixes.
📚 Docs
- Full HTTP API reference added (docs/api.md).
- Updated server setup guide (docs/server.md).
📦 Downloads
This release includes two files.
- vdroid-scripter, the server. It needs OpenCV, FFmpeg, Tesseract OCR and ADB on the host. The easiest path is to run
install.shfrom the repo, which installs everything and builds it for you. See docs/server.md for details. - vdroid-scripter.apk, the example Android client. Enable "install from unknown sources", install it, then point it at your server's IP and port (default is :8080).
⚠️ Known limitations
- Hardware navigation buttons (Back, Home, Recents) are not supported yet, so devices that rely on physical or capacitive navigation keys cannot be navigated back from the client. This is planned for a future release.
- Wireless screen streaming compatibility varies by device and manufacturer. See the README compatibility notes.