Repository navigation
VDroid Scripter, beta-2.0
Second beta of VDroid Scripter (Visual Android Scripter). beta-1.0 gave you a library, steps and routes. This release makes them travel: one library now serves phones of different sizes and densities, one route serves any text you pass to it, and typing needs no keyboard setup at all. The AI seat grew the most: two working modes, the ability to see the screen and curate its own icons, and a much smaller bill per turn. Underneath, OCR reads screens it used to miss and the server no longer depends on your distribution's libraries.
✨ Highlights
- ⌨️ Typing with no keyboard setup.
type_textnow reads the open keyboard off the live frame, recognises its layout by the language of the step (QWERTY, AZERTY, QWERTZ, ЙЦУКЕН and the other European layouts) and taps the keys. Capitals go through Shift, digits through the number row, and the localenumerictypes on a numeric keypad or an app's own dial pad. Nothing about keyboards is stored any more, and the key editor is gone. - 🧳 A library that travels between phones. A recorded gesture is scaled to the screen it is replayed on, so it covers the same share of the display. A template image is saved with the screen density of its device and resized to the density of the phone it is searched on. One crop and one recording serve phones of different resolutions, as long as they draw the element the same way.
- 🧮 Routes with arguments. Put
%sin the text a route types or looks for, and pass the values when you run it:args=Молоко&args=Молоко. One route now adds any item to a basket or messages any contact.args_localenames the language of the values, and a wrong number of values is refused before anything runs. - ✂️ Run any slice of a route.
end_idjoinsstart_id, both inclusive. Run a route up to a step, from a step, or just the part in between. - 🧭 Two modes for the AI. The MCP server starts in navigator: it records nothing, treats a request as a goal, and reaches it by chaining the routes you already have, picking the slice and the arguments itself instead of asking which route to run. Switch it to explorer with a route name and it works a goal out on unknown screens, recording every step that succeeded into that route as it goes.
- 👁️ Eyes for the agent.
GET /devices/{serial}/capturereturns the current frame as JPEG or PNG. A vision-capable agent uses the newcapturetool wherescanis not enough, andsave_imageto crop an icon into the library by itself.scanstays first and stays the source of landmark values. - 🪙 A cheaper, faster agent loop.
scanhands the agent a compact table, about three times fewer tokens than the JSON, and leaves out low-confidence noise.queue_stepsandrun_routewait for the batch and answer with the result, so there is no polling. The built-in instructions and tool descriptions were cut down to about 4.6k tokens per turn. Four new tools bring the total to 19:capture,save_image,edit_routeandset_mode. - 🔤 OCR that reads more screens. Text is now read twice, with a global and a local binarization, and the results are merged. White launcher labels over a photo wallpaper, which used to return nothing, are found. Every word carries a confidence, and
ёmatchesе, soПятерочкаfindsПятёрочка. - 📦 A self-contained server.
install.shbuilds OpenCV, FFmpeg, Leptonica and Tesseract from pinned sources as static libraries and links them into the binary. A distribution upgrade to OpenCV 5 or a new FFmpeg can no longer break it, and the Tesseract language files are downloaded for you.
💥 Breaking changes
- The keyboard endpoints are removed.
GET /devices/{serial}/keyboard,POST /devices/{serial}/edit_keyboard,GET /devices/{serial}/reset_keyboardandGET /devices/{serial}/delete_buttonno longer exist, along with theKEYBOARDS_DIRsetting and the keyboard editor in the Android client.type_textneeds the keyboard already open and showing the language of the step. Punctuation and symbols fail the step. - Routes have no
prompt. The field is gone from the model, the API andsave_route. Routes saved by beta-1.0 still load, and the prompt is dropped the next time they are saved. %sin a text landmark is a placeholder. A route that contains one must be run with matchingargs. Saving a route clears thelocaleof every landmark that holds a%s, because the language comes with the run.POST /save_imagecrops the video frame, not an ADB screenshot. Rectangles are in video-frame coordinates, the same onesscan,rectanglesandcaptureuse. Each image now has a smallimages/<name>.jsonnext to it with the density.- MCP tools behave differently.
queue_stepsandrun_routeblock until the batch is done, for up to three minutes, andwait_for_sessionis only for what runs longer.scanreturns a table instead of JSON. Routes are recorded in explorer mode or written withsave_route, and the agent changes modes only when you tell it to. - The server writes no log files. The
LOGSsetting is gone and logs go to stdout. When the MCP server startsvdroid-scripterfor you, its output is appended tologs/server.login the data directory. install.shno longer uses distribution packages for the native libraries. The first run builds them, which takes ten to fifteen minutes. Later runs reuse the result.
🛠 Reliability and fixes
- The phone's screen stays awake while a session is open. Before, scrcpy's stay-awake flag was silently ignored, the screen timed out, and on some phones the stream came back as a frozen black frame.
- The session status turns to
running <step>in the same call that queues the steps, so a poller never reads a staleidle. - OCR reads text inside dark regions with light cards and chat bubbles correctly, including when the open keyboard shortens the background.
- Library images are cut from the same video frame the scan and the matcher see, instead of a separate screenshot with slightly different pixels.
- Typing is twice as fast: the pause between key taps is now 50 to 150 ms.
- Scan results and the devices list carry more: OCR
confidenceon text entries, and the screendensityof every device. - Deleting an image removes its density file too.
- Unused code, fields and settings were removed across the server.
- Dependencies refreshed: Go 1.27.1, MCP Go SDK 1.8.0, Android Gradle plugin 9.4.1, Gradle 9.8.0, Compose BOM 2026.09.00, Ktor 3.6.0 and the smaller AndroidX updates.
📚 Docs
- The README explains how typing works, the new installation path and where the language files live, and the MCP section covers the new tools, both modes, vision and route arguments.
docs/api.mdaddscapture, route arguments,start_idandend_idslices, image density and gesture scaling, and OCR confidence. The keyboard section is removed.
📦 Downloads
This release includes three files.
- vdroid-scripter, the server, built for macOS on Apple Silicon. It is self-contained now and links only system libraries. It still needs ADB on the host, and the Tesseract language files in
/usr/local/share/vdroid_scripter/tessdata(or whereverTESSDATA_PREFIXpoints). On every other platform, and as the easiest path on macOS too, runinstall.shfrom the repo, which builds everything and downloads the language files for you. See the README for details. - vdroid-mcp, the MCP server, built for macOS on Apple Silicon. It is pure Go, so on any other platform one command builds it:
cd mcp_server && go build -o vdroid-mcp .Register it withclaude mcp add vdroid -- <path>/vdroid-mcp. - vdroid-client-beta-2.0.apk, the example Android client, shrunk and obfuscated. Enable "install from unknown sources", install it, then point it at your server's IP and port (default is :8080). If it refuses to install over an older build, uninstall the old one first.
⚠️ Known limitations
- Hardware navigation buttons (Back, Home, Recents) are still not supported. Use on-screen navigation in your flows.
- There is no authentication and no encryption. Use the server on a trusted local network only.
type_texttypes letters, space and number-row digits. Punctuation, symbols and switching the keyboard language are not supported yet.- Scaling an image by density cannot help when another Android version or vendor skin draws the element differently. Save a second image under a variant name for that.
- The Android client cannot pass arguments, so a route with
%splaceholders runs only through the API or the MCP server. - The stay-awake setting holds only while the phone is powered over USB, AC or wireless charging. If the screen is turned off during a session, close the session and open it again.
- Gesture recording keeps only the first finger and assumes the device's natural portrait orientation.
- No YOLO model is bundled. Drop your own
best.onnxandobj.namesinto theyolo/data directory to useyololandmarks. - Wireless screen streaming compatibility varies by device and manufacturer. See the README compatibility notes.