Skip to content

beta-2.0

Latest

Choose a tag to compare

@VaasKout VaasKout released this 02 Oct 13:02

VDroid Scripter, beta-2.0

Second beta of VDroid Scripter (Visual Android Scripter). beta-1.0 gave you a library, steps and routes. This release makes them travel: one library now serves phones of different sizes and densities, one route serves any text you pass to it, and typing needs no keyboard setup at all. The AI seat grew the most: two working modes, the ability to see the screen and curate its own icons, and a much smaller bill per turn. Underneath, OCR reads screens it used to miss and the server no longer depends on your distribution's libraries.

✨ Highlights

  • ⌨️ Typing with no keyboard setup. type_text now reads the open keyboard off the live frame, recognises its layout by the language of the step (QWERTY, AZERTY, QWERTZ, ЙЦУКЕН and the other European layouts) and taps the keys. Capitals go through Shift, digits through the number row, and the locale numeric types on a numeric keypad or an app's own dial pad. Nothing about keyboards is stored any more, and the key editor is gone.
  • 🧳 A library that travels between phones. A recorded gesture is scaled to the screen it is replayed on, so it covers the same share of the display. A template image is saved with the screen density of its device and resized to the density of the phone it is searched on. One crop and one recording serve phones of different resolutions, as long as they draw the element the same way.
  • 🧮 Routes with arguments. Put %s in the text a route types or looks for, and pass the values when you run it: args=Молоко&args=Молоко. One route now adds any item to a basket or messages any contact. args_locale names the language of the values, and a wrong number of values is refused before anything runs.
  • ✂️ Run any slice of a route. end_id joins start_id, both inclusive. Run a route up to a step, from a step, or just the part in between.
  • 🧭 Two modes for the AI. The MCP server starts in navigator: it records nothing, treats a request as a goal, and reaches it by chaining the routes you already have, picking the slice and the arguments itself instead of asking which route to run. Switch it to explorer with a route name and it works a goal out on unknown screens, recording every step that succeeded into that route as it goes.
  • 👁️ Eyes for the agent. GET /devices/{serial}/capture returns the current frame as JPEG or PNG. A vision-capable agent uses the new capture tool where scan is not enough, and save_image to crop an icon into the library by itself. scan stays first and stays the source of landmark values.
  • 🪙 A cheaper, faster agent loop. scan hands the agent a compact table, about three times fewer tokens than the JSON, and leaves out low-confidence noise. queue_steps and run_route wait for the batch and answer with the result, so there is no polling. The built-in instructions and tool descriptions were cut down to about 4.6k tokens per turn. Four new tools bring the total to 19: capture, save_image, edit_route and set_mode.
  • 🔤 OCR that reads more screens. Text is now read twice, with a global and a local binarization, and the results are merged. White launcher labels over a photo wallpaper, which used to return nothing, are found. Every word carries a confidence, and ё matches е, so Пятерочка finds Пятёрочка.
  • 📦 A self-contained server. install.sh builds OpenCV, FFmpeg, Leptonica and Tesseract from pinned sources as static libraries and links them into the binary. A distribution upgrade to OpenCV 5 or a new FFmpeg can no longer break it, and the Tesseract language files are downloaded for you.

💥 Breaking changes

  • The keyboard endpoints are removed. GET /devices/{serial}/keyboard, POST /devices/{serial}/edit_keyboard, GET /devices/{serial}/reset_keyboard and GET /devices/{serial}/delete_button no longer exist, along with the KEYBOARDS_DIR setting and the keyboard editor in the Android client. type_text needs the keyboard already open and showing the language of the step. Punctuation and symbols fail the step.
  • Routes have no prompt. The field is gone from the model, the API and save_route. Routes saved by beta-1.0 still load, and the prompt is dropped the next time they are saved.
  • %s in a text landmark is a placeholder. A route that contains one must be run with matching args. Saving a route clears the locale of every landmark that holds a %s, because the language comes with the run.
  • POST /save_image crops the video frame, not an ADB screenshot. Rectangles are in video-frame coordinates, the same ones scan, rectangles and capture use. Each image now has a small images/<name>.json next to it with the density.
  • MCP tools behave differently. queue_steps and run_route block until the batch is done, for up to three minutes, and wait_for_session is only for what runs longer. scan returns a table instead of JSON. Routes are recorded in explorer mode or written with save_route, and the agent changes modes only when you tell it to.
  • The server writes no log files. The LOGS setting is gone and logs go to stdout. When the MCP server starts vdroid-scripter for you, its output is appended to logs/server.log in the data directory.
  • install.sh no longer uses distribution packages for the native libraries. The first run builds them, which takes ten to fifteen minutes. Later runs reuse the result.

🛠 Reliability and fixes

  • The phone's screen stays awake while a session is open. Before, scrcpy's stay-awake flag was silently ignored, the screen timed out, and on some phones the stream came back as a frozen black frame.
  • The session status turns to running <step> in the same call that queues the steps, so a poller never reads a stale idle.
  • OCR reads text inside dark regions with light cards and chat bubbles correctly, including when the open keyboard shortens the background.
  • Library images are cut from the same video frame the scan and the matcher see, instead of a separate screenshot with slightly different pixels.
  • Typing is twice as fast: the pause between key taps is now 50 to 150 ms.
  • Scan results and the devices list carry more: OCR confidence on text entries, and the screen density of every device.
  • Deleting an image removes its density file too.
  • Unused code, fields and settings were removed across the server.
  • Dependencies refreshed: Go 1.27.1, MCP Go SDK 1.8.0, Android Gradle plugin 9.4.1, Gradle 9.8.0, Compose BOM 2026.09.00, Ktor 3.6.0 and the smaller AndroidX updates.

📚 Docs

  • The README explains how typing works, the new installation path and where the language files live, and the MCP section covers the new tools, both modes, vision and route arguments.
  • docs/api.md adds capture, route arguments, start_id and end_id slices, image density and gesture scaling, and OCR confidence. The keyboard section is removed.

📦 Downloads

This release includes three files.

  • vdroid-scripter, the server, built for macOS on Apple Silicon. It is self-contained now and links only system libraries. It still needs ADB on the host, and the Tesseract language files in /usr/local/share/vdroid_scripter/tessdata (or wherever TESSDATA_PREFIX points). On every other platform, and as the easiest path on macOS too, run install.sh from the repo, which builds everything and downloads the language files for you. See the README for details.
  • vdroid-mcp, the MCP server, built for macOS on Apple Silicon. It is pure Go, so on any other platform one command builds it: cd mcp_server && go build -o vdroid-mcp . Register it with claude mcp add vdroid -- <path>/vdroid-mcp.
  • vdroid-client-beta-2.0.apk, the example Android client, shrunk and obfuscated. Enable "install from unknown sources", install it, then point it at your server's IP and port (default is :8080). If it refuses to install over an older build, uninstall the old one first.

⚠️ Known limitations

  • Hardware navigation buttons (Back, Home, Recents) are still not supported. Use on-screen navigation in your flows.
  • There is no authentication and no encryption. Use the server on a trusted local network only.
  • type_text types letters, space and number-row digits. Punctuation, symbols and switching the keyboard language are not supported yet.
  • Scaling an image by density cannot help when another Android version or vendor skin draws the element differently. Save a second image under a variant name for that.
  • The Android client cannot pass arguments, so a route with %s placeholders runs only through the API or the MCP server.
  • The stay-awake setting holds only while the phone is powered over USB, AC or wireless charging. If the screen is turned off during a session, close the session and open it again.
  • Gesture recording keeps only the first finger and assumes the device's natural portrait orientation.
  • No YOLO model is bundled. Drop your own best.onnx and obj.names into the yolo/ data directory to use yolo landmarks.
  • Wireless screen streaming compatibility varies by device and manufacturer. See the README compatibility notes.

Full changelog