Skip to content

## AutoCua v0.1.6: Android is here

Choose a tag to compare

@FunctionFreak FunctionFreak released this 30 Oct 22:56
· 28 commits to main since this release

This release brings Android to AutoCua: a prebuilt Android agent in every wheel, phones in the desktop app, a phone sub-agent for computer use, and an emulator for terminal runs. Phones can also connect over Wi-Fi in the app, and Telegram tasks now show up as normal chat messages.

Highlights

  • Android agent prebuilt in every wheel. No Rust and no scrcpy needed. The phone helper takes the screenshots itself.
  • Android in the desktop app. Connect a phone in Settings > Connect Device > Android, then pick Mobile use > Android in the chat box.
  • android_agent sub-agent. The computer-use agent can hand phone work to an Android sub-agent on macOS, Linux and Windows, shown live in its own box in the chat.
  • Phones over Wi-Fi in the app. Android phones you connected once over USB (Android 11+, Wireless debugging on). iPhones paired with Xcode on the same Wi-Fi.
  • Android emulator for terminal runs. device="simulation" is now the default for Android. Android 11 to 17, phone or tablet, your .apk installed, several copies in parallel.
  • Telegram tasks as chat messages. A task sent from Telegram runs like a typed one, with a small "sent via: Telegram" pill.

Desktop app

  • New Mobile via: Wi-Fi / Wired toggle in Mobile use and Computer use. Wired is the default, and there is no silent fallback either way.
  • Phone pairing in the chat box: the box is read only while a phone pairs, a circle turns beside the mode picker, and if pairing fails the error is shown with a red arrow to try again.
  • The phone window (simulator and emulator view) is frameless on Windows and Linux, with the video under the window buttons. Resize, snap and maximize all work.
  • The phone window now opens on Linux, and you can drag, resize and maximize it on Wayland.
  • The Android emulator shows in the same window as iOS Simulator runs, several side by side.
  • iOS options show only on a Mac.
  • The Rust agents build and adb is offered in the terminal before any window opens.
  • Terminal runs use the API key saved in Settings when .env has none.

Agents and tools

  • Android tools: web, vault (logins in the "android" section, PIN pads typed for you), back (system Back), video_player (streaming, pause, play) and ground_scroll.
  • ground_scroll on iOS and Android: swipe from any point of the screenshot, for lists with no element id.
  • iPhone over Wi-Fi: WebDriverAgent starts through Xcode. When several iPhones are paired, the unlocked one goes first.
  • Taps, typing and scrolls on Android no longer block the app while adb works.

Setup and install

  • Nothing is downloaded without a yes. One question on the terminal before adb, WebDriverAgent, Chrome, the OCR models, the emulator and similar downloads, each from its maker's own server. Set AutoCua_AUTO_DOWNLOAD=1 for CI.
  • An adb you installed yourself (Android Studio, Homebrew, apt) is used as it is.
  • iOS setup moved into setup_mac.sh behind a timed question. The iPhone disk image comes from Xcode's own copy.
  • THIRD_PARTY_LICENSES.txt ships in the wheel. Rust is pinned to 1.97.1.

Fixes

  • Every print in the app could fail with I/O operation on closed file. Fixed.
  • Telegram runs were saved but showed nothing live in the window. Fixed.
  • A run without save_conversation=True wiped the last saved conversation. Now only debug/ is cleared on every run (all six agents).
  • The iPhone pick took the newest paired phone even when another one was plugged in. It now takes the connected one.
  • A dozing Android phone on Wi-Fi is found again (it used to drop out of adb's list).
  • "Device connected" no longer stays as the chat box hint after a run.
  • A late answer from an old iOS pick no longer flips the picker.
  • Closing the phone window by its button no longer crashes at exit.
  • A run that fails while starting keeps the reopened chat's saved steps.
  • Parallel web and emulator runs read their output in UTF-8, so emoji lines no longer break on Windows.
  • The Settings cog no longer jiggles on a click.

Install

pip install -U autocua

Prebuilt wheels (Python 3.10+):

Platform Wheel
macOS, Apple silicon macosx_11_0_arm64
Windows x64 win_amd64
Linux x86_64 (glibc 2.17+) manylinux_2_17_x86_64

Other platforms build from source and need Rust.

Full changelog: 9205ff9...v0.1.6