agent-flutter

CLI for AI agents to control Flutter apps via Dart VM Service + Marionette. Same @ref + snapshot workflow as agent-device and agent-browser.

Prerequisites

Your Flutter app needs Marionette

Add marionette_flutter to your Flutter app:

# pubspec.yaml
dev_dependencies:
  marionette_flutter: ^0.3.0

Initialize in your app's main.dart (debug mode only):

import 'package:marionette_flutter/marionette_flutter.dart';

void main() {
  assert(() {
    MarionetteBinding.ensureInitialized();
    return true;
  }());
  runApp(const MyApp());
}

Runtime requirements

• Node.js 18+
• ADB accessible (adb devices shows your target)
• Flutter app running via flutter run (debug or profile mode — exposes Dart VM Service)

Note: adb install of a pre-built APK won't work — the Dart VM Service is only exposed when launched via flutter run.

Quick start

# Install
npm install -g agent-flutter-cli

# Verify prerequisites
agent-flutter doctor

# Connect to running Flutter app (auto-detects VM Service URI from logcat)
agent-flutter connect

# See all widgets with refs
agent-flutter snapshot -i

# Interact
agent-flutter press @e3
agent-flutter fill @e5 "hello world"

# Disconnect
agent-flutter disconnect

If you already have a VM Service URI:

agent-flutter connect ws://127.0.0.1:38047/abc=/ws

Command reference

Command	Description	Example
connect [uri]	Connect to Flutter VM Service	agent-flutter connect
disconnect	Disconnect from Flutter app	agent-flutter disconnect
status	Show connection state	agent-flutter status
snapshot [-i] [-c] [-d N] [--diff]	Capture widget snapshot with refs	agent-flutter snapshot -i
press <ref> | <x> <y>	Tap by ref, coordinates, or ref via ADB (--adb)	agent-flutter press @e3
fill <ref> <text>	Enter text by ref	agent-flutter fill @e5 "hello"
get <property> <ref>	Read text, type, key, or attrs	agent-flutter get attrs @e3
find <locator> <value> [action] [arg]	Find by key, text, or type	agent-flutter find text "Submit" press
wait <condition> [target]	Wait for exists, visible, text, gone, or delay ms	agent-flutter wait text "Welcome"
is <condition> <ref>	Assert exists or visible	agent-flutter is exists @e3
scroll <target>	Scroll to ref or direction	agent-flutter scroll down
swipe <direction>	ADB swipe gesture	agent-flutter swipe left
back	Android back button (ADB)	agent-flutter back
home	Android home button (ADB)	agent-flutter home
screenshot [path]	Capture screenshot	agent-flutter screenshot /tmp/screen.png
reload	Trigger Flutter hot reload	agent-flutter reload
logs	Get Flutter app logs	agent-flutter logs
schema [command]	Output command schema (JSON)	agent-flutter schema press
doctor	Check prerequisites and diagnose issues	agent-flutter doctor

Global flags

Flag	Description
--device <id>	ADB device ID (default: emulator-5554)
--json	Force JSON output
--no-json	Force human-readable output
--dry-run	Resolve target without executing
--help	Show help (--help --json returns schema)

Snapshot format

@e1 [button] "Submit"  key=submit_btn
@e2 [textfield] "Email"  key=email_input
@e3 [label] "Welcome back"

• Refs are sequential per snapshot (@e1, @e2, ...) and reset on each snapshot call.
• Re-run snapshot after UI-changing commands (press, fill, scroll, reload) — refs may shift.
• snapshot --json returns objects with ref, type, label, key, visible, bounds.

Environment variables

Variable	Purpose	Default
AGENT_FLUTTER_DEVICE	ADB device ID	emulator-5554
AGENT_FLUTTER_URI	VM Service URI for connect	auto-detect from logcat
AGENT_FLUTTER_TIMEOUT	Default wait timeout (ms)	10000
AGENT_FLUTTER_JSON	JSON output mode (1)	unset

Precedence: CLI flag > env var > built-in default.

Exit codes

Code	Meaning
0	Success
1	Assertion false (is command only)
2	Error

Development

This repo is the publish target. Source of truth is beastoin/autoloop — all code changes go through autoloop's phase-gated build loop (program → implement → eval → keep/revert), then get copied here for npm publish.

Do not edit this repo directly. To make changes, add a new phase program in autoloop.

# For local testing only
git clone https://github.com/beastoin/agent-flutter.git
cd agent-flutter
npm install

# Typecheck
npm run typecheck

# Unit tests
npm run test:unit

# Build (bundle TypeScript to dist/)
npm run build

# Run locally
node dist/cli.mjs --help

Related tools

Tool	Platform	Transport
agent-flutter	Flutter apps	Dart VM Service + Marionette
autoloop	Native mobile (Android/iOS) + autoloop build system	daemon + platform runners
agent-browser	Web apps	browser automation

All three share the same snapshot → @ref → press/fill/wait/is workflow.

License

MIT