Let AI see, understand, and control your Flutter apps
FlutterPilot is an MCP (Model Context Protocol) server paired with a lightweight Dart package that gives AI assistants -- Claude Desktop, Claude Code, or any MCP-compatible client -- the ability to see, understand, and interact with running Flutter debug apps through natural language.
Point Claude at your running app and ask it to take screenshots, inspect the widget tree, tap buttons, type into text fields, scroll, trigger hot reload, or evaluate Dart expressions -- all without writing test code or manually navigating the UI.
- Screenshot capture -- see your app exactly as users do (PNG, returned as an image to the AI)
- Widget tree inspection -- understand app structure as structured JSON, with configurable depth
- Semantics/accessibility tree -- compact, AI-friendly representation with labels, actions, and positions
- Interactive widget discovery -- list all buttons, text fields, and tappable elements with coordinates
- Tap, scroll, type, long press -- full interaction via coordinates or smart finders (by text, key, type, or semantics label)
- Hot reload -- apply code changes instantly without restarting app state
- Dart expression evaluation -- run arbitrary Dart code in the context of the running app
- Works everywhere -- Claude Desktop, Claude Code, and any MCP client
# pubspec.yaml
dev_dependencies:
flutter_pilot: ^0.1.0// lib/main.dart
import 'package:flutter_pilot/flutter_pilot.dart';
void main() {
FlutterPilot.ensureInitialized();
runApp(const MyApp());
}Claude Desktop -- add to your claude_desktop_config.json:
{
"mcpServers": {
"flutter-pilot": {
"command": "npx",
"args": ["-y", "@flutterpilot/mcp-server"]
}
}
}Claude Code -- add to your project's .mcp.json:
{
"mcpServers": {
"flutter-pilot": {
"command": "npx",
"args": ["-y", "@flutterpilot/mcp-server"]
}
}
}flutter run --debug
# Copy the VM service URI from the console output, e.g.:
# A Dart VM Service on ... is available at: http://127.0.0.1:56789/ABCDEF=/Then tell Claude:
"Connect to my Flutter app at ws://127.0.0.1:56789/ABCDEF=/ws and take a screenshot"
+-----------------+ +-------------------+ +--------------------+
| | | | | |
| Flutter App | <---> | Dart VM Service | <---> | @flutterpilot/ |
| (debug mode) | WS | (WebSocket) | WS | core (TypeScript) |
| | | | | |
+-----------------+ +-------------------+ +--------+-----------+
|
| in-process
|
+--------v-----------+
| |
| @flutterpilot/ |
| mcp-server |
| (MCP over stdio) |
| |
+--------+-----------+
|
| MCP protocol
|
+--------v-----------+
| |
| Claude Desktop / |
| Claude Code / |
| Any MCP Client |
| |
+--------------------+
FlutterPilot connects to the Dart VM Service exposed by every Flutter debug app. The TypeScript core library (@flutterpilot/core) communicates over WebSocket using the Dart VM Service Protocol to take screenshots, walk the widget tree, dispatch gestures, and evaluate expressions. The MCP server wraps this library and exposes each capability as an MCP tool that Claude can call.
The optional flutter_pilot Dart package registers VM service extensions inside the running app to enable advanced features like semantics tree access, widget finding, and gesture dispatch that are not available through the standard VM protocol alone.
| Tool | Description | Key Parameters |
|---|---|---|
flutter_connect |
Connect to a running Flutter debug app | vmServiceUri (required) |
flutter_disconnect |
Disconnect from the current app | -- |
flutter_screenshot |
Take a PNG screenshot of the app | -- |
flutter_widget_tree |
Get the widget tree as structured JSON | maxDepth |
flutter_semantics |
Get the accessibility/semantics tree | maxDepth |
flutter_interactive |
List all interactive widgets with positions | -- |
flutter_tap |
Tap a widget by coordinates or finder | x, y or findBy + value |
flutter_enter_text |
Type text into the focused text field | text (required) |
flutter_scroll |
Scroll at a position | x, y, dy (required); dx |
flutter_long_press |
Long press a widget | x, y or findBy + value |
flutter_hot_reload |
Trigger hot reload | -- |
flutter_evaluate |
Evaluate a Dart expression at runtime | expression (required) |
| Package | Description | Version |
|---|---|---|
flutter_pilot (Dart) |
Lightweight Flutter dev dependency -- registers VM service extensions for interaction | 0.1.0 |
@flutterpilot/core (TypeScript) |
Client library for the Dart VM Service Protocol | 0.1.0 |
@flutterpilot/mcp-server (TypeScript) |
MCP server that exposes Flutter tools to Claude | 0.1.0 |
Once connected, try asking Claude:
- "Take a screenshot of the app" -- Claude will capture and display the current UI
- "What interactive widgets are on screen?" -- lists every button, text field, and tappable widget with its position
- "Tap the Increment button" -- finds the button by its label text and taps it
- "Type 'hello world' into the text field" -- focuses and types into the input
- "Hot reload the app" -- applies your latest code changes without losing state
- "Evaluate
DateTime.now().toString()in the running app" -- executes arbitrary Dart expressions and returns the result
| Feature | FlutterPilot | marionette_mcp | mcp_flutter | flutter-skill |
|---|---|---|---|---|
| Language | TypeScript | Dart | Dart | TypeScript |
| MCP SDK | Official TS SDK | Custom | Custom | N/A |
| Screenshot capture | Yes | Yes | Yes | No |
| Widget tree inspection | Yes | Yes | Yes | No |
| Semantics tree | Yes | No | No | No |
| Interactive widget listing | Yes | No | No | No |
| Tap / scroll / type | Yes | Yes | Yes | No |
| Widget finders (text, key, type, semantics) | Yes | Partial | Partial | No |
| Long press | Yes | No | No | No |
| Hot reload | Yes | Yes | No | No |
| Dart eval | Yes | No | No | No |
| Install via npm | npx @flutterpilot/mcp-server |
Manual | Manual | Manual |
| Web dashboard (planned) | Yes | No | No | No |
| SaaS model (planned) | Yes | No | No | No |
- Node.js >= 18
- pnpm >= 9
git clone https://github.com/geeksera/flutter-pilot.git
cd flutter-pilot
pnpm install
pnpm buildnode packages/mcp-server/dist/index.jsflutter-pilot/
packages/
core/ # @flutterpilot/core -- TypeScript client library
mcp-server/ # @flutterpilot/mcp-server -- MCP server
dart/ # flutter_pilot -- Dart package for Flutter apps
examples/
counter_app/ # Example Flutter app with flutter_pilot integration
docs/ # Documentation
- Phase 1 (current): Core MCP server with screenshot, inspection, interaction, and dev tools
- Phase 2: Web dashboard for visual session monitoring and replay
- Phase 3: SaaS platform with team collaboration, session history, and analytics
- Phase 4: CI/CD integration -- AI-driven visual regression testing and automated QA
Contributions are welcome. See CONTRIBUTING.md for guidelines on how to get started.
If you find a bug or have a feature request, please open an issue.
MIT -- see LICENSE for details.