Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ Flags:
- `--device <name>`
- `--udid <udid>` (iOS)
- `--serial <serial>` (Android)
- `--activity <component>` (Android; package/Activity or package/.Activity)
- `--activity <component>` (Android app launch only; package/Activity or package/.Activity; not for URL opens)
- `--session <name>`
- `--verbose` for daemon and runner logs
- `--json` for structured output
Expand All @@ -117,7 +117,7 @@ npx skills add https://github.com/callstackincubator/agent-device --skill agent-
Sessions:
- `open` starts a session. Without args boots/activates the target device/simulator without launching an app.
- All interaction commands require an open session.
- If a session is already open, `open <app>` switches the active app and updates the session app bundle.
- If a session is already open, `open <app|url>` switches the active app or opens a deep link URL.
- `close` stops the session and releases device resources. Pass an app to close it explicitly, or omit to just close the session.
- Use `--session <name>` to manage multiple sessions.
- Session scripts are written to `~/.agent-device/sessions/<session>-<timestamp>.ad` when recording is enabled with `--save-script`.
Expand All @@ -126,10 +126,21 @@ Sessions:
Navigation helpers:
- `boot --platform ios|android` ensures the target is ready without launching an app.
- Use `boot` mainly when starting a new session and `open` fails because no booted simulator/emulator is available.
- `open [app]` already boots/activates the selected target when needed.
- `open [app|url]` already boots/activates the selected target when needed.
- `reinstall <app> <path>` uninstalls and installs the app binary in one command (Android + iOS simulator in v1).
- `reinstall` accepts package/bundle id style app names and supports `~` in paths.

Deep links:
- `open <url>` supports deep links with `scheme://...`.
- Android opens deep links via `VIEW` intent.
- iOS deep link open is simulator-only in v1.
- `--activity` cannot be combined with URL opens.

```bash
agent-device open "myapp://home" --platform android
agent-device open "https://example.com" --platform ios
```

Find (semantic):
- `find <text> <action> [value]` finds by any text (label/value/identifier) using a scoped snapshot.
- `find text|label|value|role|id <value> <action> [value]` for specific locators.
Expand Down
14 changes: 9 additions & 5 deletions skills/agent-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ npx -y agent-device

## Core workflow

1. Open app: `open [app]` (`open` handles target selection + boot/activation in the normal flow)
1. Open app or deep link: `open [app|url]` (`open` handles target selection + boot/activation in the normal flow)
2. Snapshot: `snapshot` to get refs from accessibility tree
3. Interact using refs (`click @ref`, `fill @ref "text"`)
4. Re-snapshot after navigation/UI changes
Expand All @@ -41,8 +41,10 @@ npx -y agent-device
agent-device boot # Ensure target is booted/ready without opening app
agent-device boot --platform ios # Boot iOS simulator
agent-device boot --platform android # Boot Android emulator/device target
agent-device open [app] # Boot device/simulator; optionally launch app
agent-device open [app] --activity com.example/.MainActivity # Android: open specific activity
agent-device open [app|url] # Boot device/simulator; optionally launch app or deep link URL
agent-device open [app] --activity com.example/.MainActivity # Android: open specific activity (app targets only)
agent-device open "myapp://home" --platform android # Android deep link
agent-device open "https://example.com" --platform ios # iOS simulator deep link
agent-device close [app] # Close app or just end session
agent-device reinstall <app> <path> # Uninstall + install app in one command
agent-device session list # List active sessions
Expand Down Expand Up @@ -168,10 +170,12 @@ agent-device apps --platform android --user-installed
- Prefer `snapshot -i` to reduce output size.
- On iOS, `xctest` is the default and does not require Accessibility permission.
- If XCTest returns 0 nodes (foreground app changed), agent-device falls back to AX when available.
- `open <app>` can be used within an existing session to switch apps and update the session bundle id.
- `open <app|url>` can be used within an existing session to switch apps or open deep links.
- `open <app>` updates session app bundle context; URL opens do not set an app bundle id.
- If AX returns the Simulator window or empty tree, restart Simulator or use `--backend xctest`.
- Use `--session <name>` for parallel sessions; avoid device contention.
- Use `--activity <component>` on Android to launch a specific activity (e.g. TV apps with LEANBACK).
- Use `--activity <component>` on Android to launch a specific activity (e.g. TV apps with LEANBACK); do not combine with URL opens.
- iOS deep-link opens are simulator-only in v1.
- Use `fill` when you want clear-then-type semantics.
- Use `type` when you want to append/enter text without clearing.
- On Android, prefer `fill` for important fields; it verifies entered text and retries once when IME reorders characters.
Expand Down
14 changes: 14 additions & 0 deletions src/core/__tests__/open-target.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { isDeepLinkTarget } from '../open-target.ts';

test('isDeepLinkTarget accepts URL-style deep links', () => {
assert.equal(isDeepLinkTarget('myapp://home'), true);
assert.equal(isDeepLinkTarget('https://example.com'), true);
});

test('isDeepLinkTarget rejects app identifiers and malformed URLs', () => {
assert.equal(isDeepLinkTarget('com.example.app'), false);
assert.equal(isDeepLinkTarget('settings'), false);
assert.equal(isDeepLinkTarget('http:/x'), false);
});
2 changes: 1 addition & 1 deletion src/core/dispatch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ export async function dispatchCommand(
await interactor.openDevice();
return { app: null };
}
await interactor.open(app, { activity: context?.activity });
await interactor.open(app, { activity: context?.activity, appBundleId: context?.appBundleId });
return { app };
}
case 'close': {
Expand Down
5 changes: 5 additions & 0 deletions src/core/open-target.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
export function isDeepLinkTarget(input: string): boolean {
const value = input.trim();
if (!value) return false;
return /^[A-Za-z][A-Za-z0-9+.-]*:\/\/.+/.test(value);
}
92 changes: 92 additions & 0 deletions src/daemon/handlers/__tests__/session.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -120,3 +120,95 @@ test('boot succeeds for supported device in session', async () => {
assert.equal(response.data?.booted, true);
}
});

test('open URL on existing iOS session clears stale app bundle id', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'ios-session';
sessionStore.set(
sessionName,
{
...makeSession(sessionName, {
platform: 'ios',
id: 'sim-1',
name: 'iPhone 15',
kind: 'simulator',
booted: true,
}),
appBundleId: 'com.example.old',
appName: 'Old App',
},
);

let dispatchedContext: Record<string, unknown> | undefined;
const response = await handleSessionCommands({
req: {
token: 't',
session: sessionName,
command: 'open',
positionals: ['https://example.com/path'],
flags: {},
},
sessionName,
logPath: path.join(os.tmpdir(), 'daemon.log'),
sessionStore,
invoke: noopInvoke,
dispatch: async (_device, _command, _positionals, _out, context) => {
dispatchedContext = context as Record<string, unknown> | undefined;
return {};
},
ensureReady: async () => {},
});

assert.ok(response);
assert.equal(response?.ok, true);
const updated = sessionStore.get(sessionName);
assert.equal(updated?.appBundleId, undefined);
assert.equal(updated?.appName, 'https://example.com/path');
assert.equal(dispatchedContext?.appBundleId, undefined);
});

test('open app on existing iOS session resolves and stores bundle id', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'ios-session';
sessionStore.set(
sessionName,
{
...makeSession(sessionName, {
platform: 'ios',
id: 'sim-1',
name: 'iPhone 15',
kind: 'simulator',
booted: true,
}),
appBundleId: 'com.example.old',
appName: 'Old App',
},
);

let dispatchedContext: Record<string, unknown> | undefined;
const response = await handleSessionCommands({
req: {
token: 't',
session: sessionName,
command: 'open',
positionals: ['settings'],
flags: {},
},
sessionName,
logPath: path.join(os.tmpdir(), 'daemon.log'),
sessionStore,
invoke: noopInvoke,
dispatch: async (_device, _command, _positionals, _out, context) => {
dispatchedContext = context as Record<string, unknown> | undefined;
return {};
},
ensureReady: async () => {},
});

assert.ok(response);
assert.equal(response?.ok, true);
const updated = sessionStore.get(sessionName);
assert.equal(updated?.appBundleId, 'com.apple.Preferences');
assert.equal(updated?.appName, 'settings');
assert.equal(dispatchedContext?.appBundleId, 'com.apple.Preferences');
});
7 changes: 4 additions & 3 deletions src/daemon/handlers/session.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import fs from 'node:fs';
import { dispatchCommand, resolveTargetDevice } from '../../core/dispatch.ts';
import { isCommandSupportedOnDevice } from '../../core/capabilities.ts';
import { isDeepLinkTarget } from '../../core/open-target.ts';
import { AppError, asAppError } from '../../utils/errors.ts';
import type { DeviceInfo } from '../../utils/device.ts';
import type { DaemonRequest, DaemonResponse, SessionAction, SessionState } from '../types.ts';
Expand Down Expand Up @@ -299,7 +300,7 @@ export async function handleSessionCommands(params: {
};
}
let appBundleId: string | undefined;
if (session.device.platform === 'ios') {
if (session.device.platform === 'ios' && !isDeepLinkTarget(appName)) {
try {
const { resolveIosApp } = await import('../../platforms/ios/index.ts');
appBundleId = await resolveIosApp(session.device, appName);
Expand Down Expand Up @@ -340,10 +341,10 @@ export async function handleSessionCommands(params: {
}
let appBundleId: string | undefined;
const appName = req.positionals?.[0];
if (device.platform === 'ios') {
if (device.platform === 'ios' && appName && !isDeepLinkTarget(appName)) {
try {
const { resolveIosApp } = await import('../../platforms/ios/index.ts');
appBundleId = await resolveIosApp(device, req.positionals?.[0] ?? '');
appBundleId = await resolveIosApp(device, appName);
} catch {
appBundleId = undefined;
}
Expand Down
23 changes: 22 additions & 1 deletion src/platforms/android/__tests__/index.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { parseAndroidLaunchComponent } from '../index.ts';
import { openAndroidApp, parseAndroidLaunchComponent } from '../index.ts';
import type { DeviceInfo } from '../../../utils/device.ts';
import { AppError } from '../../../utils/errors.ts';
import { findBounds, parseUiHierarchy } from '../ui-hierarchy.ts';

test('parseUiHierarchy reads double-quoted Android node attributes', () => {
Expand Down Expand Up @@ -89,3 +91,22 @@ test('parseAndroidLaunchComponent returns null when no component is present', ()
const stdout = 'No activity found';
assert.equal(parseAndroidLaunchComponent(stdout), null);
});

test('openAndroidApp rejects activity override for deep link URLs', async () => {
const device: DeviceInfo = {
platform: 'android',
id: 'emulator-5554',
name: 'Pixel',
kind: 'emulator',
booted: true,
};

await assert.rejects(
() => openAndroidApp(device, ' https://example.com/path ', '.MainActivity'),
(error: unknown) => {
assert.equal(error instanceof AppError, true);
assert.equal((error as AppError).code, 'INVALID_ARGS');
return true;
},
);
});
18 changes: 18 additions & 0 deletions src/platforms/android/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { withRetry } from '../../utils/retry.ts';
import { AppError } from '../../utils/errors.ts';
import type { DeviceInfo } from '../../utils/device.ts';
import type { RawSnapshotNode, SnapshotOptions } from '../../utils/snapshot.ts';
import { isDeepLinkTarget } from '../../core/open-target.ts';
import { waitForAndroidBoot } from './devices.ts';
import { findBounds, parseBounds, parseUiHierarchy, readNodeAttributes } from './ui-hierarchy.ts';

Expand Down Expand Up @@ -157,6 +158,23 @@ export async function openAndroidApp(
if (!device.booted) {
await waitForAndroidBoot(device.id);
}
const deepLinkTarget = app.trim();
if (isDeepLinkTarget(deepLinkTarget)) {
if (activity) {
throw new AppError('INVALID_ARGS', 'Activity override is not supported when opening a deep link URL');
}
await runCmd('adb', adbArgs(device, [
'shell',
'am',
'start',
'-W',
'-a',
'android.intent.action.VIEW',
'-d',
deepLinkTarget,
]));
return;
}
const resolved = await resolveAndroidApp(device, app);
if (resolved.type === 'intent') {
if (activity) {
Expand Down
24 changes: 24 additions & 0 deletions src/platforms/ios/__tests__/index.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { openIosApp } from '../index.ts';
import type { DeviceInfo } from '../../../utils/device.ts';
import { AppError } from '../../../utils/errors.ts';

test('openIosApp rejects deep links on iOS physical devices', async () => {
const device: DeviceInfo = {
platform: 'ios',
id: 'ios-device-1',
name: 'iPhone Device',
kind: 'device',
booted: true,
};

await assert.rejects(
() => openIosApp(device, 'https://example.com/path'),
(error: unknown) => {
assert.equal(error instanceof AppError, true);
assert.equal((error as AppError).code, 'UNSUPPORTED_OPERATION');
return true;
},
);
});
19 changes: 17 additions & 2 deletions src/platforms/ios/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import type { ExecResult } from '../../utils/exec.ts';
import { AppError } from '../../utils/errors.ts';
import type { DeviceInfo } from '../../utils/device.ts';
import { Deadline, isEnvTruthy, retryWithPolicy, TIMEOUT_PROFILES, type RetryTelemetryEvent } from '../../utils/retry.ts';
import { isDeepLinkTarget } from '../../core/open-target.ts';
import { bootFailureHint, classifyBootFailure } from '../boot-diagnostics.ts';

const ALIASES: Record<string, string> = {
Expand Down Expand Up @@ -35,8 +36,22 @@ export async function resolveIosApp(device: DeviceInfo, app: string): Promise<st
throw new AppError('APP_NOT_INSTALLED', `No app found matching "${app}"`);
}

export async function openIosApp(device: DeviceInfo, app: string): Promise<void> {
const bundleId = await resolveIosApp(device, app);
export async function openIosApp(
device: DeviceInfo,
app: string,
options?: { appBundleId?: string },
): Promise<void> {
const deepLinkTarget = app.trim();
if (isDeepLinkTarget(deepLinkTarget)) {
if (device.kind !== 'simulator') {
throw new AppError('UNSUPPORTED_OPERATION', 'Deep link open is only supported on iOS simulators in v1');
}
await ensureBootedSimulator(device);
await runCmd('open', ['-a', 'Simulator'], { allowFailure: true });
await runCmd('xcrun', ['simctl', 'openurl', device.id, deepLinkTarget]);
return;
}
const bundleId = options?.appBundleId ?? (await resolveIosApp(device, app));
if (device.kind === 'simulator') {
await ensureBootedSimulator(device);
await runCmd('open', ['-a', 'Simulator'], { allowFailure: true });
Expand Down
4 changes: 2 additions & 2 deletions src/utils/args.ts
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ CLI to control iOS and Android devices for AI agents.

Commands:
boot Ensure target device/simulator is booted and ready
open [app] Boot device/simulator; optionally launch app
open [app|url] Boot device/simulator; optionally launch app or deep link URL
close [app] Close app or just end session
reinstall <app> <path> Uninstall + install app from binary path
snapshot [-i] [-c] [-d <depth>] [-s <scope>] [--raw] [--backend ax|xctest]
Expand Down Expand Up @@ -227,7 +227,7 @@ Flags:
--device <name> Device name to target
--udid <udid> iOS device UDID
--serial <serial> Android device serial
--activity <component> Android activity to launch (package/Activity)
--activity <component> Android app launch activity (package/Activity); not for URL opens
--session <name> Named session
--verbose Stream daemon/runner logs
--json JSON output
Expand Down
4 changes: 2 additions & 2 deletions src/utils/interactors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ export type RunnerContext = {
};

export type Interactor = {
open(app: string, options?: { activity?: string }): Promise<void>;
open(app: string, options?: { activity?: string; appBundleId?: string }): Promise<void>;
openDevice(): Promise<void>;
close(app: string): Promise<void>;
tap(x: number, y: number): Promise<void>;
Expand Down Expand Up @@ -60,7 +60,7 @@ export function getInteractor(device: DeviceInfo, runnerContext: RunnerContext):
};
case 'ios':
return {
open: (app) => openIosApp(device, app),
open: (app, options) => openIosApp(device, app, { appBundleId: options?.appBundleId }),
openDevice: () => openIosDevice(device),
close: (app) => closeIosApp(device, app),
screenshot: (outPath) => screenshotIos(device, outPath),
Expand Down
Loading