Automated flows for repetitive agent-device setup sequences

[kacper-mikolajczak]

Background

Phase 1 of the agentic mobile testing PoC (#87030) ships the agent-device skill at .claude/skills/agent-device/ via #87662. The skill drives iOS/Android by feeding accessibility tree snapshots to the LLM, which reasons about every tap, scroll, and assertion. In practice, most sessions share a setup prefix - signing in, clearing onboarding, navigating to a specific surface - before the agent reaches the screen the task actually concerns.

Problem

When the agent drives a recurring setup prefix entirely from LLM reasoning, each run traverses the UI via a different path and timing, which prevents the session from producing a deterministic, quickly-reached starting state for the screen under test.

Solution

Introduce an automated-flow mechanism in the agent-device skill: each flow is a named, deterministic unit that drives the UI to a known surface without LLM reasoning. Concrete authoring format, storage layout, drift handling, and the initial flow set are decided in the implementation follow-up.

Issue Owner

Current Issue Owner: @kacper-mikolajczak

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~022046528898951391896
• Upwork Job ID: 2046528898951391896
• Last Price Increase: 2026-04-21

[kacper-mikolajczak]

CC @rlinoz @Julesssss @BartekObudzinski

[MelvinBot]

Proposal

What is the root cause of that problem?

The agent-device skill (PR #87662) delegates all UI navigation to the LLM — including repetitive setup steps like sign-in, onboarding dismissal, and navigating to a target screen. Because the LLM reasons from scratch each time, the path and timing vary between runs, making the starting state for the actual test non-deterministic and slow.

What changes do you think we should make in order to solve the problem?

Introduce an automated-flow mechanism inside the agent-device skill that provides named, deterministic navigation sequences the agent can invoke instead of reasoning through setup UI step by step.

Key design points:

1. Flow definition format — Each flow is a named YAML/JSON file (e.g., flows/sign-in.yml) containing an ordered list of device actions (tap, type, scroll, wait-for) with accessibility-id or text selectors. No LLM reasoning runs during flow execution.

2. Storage layout — Flows live in .claude/skills/agent-device/flows/ so they ship with the skill and are version-controlled alongside it.

3. Invocation — The skill's SKILL.md is updated to instruct the agent: "Before reasoning through setup steps, check if a matching automated flow exists and invoke it via agent-device run-flow <name>." The agent only falls back to LLM-driven navigation when no flow covers the needed path.

4. Drift handling — If a flow step fails (selector not found after timeout), the flow aborts with a clear error naming the failing step and selector. The agent can then fall back to LLM reasoning for that segment and flag the drift for a human to update the flow definition.

5. Initial flow set — Start with the two highest-value flows: sign-in (credentials → authenticated home screen) and navigate-to-chat (home → specific chat by name). These cover the most common setup prefix.

What alternative solutions did you explore? (Optional)

• Hardcoded deep-link URLs — faster but not all screens support deep links, and it skips the UI path that tests often need to validate.
• Snapshot/restore of simulator state — would give instant starting state but ties flows to specific simulator images, making them fragile across iOS/Android version bumps.
• Recorded Maestro/Detox scripts — heavier toolchain dependency; the YAML-based approach keeps flows lightweight and co-located with the skill.

---

Next Steps for Contributor+ team: Reply with @MelvinBot implement this to create a draft PR, @MelvinBot <your feedback> to refine this analysis, or explain why you are rejecting Melvin's proposal.

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @cretadn22 (External)

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~022046528898951391896

[KJ21-ENG]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Every agent-device session starts with the same setup prefix (sign-in, onboarding dismissal, navigation to a target surface) before it reaches the screen the task actually concerns. Because the LLM re-plans that prefix from scratch against the current a11y snapshot each run, the path, timing, and final state vary between runs, so there is no deterministic, quickly-reached starting state for the screen under test.

What is the root cause of that problem?

The agent-device skill shipped in PR #87662 is intentionally "glue code, not vendored copy": its SKILL.md points the LLM at the upstream CLI's skill files and delegates every tap/scroll decision to LLM reasoning over the live a11y snapshot. There is no layer between the planner and the device driver for named, replayable setup sequences. The LLM therefore re-derives trivial paths (tap "Continue", dismiss the onboarding modal, tap the workspace switcher) on every run, which compounds three problems: (1) snapshot-to-action variance produces different concrete paths; (2) timing varies because reasoning latency is non-uniform; (3) there is no stable post-condition to assert that setup finished successfully, so downstream test assertions sometimes run against a transitional screen.

What changes do you think we should make in order to solve the problem?

Add a flows layer to the existing skill, using the smallest primitives that fit the skill's "glue code" architecture so we do not introduce a new DSL, a new runtime, or a change to allowed-tools.

1. Storage. Add .claude/skills/agent-device/flows/<name>.md. Each file is a plain procedure with: a one-line purpose, a Preconditions block (app state assumed on entry), a numbered list of steps where each step is a concrete agent-device CLI invocation plus a Post-condition: line describing the a11y element or text that must be visible before continuing, and a final Exit state block describing the screen the flow leaves the device on.

2. Selectors. Every step uses accessibility-id selectors, never visible text, so flows survive locale switches and copy changes. Where a target element lacks a stable a11y id, the fix is to add one in product code as part of the flow's initial PR — flows should not normalize over brittle selectors.

3. Invocation. Extend SKILL.md with a short section instructing the agent: before reasoning through setup, list flows/ and if a matching named flow exists, execute its steps verbatim via agent-device CLI and assert each post-condition; only fall back to LLM reasoning when no flow matches or a post-condition fails. No new subcommand, no parser — the existing Bash(agent-device *) allow-list already covers this.

4. Drift handling. When a post-condition fails, the skill (a) writes a drift report to flows/_drift/<name>-<iso-timestamp>.md containing the failing step, the expected selector, and the current a11y snapshot excerpt; (b) falls back to LLM-driven navigation for the remainder of the flow; (c) surfaces the drift report path in the session summary so a human can reconcile the flow in a follow-up PR. Add flows/_drift/ to .gitignore.

5. Initial flow set. Ship three flows, chosen to cover the setup prefix the issue calls out verbatim ("signing in, clearing onboarding, navigating to a specific surface"): sign-in.md (credentials → authenticated LHN), dismiss-onboarding.md (fresh-account onboarding modal → LHN), open-workspace-settings.md (LHN → workspace settings root). Two is not enough — the onboarding dismissal is a distinct, common source of flakiness and deserves its own named flow.

6. Discoverability. Add a flows/README.md that enumerates each flow's name, purpose, preconditions, and exit state. SKILL.md references this file so the agent learns the available flow set without scanning filesystem contents during reasoning.

Scope boundaries. Files touched: .claude/skills/agent-device/SKILL.md (extend with the flows section), new files under .claude/skills/agent-device/flows/, and .gitignore (add flows/_drift/). Platforms: iOS and Android, via the existing agent-device CLI. Explicitly not touched: the upstream agent-device npm package (no new subcommand, no API change), the allowed-tools declaration in SKILL.md (existing Bash(agent-device *) already covers flow execution), runtime app code (except for adding missing a11y ids as flows require them, which are trivial, localized changes), and other skills (coding-standards, onyx, playwright-app-testing, sentry). No backend or API contract changes. This work depends on PR #87662 landing first.

What alternative solutions did you explore? (Optional)

• YAML DSL with a new agent-device run-flow <name> subcommand — this is the shape of the existing MelvinBot proposal on the issue. It is workable but pushes two things upstream (a parser and a subcommand) that the skill was deliberately designed to avoid when PR [NoQA] Add agent-device glue-code skill for mobile testing #87662 moved off the earlier "vendored copy" approach. It also forces flows to live in the npm package's release cadence rather than in-repo with the product code whose a11y ids they depend on. Melvin's proposal is correct that flows should be named, file-based, and selector-driven with drift fallback; it is wrong to route invocation through a new CLI subcommand and a new YAML schema when a markdown procedure executed by the existing CLI covers the same ground with zero new surface area. Its two-flow initial set (sign-in, navigate-to-chat) also omits the onboarding dismissal step that the issue explicitly names.
• Deep-link URLs for setup — fastest to reach a surface, but bypasses the UI path that many tests need to validate, and not every target screen has a stable deep link.
• Simulator snapshot/restore — instant starting state, but ties flows to specific simulator images, making them fragile across iOS/Android version bumps and unusable on physical devices.
• Record-and-replay (LLM drives once, concrete steps extracted) — attractive but moves the determinism problem into the recording step; a hand-authored flow with explicit post-conditions is both smaller and easier to review.

[JasimMalook]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Agent-device sessions repeatedly execute the same setup steps (sign-in, onboarding dismissal, navigation), but rely entirely on LLM reasoning, resulting in inconsistent paths, variable timing, and non-deterministic starting states for tests.

---

What is the root cause of that problem?

The system lacks a persistence layer for previously successful navigation paths.

Each session:

• Recomputes setup steps from scratch using LLM reasoning
• Does not reuse known-good execution paths
• Has no memory of stable UI transitions

So the issue is not only missing deterministic flows, but absence of execution reuse and learning across sessions.

---

What changes do you think we should make in order to solve the problem?

Instead of only static predefined flows, introduce a hybrid execution model with caching:

1. Add a "flow cache" layer:

• After a successful setup sequence, persist the executed steps as a reusable flow:
  .cache/agent-device/flows/.json

• The hash is derived from:

  • app version
  • platform (iOS/Android)
  • entry screen

---

2. On new session:

• Before invoking LLM reasoning:
  • Check if a cached flow exists
  • Replay it deterministically

---

3. Validation during replay:

• After each step, verify expected UI state (a11y snapshot match)
• If mismatch occurs:
  • Abort replay
  • Fallback to LLM reasoning
  • Update cache with corrected path

---

4. Lightweight learning loop:

• When LLM completes a new setup path successfully:
  • Save it as a new cached flow
• Over time, system converges to stable deterministic paths without manual authoring

---

5. Safety constraints:

• Do not reuse flows across different accounts or auth states
• Invalidate cache on major UI changes (e.g., version mismatch)

---

What alternative solutions did you explore? (Optional)

• Static predefined flows (YAML/Markdown)
  → Rejected because they require manual maintenance and drift quickly

• Deep links or simulator snapshots
  → Rejected due to limited coverage and platform constraints

• Pure LLM reasoning (current approach)
  → Rejected due to non-determinism and inefficiency

[nabi-ebrahimi]

Proposal

Please re-state the problem that we are trying to solve in this issue.

agent-device sessions repeatedly spend time signing in, clearing onboarding, and navigating to the same starting screen before the actual task begins. Because that setup is currently recreated through normal deep-link handling and UI navigation every run, the agent reaches the target surface with inconsistent timing and state instead of starting from one deterministic baseline.

What is the root cause of that problem?

The root cause is that app startup only supports the normal deep-link path, and that path always hands control back to dynamic runtime navigation.

The first place this shows up is in src/DeepLinkHandler.tsx, where every initial URL is always treated as a normal deep link:

App/src/DeepLinkHandler.tsx

Lines 42 to 58 in 9cb33cd

	// If the app is opened from a deep link, get the reportID (if exists) from the deep link and navigate to the chat report
	Linking.getInitialURL().then((url) => {
	onInitialUrl(url as Route);
	if (url) {
	if (conciergeReportID === undefined) {
	Log.info('[Deep link] conciergeReportID is undefined when processing initial URL', false, {url});
	}
	if (introSelected === undefined) {
	Log.info('[Deep link] introSelected is undefined when processing initial URL', false, {url});
	}
	openReportFromDeepLink(url, allReports, isAuthenticated, conciergeReportID, introSelected, isSelfTourViewed, betas);
	} else {
	Report.doneCheckingPublicRoom();
	}
	endSpan(CONST.TELEMETRY.SPAN_BOOTSPLASH.DEEP_LINK);

That means every incoming URL is treated as a normal deep link and pushed through the regular navigation flow. There is no small deterministic branch for "just put the app into a known authenticated test state".

The same assumption exists in src/libs/Navigation/NavigationRoot.tsx, where initialUrl is always parsed as a URL:

App/src/libs/Navigation/NavigationRoot.tsx

Lines 106 to 108 in 9cb33cd

	const initialState = useMemo(() => {
	const path = initialUrl ? getPathFromURL(initialUrl) : null;
	if (path?.includes(ROUTES.MIGRATED_USER_WELCOME_MODAL.route) && shouldOpenLastVisitedPath(lastVisitedPath) && isOnboardingCompleted && authenticated) {

So initialUrl is assumed to always be a URL-shaped input, not a known internal route that the app can trust and open directly.

This matters because the onboarding gates are driven from Onyx selectors in src/selectors/Onboarding.ts:

App/src/selectors/Onboarding.ts

Lines 12 to 22 in 9cb33cd

	function hasCompletedGuidedSetupFlowSelector(onboarding: OnyxValue<typeof ONYXKEYS.NVP_ONBOARDING>): boolean | undefined {
	// Onboarding is an empty object for old accounts and accounts migrated from OldDot
	if (isEmptyObject(onboarding)) {
	return true;
	}
	if (!isEmptyObject(onboarding) && onboarding?.hasCompletedGuidedSetupFlow === undefined) {
	return true;
	}
	return onboarding?.hasCompletedGuidedSetupFlow;

App/src/selectors/Onboarding.ts

Lines 51 to 56 in 9cb33cd

	function hasSeenTourSelector(onboarding: OnyxValue<typeof ONYXKEYS.NVP_ONBOARDING>): boolean | undefined {
	if (isEmptyObject(onboarding)) {
	return false;
	}
	return !!onboarding?.selfTourViewed;

hasCompletedGuidedSetupFlowSelector() and hasSeenTourSelector() are what determine whether onboarding-related setup still needs to happen. Since there is no startup bootstrap path that can locally establish those preconditions, the agent has to recreate them indirectly by walking the UI.

In simple terms: the app has no deterministic startup primitive for test setup. It only knows how to "figure it out live", which is exactly what makes agent-device setup slow and non-deterministic.

What changes do you think we should make in order to solve the problem?

We should add a small dev-only bootstrap path for agent-device instead of introducing a new flow framework, new DSL, or cached replay system.

We should change the initial URL handling in src/DeepLinkHandler.tsx so it can intercept a dedicated bootstrap deep-link before the normal deep-link flow runs:

App/src/DeepLinkHandler.tsx

Lines 42 to 58 in 9cb33cd

	// If the app is opened from a deep link, get the reportID (if exists) from the deep link and navigate to the chat report
	Linking.getInitialURL().then((url) => {
	onInitialUrl(url as Route);
	if (url) {
	if (conciergeReportID === undefined) {
	Log.info('[Deep link] conciergeReportID is undefined when processing initial URL', false, {url});
	}
	if (introSelected === undefined) {
	Log.info('[Deep link] introSelected is undefined when processing initial URL', false, {url});
	}
	openReportFromDeepLink(url, allReports, isAuthenticated, conciergeReportID, introSelected, isSelfTourViewed, betas);
	} else {
	Report.doneCheckingPublicRoom();
	}
	endSpan(CONST.TELEMETRY.SPAN_BOOTSPLASH.DEEP_LINK);

After:

const bootstrapRoute = applyAgentDeviceBootstrap(url, hasAuthToken());
if (bootstrapRoute) {
    onInitialUrl(bootstrapRoute);
    Report.doneCheckingPublicRoom();
    endSpan(CONST.TELEMETRY.SPAN_BOOTSPLASH.DEEP_LINK);
    return;
}

onInitialUrl(url as Route);

if (url) {
    openReportFromDeepLink(url, allReports, isAuthenticated, conciergeReportID, introSelected, isSelfTourViewed, betas);
} else {
    Report.doneCheckingPublicRoom();
}

This change adds one early deterministic branch for agent-device. If the URL is a known bootstrap URL, we set the minimal local onboarding state and return a stable internal route. Otherwise, we keep the existing deep-link behavior unchanged.

We should add a small helper, for example src/libs/applyAgentDeviceBootstrap.ts, that only converts a bootstrap URL into a safe internal route and locally marks onboarding as already satisfied:

export default function applyAgentDeviceBootstrap(url: string, isAuthenticated: boolean): Route | null {
    if (!__DEV__ || !isAuthenticated) {
        return null;
    }

    const bootstrapRoute = getBootstrapRoute(url);
    if (!bootstrapRoute) {
        return null;
    }

    Onyx.merge(ONYXKEYS.NVP_ONBOARDING, {
        hasCompletedGuidedSetupFlow: true,
        selfTourViewed: true,
    });

    return bootstrapRoute;
}

This is minimal and fixes the actual issue because it gives agent-device a deterministic startup escape hatch without changing the normal product flow.

We should also change src/libs/Navigation/NavigationRoot.tsx so the bootstrap path can pass an internal route directly instead of forcing it through URL parsing:

App/src/libs/Navigation/NavigationRoot.tsx

Lines 106 to 108 in 9cb33cd

	const initialState = useMemo(() => {
	const path = initialUrl ? getPathFromURL(initialUrl) : null;
	if (path?.includes(ROUTES.MIGRATED_USER_WELCOME_MODAL.route) && shouldOpenLastVisitedPath(lastVisitedPath) && isOnboardingCompleted && authenticated) {

After:

const path = initialUrl ? (initialUrl.includes('://') ? getPathFromURL(initialUrl) : initialUrl) : null;

This lets the bootstrap path pass an internal route such as home, inbox, or workspaces/:policyID/overview directly into navigation instead of forcing everything through URL parsing.

In plain English, the change is:

1. Detect a special dev-only bootstrap deep link early.
2. Set the minimal local onboarding state needed to avoid repeated setup.
3. Open a known internal route directly.
4. Fall back to the existing deep-link behavior for everything else.

That is smaller, safer, and easier to maintain than adding reusable flow files or a replay/cache layer.

What specific scenarios should we cover in automated tests to prevent reintroducing this issue in the future?

None

What alternative solutions did you explore? (Optional)

I explored the same general direction as the other proposals: named flow files, step replay, and cached execution paths. I rejected those because they add more surface area than the issue needs: new flow definitions, drift handling, replay semantics, and ongoing maintenance.

I also considered solving this entirely inside the agent-device skill without touching app code. I rejected that because the issue is specifically that the app does not expose a deterministic startup primitive. Keeping all of the logic outside the app would still leave the agent dependent on live UI setup.

I also considered using only regular deep links. I rejected that because the current startup code still routes those through the normal dynamic deep-link path, so it does not actually remove the nondeterministic setup problem by itself.

The smallest better fix is to add a dedicated dev-only bootstrap entry point and let the app start in a known authenticated state on purpose.

[kacper-mikolajczak]

@MelvinBot this is a sub-ticket for #87030 created solely for work organisation purposes. We are already handling the implementation. There's no need to set bounty and expect proposals.