[device-testing] On-device testing pipeline — Android + iOS fleet (kickoff)

[lalalune]

This is the coordination thread for the on-device testing workstream of the LifeOps
Personal Assistant MVP (GitHub project 15). Every dev machine has Android and iOS
devices plugged in; those devices are the mobile e2e environment — CI has no iOS e2e
lane at all, and the Android CI lane is dispatch-only with a documented x86_64 SIGSEGV
limitation on the local-model route. This workstream makes the fleet trustworthy: one
command per platform that runs current develop on the real device, a fleet-wide
freshness check, and failure output an agent can act on without archaeology.

Doctrine and decisions adopted

• Wire existing scripts, build nothing new. The orchestrators
  (packages/app/scripts/android-e2e.mjs, ios-e2e.mjs), the physical-iPhone deploy
  pipeline (ios-device-deploy.mjs with --skip-appexes, merged PR feat(ios): --skip-appexes device-deploy mode for main-app-only testing #13174), the
  XCUITest BootCapture harness, and the renderer build stamp
  (dist/eliza-renderer-build.json with buildId + git commit) all exist. The gaps are
  defaults, one small status script, and bundle plumbing.
• Freshness is stamp-driven, never assumed. The Capacitor footgun (web bundle bakes
  into the APK/IPA at build time) is live in the Android runner today: it installs only
  when the package is missing entirely. Fix the runner; compare installed vs fresh
  buildId; reinstall on mismatch; never fall back to "assume fresh".
• Evidence is posted INLINE in issues/PRs — MP4 videos (they render inline) and JPG
  screenshots (size), NOT committed to .github/issue-evidence/. Runners end with one
  gitignored bundle dir whose inline/ contents can be posted as-is. iOS simctl
  recordings get remuxed .mov → .mp4.
• Fail fast, no soft-skips on fleet machines. Devices are guaranteed plugged in; a
  runner that green-skips "no device" is lying. The --require-evidence contract
  (Screenshot/recording capture goes green with no/stale/wrong-window evidence (audit:cloud, capture:*, Electrobun) #13624) is the model — arm it by default in the per-platform runners.
• --skip-appexes is the unattended iOS posture. Per-appex provisioning profiles
  need an Xcode session or ASC API key (owner decision, parked). The MVP surfaces —
  chat, onboarding, views, LifeOps — do not live in appexes; the runner logs the
  exclusion loudly.
• Out of scope: greening the CI x86_64 emulator, self-hosted arm64 runners, appex
  profile automation, any device-farm service, desktop capture lanes, new mobile test
  content (that's the scenarios workstream).

Research doc

packages/docs/ongoing-development/research/08-device-testing-pipeline.md — read it
before claiming anything; every claim in it carries a file:line reference, including the
verified stale-install bug and the exact stamp plumbing to reuse.

Issues (most important first)

1. [device-testing] android-e2e runs stale builds: install-if-missing skips fresh APKs — make freshness stamp-driven (P0)
2. [device-testing] devices:status — report installed buildId/commit per connected device vs develop HEAD (P0)
3. [device-testing] Triage-ready artifact bundle per device-e2e run: MP4 + JPG + logs + summary.json + junit.xml (P1)
4. [device-testing] One-command physical-iPhone e2e lane (deploy → smoke → capture → bundle) with deploy ledger (P1)
5. [device-testing] Point-of-failure forensics: auto-capture screenshot + device logs when a runner step fails (P1)
6. [device-testing] Device lease lock: stop concurrent agent sessions colliding on one simulator/device (P2)

Ordering notes: 1 and 2 are independent and can start immediately (2 depends on 1's
Android stamp reader — coordinate the shared helper in
packages/app/scripts/lib/android-device.mjs). 3 blocks the bundle-assembly halves of 4
and 5. 6 is independent.

How to claim work

1. Comment on the issue: "claiming — session/branch <branch>" so concurrent agents
   don't double-implement.
2. Move the card on project 15 to In Progress.
3. Branch as fix|feat/<issue#>-<slug> off develop; PR back to develop; sync
   (git fetch origin && git rebase origin/develop) before opening/updating.
4. Before implementing, check the issue isn't already closed and git log origin/develop on the target files — this repo's swarm merges within hours.
5. Physical devices are shared: check devices:status (once landed) or say which
   device/simulator you're occupying in the issue thread. Don't hold a device idle.

Evidence rules (binding for every PR here)

Per PR_EVIDENCE.md, with the updated mechanics: evidence goes inline in the PR
description/comments, not into .github/issue-evidence/.

• MP4 walkthrough of the real device run (screenrecord / BootCapture stitch) — must
  render inline in the PR.
• JPG before/after screenshots; for bug fixes, include the WRONG behavior (e.g. the
  stale-buildId run) as well as the fixed one.
• Logs: runner transcript + device log tail (logcat / os_log / boot-trace) as text
  blocks or attached .log.
• Real runs only — a real attached device or booted simulator, real installed build,
  stamp readback proving which build ran. No mock-in-place-of-device. If a row doesn't
  apply, write N/A — <reason> visibly.
• Confirm the running build is yours (buildId/commit in summary.json or
  devices:status output) — a screenshot of a stale install proves nothing.

Filed issues

• [device-testing] android-e2e runs stale builds: install-if-missing skips fresh APKs — make freshness stamp-driven #14334 — P0 [device-testing] android-e2e runs stale builds: install-if-missing skips fresh APKs — make freshness stamp-driven
• [device-testing] devices:status — report installed buildId/commit per connected device vs develop HEAD #14335 — P0 [device-testing] devices:status — report installed buildId/commit per connected device vs develop HEAD
• [device-testing] Triage-ready artifact bundle per device-e2e run: MP4 + JPG + logs + summary.json + junit.xml #14336 — P1 [device-testing] Triage-ready artifact bundle per device-e2e run: MP4 + JPG + logs + summary.json + junit.xml
• [device-testing] One-command physical-iPhone e2e lane (deploy → smoke → capture → bundle) with deploy ledger #14337 — P1 [device-testing] One-command physical-iPhone e2e lane (deploy → smoke → capture → bundle) with deploy ledger
• [device-testing] Point-of-failure forensics: auto-capture screenshot + device logs when a runner step fails #14338 — P1 [device-testing] Point-of-failure forensics: auto-capture screenshot + device logs when a runner step fails
• [device-testing] Device lease lock: stop concurrent agent sessions colliding on one simulator/device #14339 — P2 [device-testing] Device lease lock: stop concurrent agent sessions colliding on one simulator/device

---

Design doc · MVP board