fix(workflow): unwedge shared E2E infra (Android claims, preflight, tmux targeting, stack cap) - #4826
Merged
Merged
Conversation
Three defects made Android E2E rounds fail for reasons agents could not diagnose or recover from: - Android device claims treated an ADB serial as a device identity, but serials are recycled emulator ports. A claim was only reclaimable once its owning worktree was deleted, and worktrees outlive runs by days, so a claim left behind by a dead emulator wedged that serial permanently. emulator-5554 and emulator-5556 were both wedged this way on the shared machine — the first two serials any emulator boot takes. Claims now record the guest kernel boot id, so a foreign claim holds a serial only while it names the instance currently answering on it. - The shared E2E preflight accepted only `iOS Bundled` as Metro readiness evidence. On Android the alternative `Starting Metro Bundler` line scrolls out of the captured window on a long-lived stack, so preflight started failing with "Metro has not reached a usable state" the longer a session ran — exactly when mid-test recovery reruns login.sh. It also probed the Metro manifest as iOS regardless of the target platform. - dispatch-role.sh resolved its target with an untargeted `tmux display-message -p '#S'`, which answers with the server's current session rather than the caller's. A dispatcher outside tmux silently dropped role windows into an unrelated session; freshly created kilo-e2e-android-* emulator sessions were the usual victim, and device cleanup then killed those foreign agents. The session is now resolved through $TMUX_PANE, falling back to an own session instead of a guess. Also documents the one-time `expo prebuild --platform android` that `build` needs in a fresh worktree, since apps/mobile/android/ is git-ignored and `build` cannot generate it itself.
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryReviewed the one new commit ( Files Reviewed (3 files changed since last review)
Previous Review Summaries (2 snapshots, latest commit ac62472)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit ac62472)Status: No Issues Found | Recommendation: Merge Executive SummaryReviewed the two new commits (untargeted-tmux-session doc/script follow-ups and a dispatch-role.sh pane status echo) added since the prior review pass; both are internally consistent with the previously-reviewed claim/preflight/dispatch fixes and introduce no new correctness, security, or reliability issues. Files Reviewed (4 files changed since last review)
Previous review (commit 7e18de1)Status: No Issues Found | Recommendation: Merge Executive SummaryReviewed the Android E2E claim/preflight/dispatch fixes across all six changed files; the boot-id claim reclaim logic, platform-aware Metro readiness check, and Files Reviewed (6 files)
Reviewed by claude-sonnet-5 · Input: 26 · Output: 12.3K · Cached: 737K Review guidance: REVIEW.md from base branch |
dispatch-role.sh was only one of three places using `tmux display-message -p '#S'` without a target, which answers with the tmux server's current session rather than the caller's: - WORKFLOW.md told the starter to launch planners into that session, so planner windows can be filed under an unrelated section the same way role windows were. - The e2e-verifier definition used it to name the owner of its device slot. That is the worst case: the slot ends up owned by another session's name and is reclaimed when that session dies while the verifier is still driving a device. The machine then over-subscribes past its three slots, and concurrent emulator boots and native builds start timing out — read as flaky emulators, not as a slot leak. Both now resolve through `$TMUX_PANE`, and say what to do when it is unset (own session for the planner; test-environment blocker for the verifier) instead of guessing a target. e2e-slot.sh now rejects an owner that is not a live tmux session, so a window name or a misread session name fails loudly at acquire time instead of booking a slot that is reapable the moment it is written.
Dispatched role agents redirect stdout and stderr into the scratch log, so their tmux pane is empty for the whole run and an attached human reads a live agent as a dead one. Print the log path and the tail command in the pane before exec'ing kilo. Terminal only — nothing reaches the log, so the trailing EXITCODE marker and the sentinel contract are unchanged.
A slot capped concurrent device phases; nothing capped live dev stacks. The slot was released the moment a device phase ended while the stack stayed up, and `dev:stop` only happened when a section reached a terminal state — so any section that looped between rounds, wedged, or died left ~17 services running with nothing accounting for them. Five stacks were up on this 14-core host against a cap of three, load average 379, which is why emulator boots blew through the 8-minute envelope and read as flaky devices. e2e-slot.sh now owns both halves: - acquire records the worktree that took the slot. - release frees the slot and stops that worktree's stack, unless another held slot still covers it. A later round re-acquires and starts fresh; that restart is the price of the cap holding. - Reclaiming a dead holder's slot stops its stack too, so a section that dies mid-flight no longer leaks one forever. - status reports each holder's worktree and stack state, and warns about stacks running with no slot. - stacks [--reap] lists those and stops the workflow-owned ones. Stacks with no section run id in the name were started by hand and are only reported, never stopped. Coverage falls back to the holder's session-name prefix when a slot has no recorded worktree, so slots written before this change never make a live stack look abandoned.
iscekic
changed the title
iscekic
enabled auto-merge (squash)
July 28, 2026 14:18
jeanduplessis
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents running the kilo workflow on the mobile app kept losing rounds to Android emulator problems that looked like flaky hardware. They were three deterministic defects. Each one presents as "the emulator is misbehaving" and none of them was recoverable through a documented command, so the agent retried, rebooted, restarted the stack, and burned tokens.
1. Claims wedge a serial forever (the expensive one)
dev:mobile:android claimkeyed staleness on "has the claiming worktree been deleted". But ADB serials are recycled ports, not device identities: the next emulator to boot takesemulator-5554again, and workflow worktrees outlive their runs by days. So a claim left behind by a crashed run pinned that serial to a worktree that still exists — permanently.Both
emulator-5554andemulator-5556were wedged this way on the shared machine when I looked, from July 26 and 27, for emulators long gone. Those are the first two serials any emulator boot takes, so the next Android round was guaranteed to hit:with no way out —
releasefrom another worktree throws the same error, and there is no--force.Claims now record the guest kernel's boot id (
/proc/sys/kernel/random/boot_id), which is regenerated on every boot and stable for the instance's life. A foreign claim holds a serial only while it names the instance currently answering on it; claims from earlier instances, and pre-existing claims with no boot id, are reclaimed automatically. Same-worktree reclaim stays idempotent and refreshes the boot id when the emulator was restarted mid-run. The deleted-worktree rule is unchanged.2. Preflight's Metro readiness check is iOS-only
preflight.shacceptediOS Bundled|Starting Metro Bundleras evidence Metro was usable. On Android the bundle lines readAndroid Bundled, so the only thing that could match wasStarting Metro Bundler— which scrolls out of the 300-line capture window on any long-lived stack. Result: Android preflight starts failing withMetro has not reached a usable statethe longer a session runs, which is precisely when mid-test recovery (pm clear→ rerunlogin.sh) needs it. iOS never sees this. The manifest probe also asked for the iOS manifest regardless of target platform.Both now follow the platform.
3. The untargeted tmux session lookup (three callers)
tmux display-message -p '#S'without a target resolves the server's current session — the most recently active one — not the caller's. A dispatcher or agent that is not itself inside tmux therefore gets an arbitrary session name, and a freshly createdkilo-e2e-android-*emulator session is the natural attractor. Three places used it:dispatch-role.shfiled role windows into it. Observed live while investigating:efficient-custom-pool-e6e3-plan-reviewer-r2dand-r2ewere both running as windows insidekilo-e2e-android-hermes-mem-c716. The Android runbook's cleanup step istmux kill-session -t "$ANDROID_SESSION"— so that worktree's device cleanup would have killed two of another section's review agents mid-round, voiding them.WORKFLOW.mdtold the starter to launch planners into it, so planner windows land under an unrelated section the same way..kilo/agent/e2e-verifier.mdused it to name the owner of its device slot, and that is the expensive one. The slot ends up owned by another session's name, soe2e-slot.shreclaims it when that session dies while the verifier is still driving a device. The machine then runs more than its three concurrent device phases, and every emulator boot and native build competes — which surfaces as boot-envelope timeouts and GPU-policy relaunches, i.e. as flaky emulators rather than as a slot leak.All three now resolve through
$TMUX_PANE, which is unambiguous, and each says what to do when it is unset — own session for a dispatched role and for a planner, test-environment blocker for a verifier — instead of guessing.e2e-slot.sh acquireadditionally rejects an owner that is not a live tmux session, so a window name (which the docs already warned against) or a misread name fails loudly instead of booking a slot that is reapable the moment it is written.4. A dev stack was not capped by anything
The slot semaphore caps concurrent device phases at three. Nothing capped live dev stacks. A slot is released the moment a device phase ends (
AGENTS.md: never hold one through planning or review), whilepnpm dev:stoponly runs when a section reaches COMPLETE or BLOCKED — so a section looping between rounds, wedged, or dead leaves ~17 services up with nothing accounting for them. Found five stacks running against a cap of three, load average 379 on 14 cores (379/508/327 across 1/5/15 min). An emulator that cold-boots in 24s on an idle host cannot make the runbook's 8-minute envelope under that, and the documented response is to blame GPU mode and relaunch — so this presents as flaky emulators too.A slot and a stack are now one resource in
e2e-slot.sh:acquirerecords the worktree,releasestops that worktree's stack unless another held slot still covers it, and reclaiming a dead holder's slot stops its stack as well.statusreports each holder's worktree and stack state;stacks [--reap]lists stacks running with no slot and stops the workflow-owned ones. A stack whose name carries no section run id was started by hand and is only ever reported. Coverage falls back to the holder's session-name prefix so slots written before this change never make a live stack look abandoned.The cost is explicit: a section that releases its slot between rounds pays a stack restart on the next one. That is what makes the cap of three actually hold.
Also
Documents the one-time
npx expo prebuild --platform androidthatbuildneeds in a fresh worktree.apps/mobile/android/is git-ignored andbuilddeliberately does not generate it (matching iOS), but the runbook never said so, so agents met it as an error that contradicts "never invoke Gradle directly".Verification
dev/local/mobile-android.test.ts— three new cases: recycled serial with a live owning worktree, legacy claim without a boot id, and own-claim refresh across a reboot. Confirmed all three fail against the old staleness rule and pass with the fix; full suite 10/10,mobile-android-build20/20 together.pnpm typecheckexit 0,pnpm lint0 warnings 0 errors,pnpm run format:changed,bash -non both shell scripts.emulator-5558(stable across reads).tmux move-windowrelocatedplan-reviewer-r2dinto its planner session with the same PIDs still running, so no round was voided by the diagnosis.stacks --reaprefused both a section-shaped stack with no worktree on disk and a hand-started stack with no run id, and left all three real stacks untouched; slug-to-worktree resolution checked againstgit worktree list.remote-cli-69f6,attach-oom-11fd), taking the host from five stacks to three and the 1-minute load from 379 to 142.emulator-5558alone — still connected and in use.Not verified end to end: a full Android E2E round through this preflight, which would have meant taking the emulator away from the agent currently using it.
One rollout note: a claim written before this change carries no boot id and is reclaimable by a worktree running the fixed code, so an Android device could change hands while a pre-fix round is still live. It self-resolves as running rounds finish, and the alternative was leaving the permanent wedge in place.