fix(ios): corroborate recorded tap outcomes - #1605
Conversation
Size Report
Startup median (7 runs, lower is better):
Top changed chunks:
|
|
P1: A corroborated iOS tap can return success while dropping the target identity required by recording/replay. Exact-head iOS Smoke is also red waiting for Automation lab; classify/rerun it before readiness. The original hostile-screen false-negative path still lacks live forced-failure evidence. |
|
[P1] Avoid scheduling the no-change retry from the corroborated post-action snapshot. The corroboration capture writes the changed after-state into Please suppress pending outcome retries for corroborated success and add a runtime coordinate regression with |
57b794f to
31fed27
Compare
|
Not ready:
Also pin every documented fail-closed boundary—sparse post-capture, capture failure, presentation mismatch, and backend mismatch—and include red-before evidence in the PR validation. |
|
Reviewed against the #1599 field evidence (bsky-24: 3 cells, 2,600s burned on taps that landed but reported failure). The architecture is right — corroborate at the daemon boundary with exactly one capture, unknown stays failure — and the test suite is exactly the right paranoia, especially the retry-suppression case and the save→replay evidence-chain test. The relaunch state-clearing also closes a real gap #1587 left: the XCTest-channel penalty is bundle+TTL keyed (not PID-bound like the depth memory), so it genuinely leaked across external relaunches. Three findings: P1 — claim strength vs evidence strength: any whole-tree digest delta against a possibly stale baseline converts failure into success. The baseline is P2 — keyless baselines default to "presentation matches" and then to a mismatched capture. P3 — "Closes #1599" slightly overclaims, per the same standard applied to #1603: the issue's second ask (why the restart path doesn't restore reliability — the tap-failure storm on the #1105 profile family) isn't root-caused here. The hint copy is now honest and corroboration removes the retry-spiral harm, which is the dominant cost — but the storm source remains. Suggest Minor: corroboration adds one capture (~0.5s post-#1587) to every failed tap on hostile screens — worth a line in the ADR so nobody later "optimizes" it away without knowing the trade. Also note #1604 touches the same 🤖 Addressed by Claude Code |
|
Addressed the stale/keyless corroboration findings at exact head Red-before, bypassing these guards made a stale baseline return success and made the keyless path perform a mismatched capture. Restored, the focused corroboration suite passes 11/11 and The remaining evidence limitation is also explicit: tests exercise the production daemon/provider route but mock the runner failure dispatch; exact-head live iOS CI does not deliberately force |
|
Re-reviewed exact head |
|
Live-evidence attempt update: exact-head runner build succeeded and the dedicated All exact-head GitHub checks, including iOS Smoke, are green. The remaining blocker is specifically the independent review requirement for a live forced/reproduced production |
|
Exact-head live follow-up after releasing the stale lease:
The implementation findings are resolved, but per |
|
Re-reviewed exact head Not ready: deliberately inject or reproduce that runner failure on the exact head, then show one landed press returns the corroboration warning without redispatch, the runner remains usable, and unchanged/sparse captures remain failures. |
|
Reframing the missing reproduction (relayed from the benchmark-arc analysis; supersedes the previous blocker per review discussion): Why two live bsky-24 attempts never emitted Agreed merge gate:
🤖 Addressed by Claude Code |
…oboration (#1605 merge gate) The field failure cannot be reproduced on this head: the tap false-failures were a downstream symptom of XCTest-channel saturation, which the #1587 capture fixes removed. The seam records a real XCTIssue AFTER the real gesture inside the per-command failure-count window, so xctestRecordedFailureResponse and target invalidation fire byte-for-byte like the field failure. Armed via a decrementing /tmp flag file (the daemon regenerates tampered xctestrun templates, so env plumbing cannot reach a daemon-spawned runner); compiled only under AGENT_DEVICE_RUNNER_UNIT_TESTS. Live evidence on a daemon-spawned runner (Bluesky, ad-bsky-repro sim): - landed case: injected failure on a real Search-tab tap -> success with the corroboration warning, screen verifiably on Search, no redispatch, runner serving next commands; flag consumed exactly once. - unchanged case: injected failure on a dead-coordinate tap -> capture unchanged -> XCTEST_RECORDED_FAILURE preserved with the new honest hint; runner still usable. - field-shape race (relaunch -> full snapshot -> immediate press, 5 attempts): no natural recorded failure occurs on this head — the hostile tree needed for channel saturation is gone, corroborating the causal story.
|
Merge gate delivered in 5454517 — all four agreed points: 1. Compile-time injection seam ( 2. Deterministic live proof, both directions, full stack (daemon-spawned runner, Bluesky on the repro sim, exact head):
3. Field-shape race (relaunch → full snapshot → immediate press, 5 attempts): zero natural recorded failures — expected and corroborating: with the dev-env backend down and #1587's fixes active there is no hostile tree to saturate the channel, which is precisely the causal story. The disease needs the grind; the grind is gone. 4. Body updated: Gates: format/lint/typecheck clean, all 11 tap-outcome TS tests pass, runner test-build compiles, in-bundle gate test passed pre-rework on identical logic. 🤖 Addressed by Claude Code |
|
Re-reviewed |
…oboration (#1605 merge gate) The field failure cannot be reproduced on this head: the tap false-failures were a downstream symptom of XCTest-channel saturation, which the #1587 capture fixes removed. The seam records a real XCTIssue AFTER the real gesture inside the per-command failure-count window, so xctestRecordedFailureResponse and target invalidation fire byte-for-byte like the field failure. Armed via a decrementing /tmp flag file (the daemon regenerates tampered xctestrun templates, so env plumbing cannot reach a daemon-spawned runner); compiled only under AGENT_DEVICE_RUNNER_UNIT_TESTS. Live evidence on a daemon-spawned runner (Bluesky, ad-bsky-repro sim): - landed case: injected failure on a real Search-tab tap -> success with the corroboration warning, screen verifiably on Search, no redispatch, runner serving next commands; flag consumed exactly once. - unchanged case: injected failure on a dead-coordinate tap -> capture unchanged -> XCTEST_RECORDED_FAILURE preserved with the new honest hint; runner still usable. - field-shape race (relaunch -> full snapshot -> immediate press, 5 attempts): no natural recorded failure occurs on this head — the hostile tree needed for channel saturation is gone, corroborating the causal story.
…oboration (#1605 merge gate) The field failure cannot be reproduced on this head: the tap false-failures were a downstream symptom of XCTest-channel saturation, which the #1587 capture fixes removed. The seam records a real XCTIssue AFTER the real gesture inside the per-command failure-count window, so xctestRecordedFailureResponse and target invalidation fire byte-for-byte like the field failure. Armed via a decrementing /tmp flag file (the daemon regenerates tampered xctestrun templates, so env plumbing cannot reach a daemon-spawned runner); compiled only under AGENT_DEVICE_RUNNER_UNIT_TESTS. Live evidence on a daemon-spawned runner (Bluesky, ad-bsky-repro sim): - landed case: injected failure on a real Search-tab tap -> success with the corroboration warning, screen verifiably on Search, no redispatch, runner serving next commands; flag consumed exactly once. - unchanged case: injected failure on a dead-coordinate tap -> capture unchanged -> XCTEST_RECORDED_FAILURE preserved with the new honest hint; runner still usable. - field-shape race (relaunch -> full snapshot -> immediate press, 5 attempts): no natural recorded failure occurs on this head — the hostile tree needed for channel saturation is gone, corroborating the causal story.
5454517 to
e4e69d1
Compare
|
Rebased onto current |
e4e69d1 to
35a806b
Compare
thymikee
added
the
ready-for-human
|
Re-reviewed exact head |
|
1 participant
Summary
Partially addresses #1599 by corroborating an iOS
XCTEST_RECORDED_FAILUREwith one same-presentation post-action accessibility capture. A recent, comparable changed digest returns success with a warning; stale, keyless, unknown, unchanged, sparse, cross-backend, capture-failed, or presentation-mismatched evidence remains failure.The comparison baseline is limited to five seconds and must declare the same presentation and snapshot backend, preventing unrelated screen drift or incompatible tree shapes from becoming a successful tap. Corroborated successes preserve target-v1 recording evidence and suppress the ordinary no-change retry so a later observation cannot dispatch the tap twice.
The change also resets process-bound snapshot penalty and private-AX state after external relaunch and makes the runner recovery hint honest. It removes the retry-spiral harm described in #1599, but does not claim to root-cause the underlying XCTest recorded-failure storm on the affected profile family.
Validation
pnpm check:affected --run: 126 files, 1,049 tests passed at961c9f8fd.Division of labor (#1599)
#1587 addresses the likely root cause: the tap false-failures were a downstream symptom of XCTest-channel saturation (grinding captures leaving abandoned work in testmanagerd), which the penalty/viewport/depth fixes removed — two live bsky-24 attempts and a 5-attempt relaunch/full-snapshot race on this head could not produce a single natural
XCTEST_RECORDED_FAILURE. This PR is defense-in-depth for the residual: when the ambiguous failure does occur, one corroborating capture converts a landed tap to success-with-warning instead of a retry spiral. What remains open on #1599 narrows to confirming the saturation theory against the archived transcripts or a pre-#1587 build.