Skip to content

feat: parse and preserve .ad target-v1 evidence (ADR 0012 migration step 3)#1196

Open
thymikee wants to merge 9 commits into
mainfrom
feat/ad-target-annotations
Open

feat: parse and preserve .ad target-v1 evidence (ADR 0012 migration step 3)#1196
thymikee wants to merge 9 commits into
mainfrom
feat/ad-target-annotations

Conversation

@thymikee

Copy link
Copy Markdown
Member

Summary

Implements ADR 0012 migration step 3 — decision 3's INERT target-v1 .ad parser/writer (docs/adr/0012-interactive-replay.md). Recordings gain per-action identity annotations; replay parses and preserves them but does not enforce anything yet (that is a later migration step, gated on step 2's divergence transport).

  • Writer (src/daemon/session-target-evidence.ts): for every click/press/longpress/fill action that resolves an element target (ref or selector, through the full runtime resolution path or the native-ref fast path), emits # agent-device:target-v1 {...} immediately before the action line. Computes decision 3's record-time write algorithm steps 1-5: identity tuple (id or role+normalized label, where role = normalizeType(node.type ?? "") exactly as buildSelectorChainForNode uses), leaf-anchored ancestry (K=8, nearest-first, root-side truncated), sibling (genuine same-parent child ordinal via parentIndex), scrollRegion/viewportOrder (nearest-scrollable-ancestor partition, ordered by rect center then document order), diagnostic rect, and a record-time self-check that runs the shared replay-time classification algorithm against the record-time tree to decide verified/unverifiable.
  • Writer-parser invariant: every string field is bounded to 256 UTF-8 bytes before serialization; if the payload still exceeds 4 KiB, ancestry is reduced root-side one entry at a time down to ancestry[0], with the self-check re-run against the actually-reduced tuple at each step; the (spec-asserted-unreachable) terminal overflow falls back to verification: "unverifiable" rather than ever emitting something the parser would reject.
  • Parser (src/replay/target-identity.ts, wired into src/replay/script.ts): accepts known fields in any order, ignores unknown fields, NFC-normalizes (label additionally trims/collapses whitespace), rejects malformed/oversized annotations with INVALID_ARGS (parser rejects, never truncates), binds an annotation only to the immediately-next physical action line (any intervening/blank line is rejected), and treats an unknown future target-vN as an ordinary comment.
  • Rewrite preservation: writeReplayScript's read-then-rewrite (heal/--update) path re-emits parsed target-v1 annotations in canonical form rather than silently dropping them.
  • Nothing else changed: no replay-time enforcement, no divergence kinds, no wire changes — this is exactly migration step 3's scope.

Contradictions / findings surfaced along the way

None that required stopping — the recording pipeline does supply everything decision 3 needs, but two plumbing gaps had to be closed (documented in code comments) since the current pipeline didn't carry them yet, not because the data was unavailable:

  1. preActionNodes/the resolved node never reached session history. interactionResultExtra (src/daemon/handlers/interaction-touch-targets.ts) deliberately allowlists only ref/selector/selectorChain/refLabel/targetHittable/hint/evidence/settle into the shared commonExtra that feeds both the internal session-history payload and the public response. I did not widen that allowlist (that would leak the raw tree into the public response). Instead, buildInteractionResponseData (interaction-touch-response.ts) now attaches node/preActionNodes only onto the internal result/visualization payload it returns, and finalizeTouchInteraction (interaction-common.ts) computes the compact target-v1 evidence from them — gated on session.recordSession so the O(nodes) ancestry/identity-set scan never runs outside an active --save-script recording — then strips the raw node/tree before anything is persisted or logged. The public response never sees node/preActionNodes/targetEvidence (covered by a new test in interaction.test.ts).
  2. The native-ref fast path (click @ref/fill @ref with default options) looked up the node for its offscreen/occlusion preflight but discarded it. preflightNativeRefInteraction (resolution.ts) now also returns that already-fetched node/preActionNodes at zero extra capture cost, so recording gets evidence on that path too, not only the full-resolution path.
  3. Self-consistency bug found and fixed during testing: the identity-set scan was comparing an untruncated candidate identity against the (256-byte-bounded) recorded identity, so a node with an id/label longer than 256 bytes could fail to match itself, corrupting the self-check. Fixed by bounding candidates through the same truncation as the recorded fields (boundedLocalIdentity) — covered by a regression test.
  4. Coverage scoped to click/press/longpress/fill. get/find/wait selector reads don't currently carry node/preActionNodes into their recorded action result (selector-recording.ts's buildGetRecordResult/buildFindRecordResult only keep ref/selector/selectorChain/text), so no annotation is emitted for them. This is a legitimate scope decision, not a silent gap: decision 3's own worked example is a click, and extending to selector reads is separable follow-up work.
  5. The direct-iOS selector fast path stays unannotated for now. Per decision 2 (migration step 1, not this step), the direct-iOS path is supposed to be disabled during recording since it has no daemon tree at all. That disabling is step 1's scope; until it lands, an action dispatched through that fast path during recording simply gets no evidence (no node/tree data exists to compute it from) rather than a fabricated one.

Test plan

  • pnpm typecheck, pnpm lint (oxlint --deny-warnings), pnpm check:layering, pnpm build, pnpm check:mcp-metadata, fallow audit --base origin/main — all clean.
  • pnpm test — full suite: 386 files / 3566 tests, all green (two re-runs; one earlier run tripped the slow-test-gate on unrelated iOS runner-client tests under load, reproduced as pre-existing contention flakiness against origin/main with my changes stashed).
  • New unit coverage against the ADR's parser/writer validation bullet verbatim: src/replay/__tests__/target-identity.test.ts (v1 round trips, old/new reader compat, escaping/Unicode, normalized-role source, leaf-anchored ancestry prefix incl. root-side truncation + inserted-wrapper mismatch, malformed/unbound annotations, rect-diagnostic-only, and the full classification core incl. duplicate/unverifiable paths), src/daemon/__tests__/session-target-evidence.test.ts (record-time write algorithm, writer overflow reduction with max-size K=8 fields, fail-closed self-consistency, and a real-capture-shaped fixture with undefined hittable/anonymous wrapper nodes per the PR fix: make settle tail list real actionable targets #1172 fixture-realism pattern), src/replay/__tests__/script.test.ts and src/daemon/__tests__/session-store.test.ts (binding rule, writeReplayScript preservation end to end through SessionScriptWriter), and src/daemon/handlers/__tests__/interaction.test.ts (evidence computed only while recording, never leaked to the public response).

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.6 MB 1.6 MB +9.7 kB
JS gzip 517.6 kB 520.9 kB +3.3 kB
npm tarball 624.5 kB 627.5 kB +3.0 kB
npm unpacked 2.2 MB 2.2 MB +9.5 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.8 ms 28.4 ms -0.4 ms
CLI --help 56.5 ms 57.3 ms +0.8 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/selector-runtime.js +207 B +87 B
dist/src/interaction.js +228 B +63 B
dist/src/internal/daemon.js +93 B +35 B
dist/src/runtime.js +133 B +30 B
dist/src/session.js -6 B -23 B

@thymikee thymikee left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review — ADR 0012 migration step 3 (.ad target-v1 evidence, inert)

Reviewed src/replay/target-identity.ts, src/daemon/session-target-evidence.ts, the parser wiring in
src/replay/script.ts/script-formatting.ts, the response-shape plumbing in
interaction-touch-response.ts/interaction-common.ts/resolution.ts, and all new tests, line by line
against decision 3. Also independently re-ran/confirmed CI: gh pr checks 1196.

Blocking (CI is currently red, not flaky)

1. Coverage check fails — real regression, not a wire leak. preflightNativeRefInteraction
(resolution.ts:490-522) now spreads node/preActionNodes into the PressCommandResult/
FillCommandResult returned by the native-ref fast path (interactions.ts maybeTapRefTarget/
maybeFillRefTarget, ...preflight spread). test/integration/interaction-contract/native-ref.contract.test.ts
native-ref responseConstruction: fast-path result carries the canonical ref field set asserts
result.node === undefined at this exact layer and now fails (confirmed via gh run view --log-failed:
1/3777 tests failing, isolated to this one assertion). This is an ADR-0011-governed provider-contract test
(NATIVE_REF_COVERAGE guarantee registry), not an incidental unit test.

I traced the full path to confirm there is no actual public wire leak: buildInteractionResponseData
(interaction-touch-response.ts:128-138) attaches node/preActionNodes only to the internal
visualization object, never responseData; interactionResultExtra (interaction-touch-targets.ts:173-198)
is a hard allowlist (ref/refLabel/selectorChain/targetHittable/hint/evidence/settle) that never
copies them across; and extractTargetEvidenceForRecording (interaction-common.ts:102-117) strips them
before sessionStore.recordAction persists anything. So this is a stale contract test, not a security
issue — but it's a real, currently-failing gate the PR's own checklist doesn't mention (pnpm test only runs
the unit-core+android-adb vitest projects; this test lives under provider-integration, exercised only by
pnpm test:coverage/CI's Coverage job, which the PR apparently never ran locally). Needs either an updated
assertion (with a comment tying the new shape to ADR 0012 decision 3) or a guarantee-registry note.

2. Lint & Format check fails. pnpm format:check (oxfmt) reports formatting issues in exactly the 6
files this PR touches: src/daemon/__tests__/session-target-evidence.test.ts,
src/daemon/handlers/interaction-common.ts, src/daemon/session-target-evidence.ts,
src/replay/__tests__/script.test.ts, src/replay/__tests__/target-identity.test.ts,
src/replay/target-identity.ts. Trivial (pnpm format / oxfmt --fix), but currently a red CI check —
oxlint itself passes, so this is a separate, apparently-unrun gate.

Both confirmed independently via gh run view <run> --job <job> --log-failed (exact log excerpts pulled),
not just relayed from a subagent.

Design notes (not blockers for this inert step, but relevant before step 4 lands)

3. classifyTargetBindingMatch path 6 collapses two spec-distinct outcomes. Decision 3 says: when
sibling or region-scoped viewportOrder isolates exactly one member, "compare with W as in paths 4/5" — i.e.
if the isolated member ≠ winner, that's the same class as path 5 (identity-mismatch), distinct from true
non-isolation fall-through (identity-unverifiable, up to 5 candidates). The current return type
(target-identity.ts:416-421) only has {path:6, outcome:'verified'|'unverifiable'} — no reason,
unlike path 5's reason:'unique-but-wrong' — so both sub-cases collapse into one undifferentiated
unverifiable. Harmless today (the writer's self-check only reads the boolean outcome), but the module's
own docstring frames this exact function as what "a future migration step (4) reuses... so the two can never
drift" — as written it can't yet drive decision 4's kind/candidates wire shape without extension. Flag
for whoever picks up step 4, not this PR.

4. Overflow-reduction loop sizes the payload against a placeholder before the real value is known.
computeTargetEvidence's reduction loop (session-target-evidence.ts:89-94) measures
serializeTargetAnnotationV1(candidate) while candidate.verification is still the placeholder
'verified', then overwrites it with the real self-check result — without re-measuring. "unverifiable"
is 4 bytes longer than "verified" once serialized, so if a candidate's placeholder-sized payload lands
within 4 bytes of the 4096 cap and the real self-check yields 'unverifiable', the writer could emit a
payload its own parser would reject — violating the explicit "writer must never emit a payload its own
parser rejects" invariant. The terminal/floor branch (lines 95-107) does re-check and drops rect as a
fallback; the general loop body does not apply the same defense.

I traced the self-check's actual reachability and believe this is very hard to hit in practice: by
construction, the winner is always a member of every candidate set the self-check derives from the winner's
own computed sibling ordinal / region partition, so runRecordTimeSelfCheck returns 'verified' on every
normal path — 'unverifiable' only arises from a genuine capture anomaly (e.g. duplicate node indices),
exactly the rare case decision 3 calls out as a fail-closed safety valve. But that's precisely when the
missing re-check would matter. Recommend applying the same recheck-and-drop-rect treatment uniformly, or
sizing against the worst-case ('unverifiable') string length up front.

Verified sound (spec fidelity, line by line against decision 3)

  • Canonical field order, NFC + label trim/collapse, case sensitivity, empty-after-normalization omission —
    all correct and tested (serializeTargetAnnotationV1 field-order test, NFC/whitespace test).
  • 256-byte field / 4096-byte payload / K=8 ancestry / non-negative-safe-integer bounds all enforced by the
    parser with INVALID_ARGS, including exact boundaries: 4096 bytes accepted, 4097 rejected; 2^53 and
    floats/negatives rejected for sibling/viewportOrder; writer-side truncation is surrogate-pair-safe
    (dedicated test at the exact 3-byte/4-byte-emoji boundary).
  • Parser accepts any field order, ignores unknown fields, treats target-vN (N≠1) as an ordinary comment,
    and correctly rejects a line that merely mentions the tag in prose (regex is start-anchored).
  • Binding rule (parseReplayScriptDetailed in script.ts) is a correct state machine: blank line, another
    comment, an env directive, or end-of-script while an annotation is pending all reject with INVALID_ARGS;
    no path lets an annotation silently drift to a non-adjacent action. Both writers
    (session-script-writer.ts's live recorder and script.ts's writeReplayScript heal/rewrite path) push
    the annotation and its action line as consecutive array entries — the binding is structurally guaranteed,
    not just tested incidentally.
  • Ancestry leaf-anchored prefix matching (root-side-truncation tolerant, inserted-wrapper rejecting) and
    local-identity matching (id-wins, absent-label-equals-absent) match decision 3's prose exactly, with tests
    for both directions.
  • The self-caught truncation bug (identity-set scan comparing an untruncated candidate against the bounded
    recorded identity) is genuinely fixed in 3f8097d9b. I grepped every identity-comparison site in
    session-target-evidence.ts: buildAncestryChain, computeScrollRegionKey, and the identitySet filter
    all now go through boundedLocalIdentity consistently — the first two already did before the fix, so the
    fix closed the one remaining gap. Has its own correctly-targeted regression test (500-byte id).
  • Privacy/wire-shape (full trace, see finding #1 above): no public CLI/MCP response shape gains
    node/preActionNodes, and the on-disk .ad/session-history artifacts only ever contain the compact
    target-v1 JSON, never a raw tree.
  • Scope confinement to click/press/longpress/fill is structural, not incidental: buildInteractionResponseData's
    kind:'runtime' branch only accepts PressCommandResult | FillCommandResult | LongPressCommandResult.
    scroll/swipe/pinch/type/get/is never populate result.node, and
    extractTargetEvidenceForRecording's early return degrades silently for all of them, including
    point-target fill (fill x y "text") and type's raw keyboard entry (both confirmed to route through
    finalizeTouchInteraction but never carry node).
  • True inertness confirmed: grepped the whole tree for .targetEvidence reads outside the
    parser/writer/session-recording files touched by this PR — nothing in replay execution consumes it yet.
  • Fixture realism: bottomTabsRealCaptureFixture (session-target-evidence.test.ts) is a genuine
    real-capture shape (undefined hittable on plain text, an anonymous role-only wrapper ancestor,
    hittable:false on real buttons) and exercises ancestry/sibling/scrollRegion plus a full round trip. It
    would not by itself have caught the untruncated-identity bug (no field exceeds 256 bytes there) — that's
    covered by its own separate, precisely-targeted regression test instead, which is the right way to pin a
    boundary condition rather than relying on a "realistic" fixture to organically hit it.

Not completed: live device verification (axis 8)

Spawned a background agent to record a session against the iPhone 17 Pro simulator / RN playground
(open --save-script ... --relaunch + rne://stack-prevent-remove deep link + press "Push Article" + a
fill), cat the resulting .ad, and replay it to confirm inertness. The agent's tool calls were interrupted
by the user mid-run (simulator/build tool permissions denied) and it did not produce a final report before
stopping. I did not force a retry. Given the static trace above is unusually thorough (I read every
comparison/serialization site directly, not just the tests), I'm comfortable shipping this review without
that piece, but it remains a real gap — recommend a manual spot-check of the .ad output (annotation lines
present immediately before press/fill, plausible sibling/viewportOrder, verification:"verified") before
merge, since it's cheap and this is exactly the kind of thing static review can miss.

Gates

  • gh pr checks 1196: 21 checks — 19 pass, Coverage fail, Lint & Format fail (both detailed above).
    Everything else (Typecheck, Build, Bundle Size, CodeQL, Fallow Code Quality, Integration Tests, Layering
    Guard, Smoke Tests ×4, Swift Runner Unit Compile, Packaged CLI, Web Platform Smoke, iOS Runner Swift
    Compatibility) passes.
  • Local pnpm typecheck/pnpm lint/pnpm build: pass.
  • Local pnpm test (unit-core + android-adb, 386 files/3566 tests): green; the 17 failures seen on one full
    run were isolated to platforms/apple/core/__tests__/index.test.ts and runner-client.test.ts
    (timeout-pattern under contention) and passed cleanly when rerun alone — consistent with this repo's known
    flaky-under-contention iOS process-spawn tests, not a regression.
  • pnpm fallow / Fallow Code Quality CI check: clean, no findings against origin/main.

Verdict

Needs changes before merge — two concrete, cheap-to-fix blockers (the stale native-ref contract
assertion, and formatting on the 6 touched files), plus one recommended pre-merge step (live spot-check of a
recorded .ad file, since I couldn't complete that here). The spec-fidelity work itself is strong: I did not
find any deviation from decision 3's normative text in the parser, the binding rule, or the writer's
record-time algorithm, and the one bug the author self-caught (untruncated candidate vs. bounded recorded
identity) is genuinely fixed everywhere it needed to be. The two design notes (#3, #4) are worth a comment
but not a blocker for an explicitly inert PR.

thymikee added a commit that referenced this pull request Jul 10, 2026
…reasons, worst-case sizing

- native-ref contract test: deliberately updated to assert the new ADR 0012
  boundary — the preflight's node/preActionNodes ride the INTERNAL runtime
  result and visualization payload only; the public responseData never
  carries node/preActionNodes/targetEvidence (asserted directly against
  buildInteractionResponseData).
- oxfmt formatting on all touched files.
- classifyTargetBindingMatch path 6 now distinguishes decision 3's two
  spec-distinct outcomes: a signal isolating a member that differs from the
  winner ('signal-isolated-wrong', the paths-4/5 comparison class, future
  identity-mismatch) vs true fall-through ('no-signal-isolation', future
  identity-unverifiable), so step 4 can consume the reasons directly.
- writer reduction loop sizes every candidate against the worst-case
  verification value ("unverifiable" is 4 bytes longer than "verified"), so
  a fail-closed self-check downgrade can never push an accepted payload over
  the 4 KiB cap; pinned by a 1-byte-granularity sweep across the boundary
  that fails against the old placeholder-sized check.
@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the thorough review — all four findings addressed in 7951072.

Blocker 1 (Coverage / native-ref contract test): Updated native-ref.contract.test.ts responseConstruction deliberately to assert the new correct boundary rather than the stale shape. The test now asserts the ADR 0012 split explicitly: the preflight's already-fetched node/preActionNodes ARE present on the internal runtime result (that reuse is the point — zero extra capture cost for record-time evidence), and — asserted directly against buildInteractionResponseData — they ride only the internal visualization payload while the public responseData never carries node/preActionNodes/targetEvidence. Comment in the test ties the boundary to decision 3 and to extractTargetEvidenceForRecording's pre-persistence strip. You were right that my local gate never ran this project: ran vitest --project provider-integration --project interaction-contract locally now — 39 files / 168 tests green.

Blocker 2 (oxfmt): pnpm format run; pnpm format:check clean locally.

Design note 3 (path-6 collapse): Took this now rather than deferring — classifyTargetBindingMatch's return type distinguishes the two spec-distinct path-6 outcomes: { path: 6, outcome: 'unverifiable', reason: 'signal-isolated-wrong' } when a signal isolates exactly one member that differs from W (decision 3's "compare with W as in paths 4/5" — the identity-mismatch class), vs reason: 'no-signal-isolation' for true fall-through (the identity-unverifiable-with-candidates class). Step 4 can now map reason straight onto divergence kind without re-deriving. New test covers both isolation routes (sibling and region-scoped viewportOrder) denoting a non-winner.

Design note 4 (placeholder sizing): Also taken now, via your second suggested option: the reduction loop sizes every candidate against the worst-case serialization (verification: "unverifiable", 4 bytes longer), so whatever the real self-check returns provably fits — the 4-byte window is closed structurally instead of patched after the fact. Pinned by a new test that sweeps a tunable root-side label length in 1-byte steps across the 4 KiB boundary and asserts the emitted evidence still fits when re-serialized with the longer verification value; I verified the sweep fails against the old placeholder-sized check (overflow at sweep offset 179) before landing the fix. The unreachable floor branch keeps its existing recheck-and-drop-rect defense.

Live verification (axis 8): Deferred — the simulator is currently in active use by the maintainer, so I'm not opening sessions or touching Metro ports right now. Will run the recorded-.ad spot-check (annotation lines immediately before press/fill, plausible sibling/viewportOrder, verification:"verified", replay-inertness) as a pre-merge step once the device frees up, and paste the annotation lines here.

Local gates after the changes: typecheck, oxlint, format:check, layering, build, fallow audit vs origin/main all clean; full pnpm test green (386 files / 3567 tests — one contended run reproduced the known flaky iOS process-spawn timeouts, all passing in isolation); provider-integration + interaction-contract projects green.

@thymikee

Copy link
Copy Markdown
Member Author

Review — 🟢 SHIP-WITH-NITS

A large but well-scoped and exceptionally well-tested change: an inert target-v1 writer (session-target-evidence.ts), a tree-agnostic parser/spine (replay/target-identity.ts), and binding/preservation wiring into the replay script parser and writer. I traced the full data path (resolution.tsbuildInteractionResponseData attaches node/preActionNodes onto the visualization channel only → dispatchRuntimeInteractionfinalizeTouchInteraction strips them and computes evidence gated on recordSession) and confirmed the raw AX tree never reaches the public response, session history, or overlay telemetry. The parser rejects malformed/oversized/unbound annotations with INVALID_ARGS and treats future target-vN as ordinary comments. The "inert, no enforcement" scope holds.

Findings:

  1. src/replay/target-identity.ts:2039 (low, hardening) — parseRequiredRoleField(undefined) returns '' for a missing top-level role. An adversarial/hand-authored # …target-v1 {"verification":"verified"} parses to role:'' rather than being rejected. Harmless while inert, but once step-4 enforcement lands, matchesLocalIdentity with an empty role can match anonymous wrapper nodes and yield a false verified/rebind. The empty-role allowance is only spec-justified for ancestry entries. Consider rejecting a missing top-level role (no test covers this case), or add an explicit "missing role is tolerated" note now.
  2. src/daemon/handlers/interaction-touch-response.ts:~132 (cosmetic) — comment overstates data flow. It says node rides on visualization for "session history / touch overlay input," but finalizeTouchInteraction strips it before both recordAction and recordTouchVisualizationEvent. Code is safer than the comment.
  3. src/daemon/handlers/interaction-common.ts:672 (trivial) — no trailing newline. No linter enforces it; cosmetic.

Cleared: truncateToUtf8Bytes surrogate handling, __proto__ prototype-pollution is a no-op (only known keys read, raw never spread), binding rule rejects intervening blank/comment/env/EOF lines, version regex rejects prose and v1abc, overflow reduction re-runs the self-check and fail-closes to unverifiable, the 256-byte self-consistency fix is real and regression-tested, noRecord actions skip both annotation and action line. Faithful to ADR-0012 step 3's parser/writer-only scope.

🤖 Generated with Claude Code

@thymikee

Copy link
Copy Markdown
Member Author

Second-review nits addressed in 06eea52:

1 (missing top-level role): Taken as real hardening, not just a note — the parser now rejects a MISSING role key with INVALID_ARGS at the top level, in every ancestry entry, and in scrollRegion (target-v1 "role" is required. / "ancestry[i].role" / "scrollRegion.role"). Rationale documented at parseRequiredRoleField: the writer emits role unconditionally in all three positions, so absence is only ever hand-edited/adversarial input, and tolerating it as an implicit '' would let step-4's matchesLocalIdentity match anonymous wrapper nodes. An EXPLICIT empty-string role stays accepted — that's writer-legal output for a typeless node per decision 3's ancestry note. Three new tests cover missing-role rejection in all three positions plus the empty-string acceptance.

2 (comment overstating data flow): Reworded — the comment now says visualization is a hand-off channel to finalizeTouchInteraction, which strips the raw tree BEFORE session history (recordAction) and touch overlay telemetry both receive the payload, matching what the code actually does.

3 (trailing newline): Checked all touched files byte-level — every one already ends with \n (interaction-common.ts included, 117 lines). Nothing to change; likely a stale line reference.

Gates rerun after the changes: typecheck, oxlint, format:check clean; the target-identity/script/session-target-evidence suites green (77 tests).

@thymikee

Copy link
Copy Markdown
Member Author

Two blocking ADR 0012 findings on current head 06eea529c:

  1. Direct iOS recording still bypasses target evidence. readDirectIosSelectorTapTarget and readDirectIosSelectorFillTarget disable the fast path for --verify/--settle, but not when session.recordSession is active (interaction-touch.ts:308-320, 630-639). The resulting runner-payload has no node/preActionNodes, so extractTargetEvidenceForRecording emits no target-v1. ADR 0012 explicitly requires recording to disable direct iOS when decision-3 evidence is required. Please gate both direct paths on recording and add routed click/fill recording coverage.

  2. scrollRegion is stricter than the ADR local-identity rule. computeScrollRegionKey records both id and label when both exist, and scrollRegionKeysEqual requires both to remain equal (session-target-evidence.ts:193-217). Decision 3 defines the partition key from local identity: stable ID wins; label is the fallback when no ID is recorded. A label change under the same container ID currently breaks partition matching and can produce a false identity-unverifiable. Please emit/compare ID-or-label according to that precedence and add a stable-ID/changed-label regression.

@thymikee

Copy link
Copy Markdown
Member Author

Self-audit per the maintainer directive ("if a workaround needs a paragraph to justify it, fix the code") — landed in d181c1a (+ c617506 formatting the two test files added in 852c680, which failed format:check):

Redesigned so the explanation became unnecessary:

  • The record-time node/tree now travel on a typed recordedTarget side channel of InteractionResponsePayloads, not smuggled through the visualization Record and stripped later. The construction site routes them there exclusively, finalizeTouchInteraction consumes the channel directly, and the strip helper (extractTargetEvidenceForRecording) plus its justifying doc are deleted. The public/internal split is now the type shape: the raw tree structurally cannot appear in either serialized payload, so the contract-test boundary paragraph collapsed to two one-line comments over direct assertions.
  • findNearestScrollableContainer/findNearestAncestor made generic over the node type — removes a cast and its "the cast is safe because..." comment.
  • computeLocalIdentity/boundedLocalIdentity collapsed into one always-capped identity reader — the raw/bounded split was the root cause of the earlier self-match bug and existed only to be explained at call sites.
  • rejectUnbound in the script parser takes the pending annotation as a parameter — removes a non-null assertion and the comment defending it.

Trimmed to one-line constraint statements (arguments now live in the PR body/history, not the code): the module docs in target-identity.ts/session-target-evidence.ts, the reduction-loop/floor/worst-case-sizing comments, the regex and required-role rationales, the preflight-reuse comment in resolution.ts, and the test-side comments. Spec-mapping docs (what a decision-3 field/path means) stay — those define semantics rather than argue acceptability.

Also rebased onto 852c680 (thanks for the recording-time direct-iOS gating + scrollRegion id-precedence) — the structural-channel refactor merged cleanly with it, net −75 lines.

Gates after the rebase: typecheck / oxlint / format:check / layering / build / fallow clean; unit + interaction-contract + provider-integration projects green (three provider timeouts under parallel-project contention, each passing in isolation).

@thymikee

Copy link
Copy Markdown
Member Author

Current head c6175069 still has two functional blockers and one repo-structure violation:

  1. Recorded get text/attrs actions receive no target-v1 evidence. Both runtime and direct-iOS query paths record through recordIfSession, which has no target-evidence channel, and the direct query remains enabled during recording. ADR 0012 decision 3 covers every element-resolving action and explicitly includes get context. Thread the pre-action node/tree into recording and gate the direct query when recording requires evidence; add routed get recording coverage.

  2. Missing parent linkage can be recorded as verification: "verified". buildAncestryChain silently turns an unresolved parentIndex into an empty root-like chain, after which the self-check verifies against its own derived identity set. ADR 0012 classifies missing parent linkage as a capture anomaly; fail closed to unverifiable and add a broken-parent regression.

  3. The test split is incomplete. This PR still adds evidence/privacy cases to the existing 3,600+ line interaction.test.ts, which AGENTS.md explicitly names as an aggregation not to grow. Move the new cases to focused topology-mirroring test files.

Also still required before readiness: live simulator/device evidence that records click/fill/get annotations to a real .ad, closes/flushed it, and replays that same file unchanged with annotations inert.

thymikee added 9 commits July 10, 2026 17:52
…tep 3)

Recording now emits a `# agent-device:target-v1 {...}` comment immediately
before every click/press/longpress/fill action that resolves through the
tree (ref or selector), carrying the identity/ancestry/sibling/scrollRegion/
viewportOrder tuple from decision 3's record-time write algorithm plus a
record-time self-check (verified/unverifiable). The parser accepts known
fields in any order, NFC-normalizes, rejects malformed/oversized annotations
with INVALID_ARGS, binds an annotation only to the immediately-next action
line, and treats unknown future target-vN comments as ordinary comments.
writeReplayScript's read-then-rewrite (heal) path preserves v1 annotations
in canonical form. Nothing consumes the parsed evidence yet beyond
preservation — replay-time enforcement is a later migration step.

The identity tuple's own node/tree only ever lived on the internal
visualization/session-history payload (never the public response); it is
converted to the compact target-v1 evidence and stripped before anything is
persisted or returned.
The identity-set scan in computeTargetEvidence compared an untruncated
candidate identity against the 256-byte-bounded recorded identity, so a
node whose own id/label exceeds the field cap could fail to match itself,
corrupting the record-time self-check. Compare through the same bounding
on both sides instead.
…reasons, worst-case sizing

- native-ref contract test: deliberately updated to assert the new ADR 0012
  boundary — the preflight's node/preActionNodes ride the INTERNAL runtime
  result and visualization payload only; the public responseData never
  carries node/preActionNodes/targetEvidence (asserted directly against
  buildInteractionResponseData).
- oxfmt formatting on all touched files.
- classifyTargetBindingMatch path 6 now distinguishes decision 3's two
  spec-distinct outcomes: a signal isolating a member that differs from the
  winner ('signal-isolated-wrong', the paths-4/5 comparison class, future
  identity-mismatch) vs true fall-through ('no-signal-isolation', future
  identity-unverifiable), so step 4 can consume the reasons directly.
- writer reduction loop sizes every candidate against the worst-case
  verification value ("unverifiable" is 4 bytes longer than "verified"), so
  a fail-closed self-check downgrade can never push an accepted payload over
  the 4 KiB cap; pinned by a 1-byte-granularity sweep across the boundary
  that fails against the old placeholder-sized check.
…flow comment

Second-review nits: the writer emits `role` unconditionally (top level,
ancestry entries, scrollRegion) — possibly as the empty string for typeless
nodes, which stays accepted — so a MISSING role key can only come from a
hand-edited/adversarial annotation and is now rejected with INVALID_ARGS
instead of silently parsing as an implicit empty role, which step-4
enforcement could otherwise match against anonymous wrapper nodes. Also
corrects the interaction-touch-response comment that overstated where the
raw node/tree flows (finalizeTouchInteraction strips it before both session
history and touch overlay telemetry).
…ment-comments

Maintainer directive: comments that argue a workaround is acceptable mark
code to redesign. Applied to the whole PR diff:

- The record-time node/tree now travel on a typed `recordedTarget` side
  channel of InteractionResponsePayloads instead of being smuggled through
  the visualization Record and stripped later. The construction site routes
  them there exclusively, finalizeTouchInteraction consumes the channel
  directly, and extractTargetEvidenceForRecording (the strip helper and its
  justifying doc) is deleted — the public/internal split is now enforced by
  the type shape, so the contract test asserts it in two lines instead of a
  paragraph.
- findNearestScrollableContainer/findNearestAncestor made generic over the
  node type, removing a cast plus its safety-argument comment.
- computeLocalIdentity/boundedLocalIdentity collapsed into one always-capped
  identity reader — the raw/bounded split was the root of the earlier
  self-match bug and existed only to be explained.
- parseReplayScriptDetailed's rejectUnbound takes the pending annotation as
  a parameter, removing a non-null assertion and its comment.
- Remaining paragraph-length argument-comments trimmed to one-line
  constraint statements (module docs, sizing/floor comments, regex/role
  rationales, test comments); spec-mapping docs stay.
Maintainer blockers on ADR 0012 step 3:

- get text/attrs now records target-v1 evidence: GetCommandResult carries
  the resolution tree (preActionNodes, internal — neither the recorded
  result nor the public payload copies it), recordIfSession gains the typed
  recordedTarget channel, and dispatchGetViaRuntime threads the capture
  through. The direct-iOS get query is gated during recording so the
  snapshot path supplies the evidence tree, mirroring the tap/fill gating.
  find/is stay uncovered: their results are match-set shaped (found flags,
  predicate booleans over possibly-many matches), not a single resolved
  winner — noted in the PR.
- buildAncestryChain now reports a broken parent walk (dangling parentIndex
  or cycle) instead of silently producing a root-like chain; the writer
  fails the annotation closed to 'unverifiable' per decision 3's
  capture-anomaly rule, and broken candidates cannot prove an ancestry
  prefix. Regression tests for both anomaly shapes.
- New ADR 0012 recording tests live in focused files
  (interaction-target-evidence.test.ts) instead of growing the
  interaction.test.ts aggregation; the earlier additions moved out.
@thymikee thymikee force-pushed the feat/ad-target-annotations branch from c617506 to 5b6e932 Compare July 10, 2026 16:00
@thymikee

Copy link
Copy Markdown
Member Author

All three blockers from the readiness review addressed in 68b5698 + 5b6e932 (branch rebased onto current main, so the new production-unused-exports ratchet and the layering cycle guard both ran against this work):

1. Recorded get text/attrs now carries target-v1 evidence. GetCommandResult carries the resolution tree (preActionNodes — internal: neither buildGetRecordResult nor toDaemonGetData copies it, asserted in tests), recordIfSession gained the same typed recordedTarget channel the touch path uses (evidence computed only under session.recordSession), and dispatchGetViaRuntime threads the capture through for both @ref and selector targets. The direct-iOS get query is now gated during recording (readDirectIosGetSelector), mirroring 852c680's tap/fill gating, so the snapshot path supplies the evidence tree — covered by a routed test asserting the runner querySelector is not used and the recorded action carries verified evidence including the ancestry from the resolution tree. Scope note on the remaining reads: find and is results are match-set shaped (found flags / predicate booleans over possibly-many matches, is exists deliberately counts rather than resolves), not a single resolved winner, so decision 3's identity tuple has no well-defined subject there today — excluded deliberately, not silently, and cheap to add later if their results ever grow a winner.

2. Missing parent linkage now fails closed. buildAncestryChain reports a broken walk — a parentIndex that resolves to no node, or a parent cycle — instead of silently producing a root-like chain. A broken winner walk forces verification: "unverifiable" (the self-check is bypassed: its structural signals cannot be trusted), and a broken candidate walk can no longer prove an ancestry prefix into the identity set. Regression tests cover both anomaly shapes (dangling parentIndex, cycle).

3. Test split. The earlier evidence/privacy cases are moved out of interaction.test.ts (net −107 lines there) into the focused src/daemon/handlers/__tests__/interaction-target-evidence.test.ts, which also hosts the new routed get-recording coverage.

Also on this push: rebase onto main is clean; the new check:production-exports gate passes (dropped a production-unreferenced type re-export it would have flagged); a fallow complexity finding introduced by the recording gate in dispatchDirectIosSelectorGet was fixed by extracting the guard chain rather than waiving it.

Live simulator evidence (record click/fill/get to a real .ad, flush, replay unchanged with annotations inert) is being produced as a separate spot-check run and will be posted here — the branch is otherwise ready for it: get evidence lands in the same .ad shape as touch evidence, immediately before its action line.

Gates: typecheck / oxlint / format:check / layering (incl. the new cycle guard) / build / check:production-exports / fallow audit vs main — all clean. Unit + interaction-contract + provider-integration projects green (the only full-run failures were the documented flaky-under-contention process-spawn tests, changing set per run, each passing in isolation).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant