Skip to content

fix(flow): refuse invisible-only text selectors — record coordinates instead#511

Merged
filip131311 merged 1 commit into
mainfrom
filip/flow-selector-visible-text
Jul 15, 2026
Merged

fix(flow): refuse invisible-only text selectors — record coordinates instead#511
filip131311 merged 1 commit into
mainfrom
filip/flow-selector-visible-text

Conversation

@filip131311

Copy link
Copy Markdown
Collaborator

The bug (found in the 0.15.0 full-surface QA, on both the Android emulator and a physical S24)

Recording a gesture-tap on an icon-font tab-bar item wrote a flow step that displays as an empty selector, with the coordinates gone:

steps:
  - tap:
      text:      # ← actually U+E163, an icon-font Private Use Area glyph

and flow-execute then silently passed against target "", tapping an element no reader of the flow can predict. Root cause (hexdump of the QA-recorded file: ee 85 a3 = U+E163): icon fonts expose their glyph as the element's accessibility label; deriveSelector accepted any label that survived trim(), so a PUA code point counted as "stable text". This violated the rewrite's own documented contract — "otherwise coordinates are kept with a warning" — whose fallback path existed and worked, but never ran for this case.

The fix — one predicate, three layers

hasVisibleText() (strip Unicode Cf format / Co private-use / Cc control, then trim; require something left):

  1. deriveSelector — invisible-only label/value is not stable text; fall through to role, then to the recorder's existing kept-coordinates path.
  2. selectorToYaml — the empty-text guard previously covered only the bare-string spelling; it now also rejects empty/invisible text on the strict {text: …} map form, so a hand-built selector fails loudly instead of writing an unreadable flow file.
  3. selectorSchema — parse-side refinement, so flow files already recorded broken on 0.15.0 fail with an actionable message ("text must contain at least one visible character … select by identifier or role instead") instead of false-passing at replay. This also hardens the await-ui-element tool surface, which shares the schema.

Text that merely contains an icon glyph alongside real text ("⟨U+E163⟩ Explore") stays valid throughout.

Verification

  • Unit: deriveSelector PUA/ZWSP/mixed cases, serialize + parse rejection round-trips (explicit \uXXXX escapes, no invisible bytes in the test source).
  • Full tool-server suite: 2667 passed.
  • Live on the Pixel_9 emulator against the original QA repro (my-app tab bar, same coordinates): recording now produces tap: {x: 0.75, y: 0.947} with the documented "kept coordinates (brittle)" warning — the role fallback was correctly caught by the existing resolves-to-a-different-element guard — replay passes 2/2 on the coordinate targets, and executing the old broken YAML now fails parse with the visible-character error.

🤖 Generated with Claude Code

…instead

The recorder's gesture-tap→selector rewrite trusted any label that survived
trim(). Icon-font tab bars expose their glyph as the accessibility label —
a Private Use Area code point like U+E163 — so deriveSelector minted a
'stable' text selector that serializes to YAML which DISPLAYS as
`text:` (empty), is meaningless outside the app's private font, and at
replay silently matched an element no reader of the flow can predict.
Reproduced from the 0.15.0 QA flow files (hexdump: ee 85 a3 = U+E163).

Three layers, one predicate (hasVisibleText: strip Cf/Co/Cc, then trim):

- deriveSelector: text with no visible characters is not stable; fall
  through to role, then to the recorder's kept-coordinates path — which
  already worked and now actually runs for this case.
- selectorToYaml: the empty-text guard now covers the strict map spelling
  too (it only guarded the bare-string form) and rejects invisible-only
  text — a hand-built selector fails loudly instead of writing an
  unreadable flow.
- selectorSchema: parse-side refinement rejects `text` with no visible
  characters, so flow files already recorded broken on 0.15.0 fail with
  an actionable message instead of false-passing at replay.

Live-verified on the Pixel_9 emulator against the original QA repro:
the same tab-bar tap now records `tap: {x, y}` with the documented
'kept coordinates (brittle)' warning, replays 2/2, and the old broken
YAML fails parse with the visible-character error.
@filip131311
filip131311 merged commit 92b8c03 into main Jul 15, 2026
6 checks passed
@filip131311
filip131311 deleted the filip/flow-selector-visible-text branch July 15, 2026 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant