feat: [android-WIP] enhance DescribeNode interface#190
Closed
pFornagiel wants to merge 1 commit intofeat/android-emulator-supportfrom
Closed
feat: [android-WIP] enhance DescribeNode interface#190pFornagiel wants to merge 1 commit intofeat/android-emulator-supportfrom
pFornagiel wants to merge 1 commit intofeat/android-emulator-supportfrom
Conversation
pFornagiel
changed the title
latekvo
added a commit
that referenced
this pull request
May 5, 2026
…ractivity flags Replace the v1 uiautomator parser with the v2 interactables-only trim from #190. The pruner now: - runs `uiautomator dump --compressed` on-device, which strips the nodes Android's accessibility layer would already skip (decorative wrappers, RN SVG sub-paths, bounds-less Compose group containers); empirically cuts a Bluesky thread dump 65 KB → 23 KB, 181 → 64 nodes with zero loss of useful info; - drops nodes whose pixel rect falls outside the screen, and drops descendants whose rect falls outside an ancestor scroll's clip rect (counted into a `scrollHidden` field so the agent knows to swipe before tapping); - drops `com.horcrux.svg.{Path,Group,Svg}View` subtrees and SystemUI chrome (status bar / nav bar background / quick settings) by default; - collapses a clickable parent + clickable child with identical bounds to the inner node, drops StaticText children whose label is a substring of an interactive parent's label, flattens layout containers and decorative ImageViews with no own info, and aggregates descendant labels into a clickable container that has no label of its own (so a profile-row tap target reads as "Alice / @alice" instead of "(no label)"); - treats `android.webkit.WebView` as an opaque single leaf — the accessibility scaffold under a WebView is always misleading; - redacts password fields: label becomes `[password]` and the secret text never reaches `value`. Surface the trim's findings on `DescribeNode` as 8 optional booleans / counters: `clickable`, `longClickable`, `scrollable`, `checkable`, `checked`, `disabled`, `password`, `scrollHidden`. iOS payloads leave them unset, so existing consumers are unaffected. The trim is keyed on absolute pixel bounds (parsed once, reused for visibility / clip / equality checks) and lowered to the public DescribeNode contract in a separate post-order pass; deeply nested hierarchies stay safe from a stack overflow. Verified end-to-end on a booted Pixel_6_API_34 against the Bluesky home feed: 44 nodes, every action button preserved with the right label, 33 clickable / 4 scrollable flags populated, no stray "0"/"2"/"8" leaf texts inside the action buttons, 8 KB JSON payload. Co-authored-by: Paweł Fornagiel <pawel.fornagiel2@gmail.com>
latekvo
added a commit
that referenced
this pull request
May 5, 2026
…croll-clip, SVG, system chrome Inline-XML coverage for every trim rule landed in the v2 parser refactor (commit 0128919, ported from #190): - duplicate-wrapper collapse (clickable parent + clickable child with identical bounds → keep inner) - password redaction (label="[password]", value never set, password flag true) - WebView opacity (DOM-side accessibility scaffold under the WebView must not bleed through as siblings) - compound-clickable label aggregation ("Alice / @alice" for a row whose own content-desc is empty) - EditText label/value separation (content-desc → label, text → value) - scroll-clip filtering with `scrollHidden` count (descendant outside ancestor scroll's viewport gets dropped, parent records the count) - com.android.systemui drop by default + `includeSystem` opt-in - React Native SVG sub-paths (com.horcrux.svg.{Path,Group,Svg}View) stripped wholesale - off-screen rect drop The fixture-based "Bluesky thread" sanity suite from #190 isn't ported because the dump it depends on (`research/android-ui-inspection/ artifacts/02_uiautomator_compressed.xml`) doesn't ship in this repo. The inline cases above cover every trim rule that suite exercises without needing an external artifact. Co-authored-by: Paweł Fornagiel <pawel.fornagiel2@gmail.com>
latekvo
added a commit
that referenced
this pull request
May 5, 2026
…croll-clip, SVG, system chrome Inline-XML coverage for every trim rule landed in the v2 parser refactor (commit 0128919, ported from #190): - duplicate-wrapper collapse (clickable parent + clickable child with identical bounds → keep inner) - password redaction (label="[password]", value never set, password flag true) - WebView opacity (DOM-side accessibility scaffold under the WebView must not bleed through as siblings) - compound-clickable label aggregation ("Alice / @alice" for a row whose own content-desc is empty) - EditText label/value separation (content-desc → label, text → value) - scroll-clip filtering with `scrollHidden` count (descendant outside ancestor scroll's viewport gets dropped, parent records the count) - com.android.systemui drop by default + `includeSystem` opt-in - React Native SVG sub-paths (com.horcrux.svg.{Path,Group,Svg}View) stripped wholesale - off-screen rect drop The fixture-based "Bluesky thread" sanity suite from #190 isn't ported because the dump it depends on (`research/android-ui-inspection/ artifacts/02_uiautomator_compressed.xml`) doesn't ship in this repo. The inline cases above cover every trim rule that suite exercises without needing an external artifact. Co-authored-by: Paweł Fornagiel <pawel.fornagiel2@gmail.com>
latekvo
added a commit
that referenced
this pull request
May 5, 2026
…croll-clip, SVG, system chrome Inline-XML coverage for every trim rule landed in the v2 parser refactor (commit 0128919, ported from #190): - duplicate-wrapper collapse (clickable parent + clickable child with identical bounds → keep inner) - password redaction (label="[password]", value never set, password flag true) - WebView opacity (DOM-side accessibility scaffold under the WebView must not bleed through as siblings) - compound-clickable label aggregation ("Alice / @alice" for a row whose own content-desc is empty) - EditText label/value separation (content-desc → label, text → value) - scroll-clip filtering with `scrollHidden` count (descendant outside ancestor scroll's viewport gets dropped, parent records the count) - com.android.systemui drop by default + `includeSystem` opt-in - React Native SVG sub-paths (com.horcrux.svg.{Path,Group,Svg}View) stripped wholesale - off-screen rect drop The fixture-based "Bluesky thread" sanity suite from #190 isn't ported because the dump it depends on (`research/android-ui-inspection/ artifacts/02_uiautomator_compressed.xml`) doesn't ship in this repo. The inline cases above cover every trim rule that suite exercises without needing an external artifact. Co-authored-by: Paweł Fornagiel <pawel.fornagiel2@gmail.com>
Member
|
discussed in person - added to android branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As stated, for reference implementation attempt