Skip to content

Better AXTree / bid marking#119

Merged
gasse merged 3 commits intomainfrom
gasse/patch_33
Sep 5, 2024
Merged

Better AXTree / bid marking#119
gasse merged 3 commits intomainfrom
gasse/patch_33

Conversation

@gasse
Copy link
Copy Markdown
Contributor

@gasse gasse commented Aug 27, 2024

  • more robust remove_redundant_static_text in flatten_axtree_to_str()
  • give bids to AXTree nodes with role="generic" (html divs and spans), by using the aria-description attribute as a fallback for bid injection (using aria-roledescription does not work for AXTree nodes with role="generic")
  • inject bids to all HTML elements, even non-standard ones

Reproducibility study:
gpt4o-mini agent

  • on miniwob 125 tasks x 5 seeds
  • on workarena.l1 33 tasks x 10 seeds

BrowserGym 0.4.2 (before change):

  • miniwob 0.544
  • miniwob with only AXTree 0.379
  • workarena 0.221
  • workarena 0.245
  • workarena 0.236
  • workarena 0.233

BrowserGym 0.5.0.dev0 (after change):

  • miniwob 0.579 -> gain as expected (can interact with div and span)
  • miniwob with only AXTree 0.365 -> within standard error?
  • workarena 0.191 -> within standard error?
  • workarena 0.218 -> within standard error?
  • workarena 0.230 -> within standard error?
  • workarena 0.233 -> within standard error?

@gasse gasse merged commit 49012bf into main Sep 5, 2024
@gasse gasse deleted the gasse/patch_33 branch September 5, 2024 18:06
layahaasini pushed a commit to layahaasini/BrowserGym that referenced this pull request Nov 21, 2025
* more robust remove_redundant_static_text for AXTree

* fallback AXTree bids using aria-description

* mark all html elements (even non-standard tags)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants