Don't think axtree works with MiniWoB++

[MurtyShikhar]

Hi all,

Thanks for the environment! I was playing around with miniwob++ when i noticed that my agents could not do anything with the axtree for the order.food task on miniwob++ task. On further investigation, i noticed that the axtree contains a lot less information here. For instance: the following is the axtree representation
`

and here is the dom representation:

Importantly, buttons for adding / removing food from cart cannot be accessed from the axtree. Let me know if i'm missing something!

[imenelydiaker]

Hey @MurtyShikhar, thank you for raising this question. Many things to note here:

1. The accessibility tree highly depends on how the website is coded. There are some best practices to respect when coding a website for it to be accessible, like adding aria labels, etc. MiniWoB++ websites are not really made for accessibility, which makes the AXTree less accurate than the DOM.
2. The AXTree you got here is expected. The buttons for adding and removing food from carts are available in the AXTree: StaticText + or StaticText -, because in the original DOM they are <span> elements and not <button>. The click event related to this span elements (texts) is coded in JS in the <script> block in the <header> [you can inspect the code here].
3. With the parameter filter_visible_only=True, the only elements the DOM and AXTree will show are those we can see, so technically you can see elements until "Spinach and goat cheese dip". Although the AXTree here is showing everything, it may be a bug on BG side, the filter is not working maybe. cc @gasse
   

Conclusion: AXTree is highly dependent on how the original website is coded. For MiniWoB++ it's better to use the DOM since it's already small and contains enough attributes like ids and classes that are human-readable.

Hope this answers your question!

[TLSDC]

Hello @MurtyShikhar,
to add to Imene answer, in AgentLab (the repo we use to run our experiments), we use html and explicitly ''fix'' the flags when running on miniwob (ie add the html flag):
https://github.com/ServiceNow/AgentLab/blob/f54dc339d493fff989f1372a8edcc7e85665bc31/src/agentlab/agents/generic_agent/exp_configs.py#L206
I don't think we are planning on fixing this in BGym as it's very specific to miniwob.

[gasse]

3. With the parameter filter_visible_only=True, the only elements the DOM and AXTree will show are those we can see, so technically you can see elements until "Spinach and goat cheese dip". Although the AXTree here is showing everything, it may be a bug on BG side, the filter is not working maybe. cc @gasse

I'll double-check what is going on here. The visible is extracted using Chromium's Intersection Observer API, but we already now it's not 100% reliable as a proxy to what is actually visible to the user on the screen, in particular for non-standard websites.

[gasse]

@imenelydiaker there was something weird with filter_visible_only=True when printing the AXTree, which turned out to require a bit of changes in the way we inject BIDs into the AXTree. I tried to fix it in this PR #119

@MurtyShikhar can you test again on Miniwob using this branch?

You might want to use skip_generic=False when printing the AXTree in MiniWob, so that the div and span elements are displayed and can be interacted with.

[MurtyShikhar]

I tried this, it is better. But still do not get BIDs for the "+" and "-". I think i'm fairly convinced by @imenelydiaker and @ThibaultLSDC - perhaps I should stick to HTML for MiniWoB++ (as has been done in almost all prior work!).

I will close this issue for now. Thanks everyone!