Bug: `visibility: hidden > visibility: visible` structure isn't included in snapshot

[dkundu56]

here is my prompt:

. Perform File Sharing Operations

• Wait for OneDrive Page to load completely.

• Wait 3-5 seconds for the files to load in the OneDrive interface.

• In the left sidebar, click the item labeled "My files" (with the folder icon below Home).

• Wait for the files to load in the main area.

• Hover slightly left of the file named "Automation_test" in the file list.

• Click the three dots (ellipsis) next to the file name "Automation_test".

• Wait for the dropdown menu to appear.

• Click on "Share" from the dropdown menu.

• After clicking "Share" from the dropdown menu, wait for 5 seconds.

• Locate the iframe with the XPath: "//iframe[@id='shareFrame']".

• Switch context to this iframe before proceeding further.

• Inside the iframe:

  • Locate the input field using the css selector: [placeholder='Add a name, group, or email']
  • Click on the input field to focus.
  • Use keyboard events to type: "autotest2@qascope.onmicrosoft.com" into the input field.
  • Wait for dropdown to appear
  • Press the Enter key to select it.
  • Wait until the blue "Send" button is visible.
  • Click on the blue "Send" button.

• Wait for a confirmation message or UI feedback indicating the share was successful.

But still its not be able to switch to iframe and perform the certain actions.

[pavelfeldman]

There is no switching and no specifying the iframe ids. Playwright MCP is like a human, what you see is what you get. It should just work.

[dkundu56]

@pavelfeldman Thanks for your prompt response.

This is Microsoft one drive page where I want to send the doc to any mail id and I am able to come to this pop-up dialogue and then I am trying to enter any mail id in the input field, but it was not able to do so. [PFA screenshot]

It says:

Result: I have logged into Microsoft 365, opened OneDrive, and located the file "Presentation.pptx". I have opened the sharing dialog for the file. However, I encountered difficulty in interacting with the sharing dialog iframe to add the recipient email and send the share invitation.

Thats why I have changed my prompt to use iframe. Any idea about this?

[Skn0tt]

This feels similar to #447. @dkundu56 could you ask your LLM to take a browser_snapshot and paste the tool result here?

[dkundu56]

@Skn0tt now this is my prompt as you suggested:

1. Login to Microsoft 365 using creds -> username: ******, password: ****** and Click Yes in Stay Signed In Page.
2. Click on Apps then click on OneDrive and take a browser_snapshot at this point
3. Perform File Sharing Operations
   • Wait 5 seconds for files to display.
   • Hover over the file named "sharepoint_sample_pdf" but do not click on it or open the file.
   • Click the three dots (ellipsis) next to the file name: "sharepoint_sample_pdf"
   • Wait for the dropdown menu to appear.
   • Click the "Share" button from the dropdown.
   • Take a browser_snapshot after clicking the "Share" button.
   • Hover over the pop-up dialog to ensure it is fully loaded.
   • Click on the input field named "Add a name, group, or email" and then from the dropdown, select the user "autotest2"
   • Click the "Send" button to share the file.
4. Close the browser session

Tool Result: I have completed the steps to login to Microsoft 365 with the provided credentials, navigated to OneDrive, hovered over the file "sharepoint_sample_pdf", clicked the options (three dots) next to it, clicked the "Share" button to open the sharing dialog, but I encountered some difficulty interacting with the sharing dialog's input field and selecting the user "autotest2." Finally, I have closed the browser session.

It failed here again.

Anything I missed or need to update?

[Skn0tt]

Please share the tool call result of the browser_snapshot call. I'd like to take a look at the page structure as perceived by Playwright MCP.

[dkundu56]

browser_snapshot.json

@Skn0tt please have a look.

[Skn0tt]

Thank you. Everything looks good on that snapshot, it lists the iframe contents. So your agent has everything it needs to interact with elements inside the iframe.

Your prompt mentions "switching to the iframe context". Switching between frames is not necessary with Playwright MCP, the agent can operate across the entire frame tree. It's abstracted away, so you don't need to worry.

Does that answer your question?

[dkundu56]

Thanks @Skn0tt for clarification, but I am using this prompt now:

Use the Playwright MCP Server to execute the following steps:

1. Login to Microsoft 365 using creds -> username: *******, password: *****and Click Yes in Stay Signed In Page.
2. Click on Apps then click on OneDrive and take a browser_snapshot at this point
3. Perform File Sharing Operations
   - Wait 5 seconds for files to display.
   - Hover over the file named "sharepoint_sample_pdf" but do not click on it or open the file.
   - Click the three dots (ellipsis) next to the file name: "sharepoint_sample_pdf"
   - Wait for the dropdown menu to appear.
   - Click the "Share" button from the dropdown.
   - Take a browser_snapshot after clicking the "Share" button.
   - Hover over the pop-up dialog to ensure it is fully loaded.
   - Click on the input field named "Add a name, group, or email" and then from the dropdown, select the user "autotest2"
   - Click the "Send" button to share the file.
6. Close the browser session

and now I am not telling my agent to switching to iframe. But its failing everytime in that diaologue pop up.

Please have a look at my prompt and let me know if anything needs to be updated.

I really want this issue to be solved then it will hugely impact our automation process and I will implement playwright mcp server to automate the web interactions scenarioes.

[Skn0tt]

Let me try reproducing this myself.

[dkundu56]

Thanks a lot @Skn0tt that would be of great help.
Let me know the update and if anything needs to be modified/corrected from my side.

[dkundu56]

I have also tried this prompt for file creation in one drive, this also got failed while selecting 'Word document' from the dropdown:

Use the Playwright MCP Server to execute the following steps:

1. Login to Microsoft 365 using creds -> username: ******, password: ***** and Click Yes in Stay Signed In Page.
2. Click on Apps then click on OneDrive
3. Take a browser_snapshot at this point
4. Perform File Creation Operations
    - Wait 5 seconds for files to display.
    - Click on "Add New" button or "+" icon to create a new file
    - Wait for the dropdown menu to appear.
    - Click the "Word document" button from the dropdown menu.
    - Wait for pop-up dialogue to appear
    - Give random name to the new document with prefix "test_doc" and click on "Create"
    - Wait for the new document to open.
    - Type "This is a test document" in the new Word document.
5. Close the browser session

Result: The login and navigation to OneDrive was successful and a browser snapshot was taken. However, I encountered difficulties interacting with the dialog to name the new Word document and create it. The interaction with the dialog elements for naming and creation is not accessible using the current selectors and context.

FYI - @Skn0tt

[dkundu56]

Hello @Skn0tt did you get any chance to look into the issue and reproduce?

[Skn0tt]

Yes, I was able to reproduce and minimise to microsoft/playwright#36296. The root problem is that Playwright doesn't interpret visibility correctly when creating an ARIA snapshot. So it's similar to #447 (comment).