Skip to content

Fix frame-aware element extraction and click handling#8

Closed
yifeizhou980429 wants to merge 1 commit into
openonion:mainfrom
yifeizhou980429:fix/frame-aware-element-click
Closed

Fix frame-aware element extraction and click handling#8
yifeizhou980429 wants to merge 1 commit into
openonion:mainfrom
yifeizhou980429:fix/frame-aware-element-click

Conversation

@yifeizhou980429
Copy link
Copy Markdown

Changes in this PR

  1. Add frame metadata to InteractiveElement
    Added a frame field to InteractiveElement
    Default is "main" for main-document elements

  2. Make element extraction frame-aware
    Updated extract_elements() to iterate over all frames, not just the main page
    Each frame now runs element extraction separately
    Frame metadata is passed into the extraction script and stored on each element

  3. Preserve iframe coordinates relative to the top page
    Added frame offset calculation so element coordinates from iframes are normalized relative to the top-level page
    This keeps element positions aligned with top-page screenshots / visual reasoning

  4. Avoid cross-frame index collisions
    Reindexed extracted elements globally after aggregation
    This prevents duplicate element indices across main page and iframe contexts

  5. Make click handling use frame-aware locators
    Updated click() to use frame-aware locator resolution
    If the element belongs to the main page, it uses page.locator(...)
    If the element belongs to an iframe, it resolves the correct frame and uses frame.locator(...)

  6. Add safe fallback for missing frame metadata
    click() now safely defaults missing frame info to "main"
    This keeps the main-page interaction path robust even when frame metadata is absent

This fix makes browser interactions much more reliable for real-world pages that contain embedded frames.

Without this change:
iframe inputs may be matched but not clicked/focused correctly
typing may go to the wrong focus target
page-level keyboard events can happen instead of field input

With this change:
main-page and iframe inputs can both be selected and clicked correctly
focus is established on the intended target more reliably
typing is more likely to go into the correct field

@yifeizhou980429 yifeizhou980429 closed this by deleting the head repository Mar 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants