v0.2.1: Set-of-Marks, visibility, bbox and more!

browsergym-core

• New features

  • 🎉 Set-of-Marks 🎉 a new method is available to easily overlay element boxes and bid attributes on top of the screenshot, following ideas from WebVoyager and OSWorld

    from browsergym.utils.obs import overlay_som
    
    ...
    obs, info = env.reset()
    screenshot_with_som = overlay_som(obs["screenshot"], obs["extra_element_properties"], fontsize = 12, linewidth = 2, tag_margin = 2)

    

  • new high-level actions upload_file and mouse_upload_file

  • new field "extra_element_properties" in each observation. Contains a dict with bid keys, which gives the extra properties computed by browsergym for every element with a bid on the current page. Example:

    {
      "23": {
        "visibility": 0.6,  # float between 0 and 1
        "bbox": [56, 345, 12, 20],  # [x, y, width, height]
        "clickable": True,  # boolean
        "set_of_marks": False,  # boolean
    }

  • new set_of_marks property (computed with JS tag browsergym_set_of_marks), following WebVoyager implementation (boolean 0 or 1, whether element should be part of the set-of-marks overlay)

  • new clickable property, extracted from Chrome's DOMSnapshot's isClickable

  • new info fields "action_exec_start", "action_exec_timeout" and "action_exec_stop" after each env.step() call, useful for video editing

  • new resizeable_window parameter in BrowserEnv to switch between setting the viewport size via Chrome (previous behavior, resizeable window and viewport) or via Playwright (new default behaviour, viewport is not resizeable)

• Breaking changes

  • changed visibility tag in JS from browsergym_is_in_viewport (boolean 0 or 1) to browsergym_visibility_ratio (value between 0.0 and 1.0), extracted as the visibility extra property (see new features)
  • BrowserEnv parameters viewport (viewport size), slow_mo (pause between playwright calls) and timeout (default playwright timeout) are now provided by the task. They can still be set in the environment's constructor to override the value provided by the task, which will display a warning.
  • each task inheriting AbstractBrowserTask must now take a seed at instantiation (in constructor), instead of via the task.setup() method. This is also where each task should decide its desired browser setting by setting its attributes task.viewport, task.slow_mo and task.timeout (see point above)

• Refactors

  • bid-based high-level actions fail faster (500 ms)
  • shorter nested bids with alphabetical bids for iframes (21-53 -> a53)
  • fix mouse display position in demo mode (absolute -> fixed)
  • modern chat theme
  • refactored coordinate computation using Chrome's DOMSnapshot instead of JS, should be more robust to edge cases
  • refactored visibility computation using the IntersectionObserver API, should be more robust to edge cases
  • more robust frame marking, supports edge cases such as sandboxed iframes, and pdf viewers in <embed> tags

browsergym-miniwob

• fixed goal conversion to text in task browsergym/miniwob/click-menu-2