Skip to content

v0.5.14: chore: 0.5.14 (#194)

Choose a tag to compare

@github-actions github-actions released this 29 Sep 13:56
· 95 commits to main since this release
04f685d
  • Navigation comparisons enable cloud resizing to match Ultrafast's viewport and retain the actual
    dimensions for both arms.
    Publishing rejects scored runs with missing or different dimensions. Ultrafast final evidence is read
    before its browser driver disconnects, so cleanup cannot change the page being graded.

  • Completion checks read fresh page state after a bounded loading wait, so asynchronous search results cannot
    be judged from the earlier page before they arrive.

  • Navigation evals retain the final document status and content, so an error page at the requested URL cannot
    pass. Browser transport timeouts are retried for both arms, and separate site probes no longer discard
    completed runs. Ultrafast stale decisions do not consume its action budget.

  • Opening a page must leave the requested destination showing. HTTP error pages no longer count as
    successful visits, and action effects report the failed response to completion checks.

  • Same-named controls can be distinguished by their own visible descriptions when they share an ancestor,
    instead of falling back to position alone.

  • The Ultrafast eval adapter recognizes the same provider outage statuses as fastbrowse, including Cloudflare
    520-524 responses, explicit provider retry markers and proxy transport failures, so outages are retried
    instead of counted as agent failures.

  • Live evals retain every outage retry in a separate attempt ledger and keep separate retry recordings.
    Arm launch order rotates between repeats. Slow completed attempts still count with their full time and cost.

  • Eval reports name Browser Use Ultrafast (jev-ultrafast) separately from the hosted Browser Use agent.
    The eval guide includes the command for their shared navigation comparison through OpenRouter.

  • The eval summary retains prior task versions across releases that publish only other suites, so returning
    tasks report their version changes instead of appearing new.