Skip to content

Webstone 0.5.0

Choose a tag to compare

View all tags
@lanyue-llk lanyue-llk released this 11 Nov 07:42
· 14 commits to main since this release
v0.5.0
f3c30da
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Browser Eval 0.5.0 is the first release whose version also names the Web Fetch benchmark edition: the pinned LexBench-Web-Fetch dataset plus this release's scoring code. It moves to dataset v2.2.0, which has corrected references and the same 550 tasks, includes the scorer fixes made since 0.4.0, and adds the Web Extract benchmark track.

Why a minor release

Web Fetch scores can change while the tasks stay the same. Rescore saved 0.4.0 answers with 0.5.0 before comparing them with new runs. The task population is unchanged, so rescoring the saved answers is enough; no new calls are needed.

Changes since 0.4.0

  • One version for the Web Fetch benchmark (#264).
    • lexbench-webfetch version prints the Browser Eval version and its scoring-source hash. It prints X.Y.Z+unreleased when the code differs from the release.
    • docs/releases.md says when to bump the patch, minor or major version, including scoring changes.
  • Corrected dataset (#266, #271). Web Fetch now pins LexBench-Web-Fetch v2.2.0. It adds the reviewed primary-content task scope revision; task hashes are unchanged.
  • Scorer fixes:
    • six scorer defects repaired, each with paired controls (#239);
    • declared image links and catalog cards with a heading price (#250);
    • saved evidence that carries retired metrics (#251);
    • structured page fields (#252);
    • spaced anchors and indented links (#253);
    • escaped punctuation in anchors (#255);
    • Lexmount answer handling (#261, #263);
    • revision references that can't be executed are now reported as unavailable (#262).
  • Lexmount evidence (#224). The native Lexmount API's execution evidence is retained, and category results are verified as uniform (#131).
  • Web Extract track: versioned field-extraction benchmarks, self-hosted engines and local models, and source-qualified URL benchmarks with independent reports (#233, #258, #259, #260).
  • Reports and repository:
    • English is now the source language of the report renderers (#240, #243, #244, #245).
    • Release-check and CI latency improved (#257).
    • Docs and tests were cleaned up (#265, #267, #269, #270).

Install

git clone --branch v0.5.0 https://github.com/lexmount/browser-eval.git
cd browser-eval
uv sync --frozen
uv run --frozen lexbench-webfetch version

The attached wheel and source archive both declare version 0.5.0. SHA256SUMS lists their checksums. Installed outside the checkout, the wheel reports 0.5.0, status: released, with scoring-source hash d0c5f12a609f622e8d97050b7d35581cd41973cd7c6a454d5267241ec2161e2b.

Benchmark input pins

The source is commit f3c30da9b25aeed866b6bc036999adb470af2d1a. Input selections and their file hashes are recorded in the release data lock.

Dataset HF commit
Lexmount/LexBench-Web-Fetch (v2.2.0) c12559d2af7db8ae6e84562ef187a5309811122e
Lexmount/LexBench-Web-Search 76d9c2b685cc6da1f2a2aaceb7daa224e67155da
Lexmount/LexBench-SimpleQA-Verified 550ef09fa823ac4d9b3859ddefed3542b5b7e65c
Lexmount/LexBench-WebMainBench 180b3cc6177df75ac9107ea8c76e3506ee96c5c9
Lexmount/LexBench-WCXB d1381fdd95505c48bc17911479a8182c0d442fb0
Lexmount/LexBench-Trajectory-Replay 27e3b11926a8c18b3b29ba83a2e391caabf49570
Lexmount/LexBench-WebArena-Replay ede6b17c21cef8a496b61e81c365934db5443919
Lexmount/LexBench-Headless-Browser 663248551204393cf38abbb18a9c5ddfa417118d
Lexmount/LexBench-Zyte-Product-Extraction 1b213908dfb737cec3cc1fe4e895b83819d23539
Lexmount/LexBench-Expanded-SWDE dbe94ac7ec2f80fdba44ae76a2b8a20b020c1e02
Lexmount/LexBench-NewsXLM efe0c74babc5fbf7d169ac31d4979d2337675d7d
Lexmount/LexBench-LiveWeb-IE 045b58d24a228b8f4f0089e89670a57c427a455f

Historical measurements keep their original input and scorer identities. See the benchmark comparison contract.

Not included: main/full content scopes (#268) are still in review and will need 0.6.0.

Validation

The release commit passed the main-branch CI workflow, including tests, installation/packaging and rendered report checks. The attached wheel passed an isolated installation outside the checkout and reports version 0.5.0 with its released scorer.

Changes since v0.4.0

Assets 5
Loading