v0.5.0

Browser Eval 0.5.0 is the first release whose version also names the Web Fetch benchmark edition: the pinned LexBench-Web-Fetch dataset plus this release's scoring code. It moves to dataset v2.2.0, which has corrected references and the same 550 tasks, includes the scorer fixes made since 0.4.0, and adds the Web Extract benchmark track.

Why a minor release

Web Fetch scores can change while the tasks stay the same. Rescore saved 0.4.0 answers with 0.5.0 before comparing them with new runs. The task population is unchanged, so rescoring the saved answers is enough; no new calls are needed.

Changes since 0.4.0

• One version for the Web Fetch benchmark (#264).
  • lexbench-webfetch version prints the Browser Eval version and its scoring-source hash. It prints X.Y.Z+unreleased when the code differs from the release.
  • docs/releases.md says when to bump the patch, minor or major version, including scoring changes.
• Corrected dataset (#266, #271). Web Fetch now pins LexBench-Web-Fetch v2.2.0. It adds the reviewed primary-content task scope revision; task hashes are unchanged.
• Scorer fixes:
  • six scorer defects repaired, each with paired controls (#239);
  • declared image links and catalog cards with a heading price (#250);
  • saved evidence that carries retired metrics (#251);
  • structured page fields (#252);
  • spaced anchors and indented links (#253);
  • escaped punctuation in anchors (#255);
  • Lexmount answer handling (#261, #263);
  • revision references that can't be executed are now reported as unavailable (#262).
• Lexmount evidence (#224). The native Lexmount API's execution evidence is retained, and category results are verified as uniform (#131).
• Web Extract track: versioned field-extraction benchmarks, self-hosted engines and local models, and source-qualified URL benchmarks with independent reports (#233, #258, #259, #260).
• Reports and repository:
  • English is now the source language of the report renderers (#240, #243, #244, #245).
  • Release-check and CI latency improved (#257).
  • Docs and tests were cleaned up (#265, #267, #269, #270).

Install

git clone --branch v0.5.0 https://github.com/lexmount/browser-eval.git
cd browser-eval
uv sync --frozen
uv run --frozen lexbench-webfetch version

The attached wheel and source archive both declare version 0.5.0. SHA256SUMS lists their checksums. Installed outside the checkout, the wheel reports 0.5.0, status: released, with scoring-source hash d0c5f12a609f622e8d97050b7d35581cd41973cd7c6a454d5267241ec2161e2b.

Benchmark input pins

The source is commit f3c30da9b25aeed866b6bc036999adb470af2d1a. Input selections and their file hashes are recorded in the release data lock.

Dataset	HF commit
Lexmount/LexBench-Web-Fetch (v2.2.0)	c12559d2af7db8ae6e84562ef187a5309811122e
Lexmount/LexBench-Web-Search	76d9c2b685cc6da1f2a2aaceb7daa224e67155da
Lexmount/LexBench-SimpleQA-Verified	550ef09fa823ac4d9b3859ddefed3542b5b7e65c
Lexmount/LexBench-WebMainBench	180b3cc6177df75ac9107ea8c76e3506ee96c5c9
Lexmount/LexBench-WCXB	d1381fdd95505c48bc17911479a8182c0d442fb0
Lexmount/LexBench-Trajectory-Replay	27e3b11926a8c18b3b29ba83a2e391caabf49570
Lexmount/LexBench-WebArena-Replay	ede6b17c21cef8a496b61e81c365934db5443919
Lexmount/LexBench-Headless-Browser	663248551204393cf38abbb18a9c5ddfa417118d
Lexmount/LexBench-Zyte-Product-Extraction	1b213908dfb737cec3cc1fe4e895b83819d23539
Lexmount/LexBench-Expanded-SWDE	dbe94ac7ec2f80fdba44ae76a2b8a20b020c1e02
Lexmount/LexBench-NewsXLM	efe0c74babc5fbf7d169ac31d4979d2337675d7d
Lexmount/LexBench-LiveWeb-IE	045b58d24a228b8f4f0089e89670a57c427a455f

Historical measurements keep their original input and scorer identities. See the benchmark comparison contract.

Not included: main/full content scopes (#268) are still in review and will need 0.6.0.

Validation

The release commit passed the main-branch CI workflow, including tests, installation/packaging and rendered report checks. The attached wheel passed an isolated installation outside the checkout and reports version 0.5.0 with its released scorer.

Changes since v0.4.0

v0.4.0

Browser Eval 0.4.0 moves Web Fetch and Web Search scoring into this repository and distributes public benchmark inputs through Hugging Face. Install the Browser Eval package to use lexbench-webfetch and lexbench-websearch; the retired standalone repositories are no longer installation dependencies.

Changes and migration

• Public datasets use the Lexmount/LexBench-* names on Hugging Face. Each selection binds a full commit and verified input hashes. Updating a dataset does not automatically change existing runs or Browser Eval's selected inputs.
• Dataset maintainers submit changes with python -m harness.benchmarks.publication --create-pr HF_REPOSITORY and the required build arguments. This replaces the direct --publish action. Merge the reviewed dataset PR before selecting its commit in Browser Eval.
• Evaluation results stay local. HTML reports can be generated, inspected and rebuilt from saved evidence; sharing through Artifact Site requires authorization. LLM verification remains optional.
• The English and Chinese READMEs distinguish datasets, evaluation code and upstream sources, and browsers/tools.

Web Fetch, Web Search and Browser Trajectory Replay are available through their track guides. Autonomous Browser Use remains experimental. Real providers require their credentials; WebArena replay also requires its self-hosted sites. Full third-party references remain subject to their original terms and acquisition requirements.

Install

git clone --branch v0.4.0 https://github.com/lexmount/browser-eval.git
cd browser-eval
uv sync --frozen
uv run --frozen browser-harness-eval validate

The attached wheel and source archive both declare version 0.4.0. SHA256SUMS lists their checksums.

Benchmark input pins

Browser Eval source is bound to software commit b220d07f57f5ce1fe6fb21db2b433105837fb85e. Input selections and their file hashes are recorded in the release data lock.

Dataset	HF commit
Lexmount/LexBench-Web-Fetch	3d8aa5cf76711a5f1609d66addc8467a9b10224e
Lexmount/LexBench-Web-Search	76d9c2b685cc6da1f2a2aaceb7daa224e67155da
Lexmount/LexBench-SimpleQA-Verified	550ef09fa823ac4d9b3859ddefed3542b5b7e65c
Lexmount/LexBench-WebMainBench	180b3cc6177df75ac9107ea8c76e3506ee96c5c9
Lexmount/LexBench-WCXB	d1381fdd95505c48bc17911479a8182c0d442fb0
Lexmount/LexBench-Trajectory-Replay	27e3b11926a8c18b3b29ba83a2e391caabf49570
Lexmount/LexBench-WebArena-Replay	fc32333defbb032da76d7921623e4bd503c48f03
Lexmount/LexBench-Headless-Browser	663248551204393cf38abbb18a9c5ddfa417118d

Historical measurements retain their original input and scorer identities. Comparisons across changed scoring rules require common-version rescoring; changed task populations may require separate results or new calls. See the benchmark comparison contract.

Validation

The release commit passed the main-branch CI workflow, including tests, installation/packaging and browser checks. The attached wheel also passed a clean offline installation, runtime-version checks and both benchmark CLI entry points.

Changes since v0.3.0

Browser Eval 0.3.0

Why this release

0.3.0 adds the merged Web Search/SERP workflow to Browser Eval and includes the shared Web Fetch capture and source-validation fixes. It is a minor software release because it adds a track and compatible capabilities to the 0.x interface.

Changes since 0.2.0

• Add reusable Search/SERP collection, scoring and reports (#58).
• Keep targeted evidence refreshes within their declared scope (#57).
• Share source validation and bounded browser capture (#61).
• Align package, lock and runtime versions; verify release-tag identity and keep project citations version-neutral (#68).

Included benchmark versions

Component	Version	Pinned source
Browser Eval	0.3.0	f5c56767c54b65f86fc2397c4be7deb91e1705d0
LexBench Web Fetch	benchmark 2.0.1	d0e78e71cf5145217cb1f2b71420a7c4194d80dd
LexBench Web Search	benchmark 1.0.0	d312653cbdc0396ffefa49ced37f82f7ca764da2

Fetch's catalog contains 550 entries. Search selects scenarios-v2 (149 cases) and factual-v1 (831 cases); the earlier scenarios-v1 snapshot remains available. A report's own manifest selects its actual population. WebMainBench, WCXB, Browser Use and Trajectory Replay retain their separate dataset and environment contracts.

The pending Fetch rule revision (#12 in lexbench-webfetch) and Search edition 1.1.0 are not included. Installing this software does not change historical report scores or adopt newer upstream benchmarks.

Validation and installation

The release commit has the same source tree as the tested PR: make ci passed 7,028 tests and 2,972 subtests, fixture workflows and four isolated installation configurations. Optional external-input/environment tests account for 129 skips. Release metadata and matching-tag checks pass on the merged commit. The attached wheel and source archive were built from that commit.

uv pip install 'git+https://github.com/lexmount/browser-eval.git@v0.3.0'

Python 3.11 or later is required. Additional tracks retain their documented optional dependencies. Existing report and benchmark bindings remain unchanged. Compare software changes.

Search catalog binding

The pinned Search 1.0.0 repository revision contains scorer changes beyond the scoring source recorded in its frozen catalog. Its strict catalog verification fails, even though the installed source remains identified by the immutable dependency pin. For the original measurement contract, combine the 1.0.0 catalog with its recorded scorer commit, as documented in that benchmark release.

Browser Eval PR #69 adopts the verified Search 1.1.0 dependency for subsequent work. Dependency synchronization does not itself schedule another software release. Compare Search scores only after rescoring both cohorts under the same benchmark edition. This 0.3.0 tag and its original dependencies remain unchanged.

Browser Eval 0.2.0

Why this release

0.2.0 gives report builds a shared software identity and provides reusable Web Fetch reporting with an independently pinned LexBench evaluator.

Included state

• Software: Browser Eval 0.2.0, commit 30645af1ba2520b4d089cbef32979e088cffe277.
• Web Fetch dependency: benchmark 2.0.1, commit d0e78e71cf5145217cb1f2b71420a7c4194d80dd.
• Web Fetch, Browser Trajectory Replay and Browser Use have their own runners and report contracts. This release predates the merged Web Search integration.
• The 550-entry Fetch catalog does not determine a report's selected population; each report retains its own inputs and scorer bindings.

Compared with 0.1.0, this release includes the extracted benchmark dependency, shared report-build provenance and Apache-2.0 project licensing. Compare source changes.

The original annotated tag has been restored to its original commit. No code or historical measurements were replaced.

Browser Eval 0.1.0

Why this release

0.1.0 preserves the original installable Browser Eval software checkpoint for reproducing earlier work.

Included state

• Software commit: 03c9a5e94619000480c92b8e517c16165dc2a7f4.
• Workflows: Web Fetch, fixed Browser Trajectory Replay and Agent Browser Use, plus an offline success/failure example.
• Web Fetch scoring implementations live inside this Browser Eval revision; no separate LexBench Fetch or Search package is declared.
• The default report profile in this revision selects a 200-task WebMainBench sample. Other report populations are defined by their own profiles; the software version does not select one universal benchmark.

This is a historical 0.x interface, before the extracted benchmark dependency and later Search integration. Use the tag's own installation instructions and dependency lock. The original annotated tag has been restored without changing its source.