Skip to content

Webstone 0.4.0

Choose a tag to compare

View all tags
@lanyue-llk lanyue-llk released this 30 Oct 12:10
· 103 commits to main since this release
v0.4.0
b220d07
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Browser Eval 0.4.0 moves Web Fetch and Web Search scoring into this repository and distributes public benchmark inputs through Hugging Face. Install the Browser Eval package to use lexbench-webfetch and lexbench-websearch; the retired standalone repositories are no longer installation dependencies.

Changes and migration

  • Public datasets use the Lexmount/LexBench-* names on Hugging Face. Each selection binds a full commit and verified input hashes. Updating a dataset does not automatically change existing runs or Browser Eval's selected inputs.
  • Dataset maintainers submit changes with python -m harness.benchmarks.publication --create-pr HF_REPOSITORY and the required build arguments. This replaces the direct --publish action. Merge the reviewed dataset PR before selecting its commit in Browser Eval.
  • Evaluation results stay local. HTML reports can be generated, inspected and rebuilt from saved evidence; sharing through Artifact Site requires authorization. LLM verification remains optional.
  • The English and Chinese READMEs distinguish datasets, evaluation code and upstream sources, and browsers/tools.

Web Fetch, Web Search and Browser Trajectory Replay are available through their track guides. Autonomous Browser Use remains experimental. Real providers require their credentials; WebArena replay also requires its self-hosted sites. Full third-party references remain subject to their original terms and acquisition requirements.

Install

git clone --branch v0.4.0 https://github.com/lexmount/browser-eval.git
cd browser-eval
uv sync --frozen
uv run --frozen browser-harness-eval validate

The attached wheel and source archive both declare version 0.4.0. SHA256SUMS lists their checksums.

Benchmark input pins

Browser Eval source is bound to software commit b220d07f57f5ce1fe6fb21db2b433105837fb85e. Input selections and their file hashes are recorded in the release data lock.

Dataset HF commit
Lexmount/LexBench-Web-Fetch 3d8aa5cf76711a5f1609d66addc8467a9b10224e
Lexmount/LexBench-Web-Search 76d9c2b685cc6da1f2a2aaceb7daa224e67155da
Lexmount/LexBench-SimpleQA-Verified 550ef09fa823ac4d9b3859ddefed3542b5b7e65c
Lexmount/LexBench-WebMainBench 180b3cc6177df75ac9107ea8c76e3506ee96c5c9
Lexmount/LexBench-WCXB d1381fdd95505c48bc17911479a8182c0d442fb0
Lexmount/LexBench-Trajectory-Replay 27e3b11926a8c18b3b29ba83a2e391caabf49570
Lexmount/LexBench-WebArena-Replay fc32333defbb032da76d7921623e4bd503c48f03
Lexmount/LexBench-Headless-Browser 663248551204393cf38abbb18a9c5ddfa417118d

Historical measurements retain their original input and scorer identities. Comparisons across changed scoring rules require common-version rescoring; changed task populations may require separate results or new calls. See the benchmark comparison contract.

Validation

The release commit passed the main-branch CI workflow, including tests, installation/packaging and browser checks. The attached wheel also passed a clean offline installation, runtime-version checks and both benchmark CLI entry points.

Changes since v0.3.0

Assets 5
Loading