Skip to content

Webstone 0.7.0

Latest
Latest

Choose a tag to compare

View all tags
@lanyue-llk lanyue-llk released this 12 Nov 15:41
· 5 commits to main since this release
v0.7.0
8d7c482
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Webstone: the touchstone for your agent's web tools.

Webstone 0.7.0 moves Web Fetch to the 530-task benchmark edition (LexBench-Web-Fetch v3.0.0). It also scores main content and full page as two separate views, both from the same saved calls.

Breaking change: Web Fetch edition v3.0.0

0.5.0 / 0.6.0 0.7.0
Web Fetch dataset v2.2.0 (c12559d2) v3.0.0 (def0249d)
Counted tasks 550 530 = the original 430 + 100 articles
10 mixed-suite dynamic examples counted not counted: they share IDs and URLs with dynamic-content tasks
10 challenge examples counted published separately, not counted

Compare v3.0.0 results separately from v2.x.

  • The five existing suites' task files and per-task rules are unchanged, so a task scores the same way. Edition totals and pass rates change because the counted population changed.
  • To rescore saved answers on v3.0.0, select the 530-task population (definitions/populations/primary-content-20261103.json). No new provider calls are needed.

New

  • Main and full content scopes (#268, #276).
    • The 530 tasks carry paired main and full acceptance profiles (content-scopes-530).
    • A run without --score-scope scores both views from the same saved responses, with no extra provider calls. --score-scope main|full computes only one.
    • --content-scope selects the scope used for acquisition, retries and the report.
  • The dataset card generated by Webstone now matches the v3.0.0 card and viewer (#280).
  • Owner review rules are written into docs/references/review-guidelines.md (#277).

Scorer

The Web Fetch scorer changed for the content scopes. The new scoring-source hash 0d1eee6f…032b is recorded in RELEASED_SCORERS, and lexbench-webfetch version reports 0.7.0, status: released.

Install

uv pip install https://github.com/lexmount/webstone/releases/download/v0.7.0/webstone-0.7.0-py3-none-any.whl

The wheel and source archive both declare 0.7.0; SHA256SUMS lists their checksums. Installed outside the checkout, the wheel reports 0.7.0, status: released.

Dataset pins

Dataset HF commit
Lexmount/LexBench-Web-Fetch (v3.0.0) def0249d88e849ab22d6108c0e0341fd54c7d38f
Lexmount/LexBench-Web-Search 76d9c2b685cc6da1f2a2aaceb7daa224e67155da

The other dataset pins are unchanged from 0.6.0 (see the release data lock).

Validation

The release commit 8d7c482f passed the main-branch CI workflow: tests, installation and packaging, and rendered report checks. The attached wheel was installed in isolation outside the checkout and reports 0.7.0 with its released scorer.

Changes since v0.6.0: #268, #272, #276, #277, #279, #280, #282.

Assets 5
Loading