Repository navigation
Releases: lexmount/lexbench-websearch
Release list
Web Search benchmark 1.1.0
Why this release
Benchmark 1.1.0 corrects numeric-answer matching while preserving the task population and requests. It also restores a verifiable catalog-to-scorer binding: the historical 1.0.0 tag contains scoring changes beyond the source recorded in its frozen catalog.
Included state
| Component | Version |
|---|---|
| Benchmark | 1.1.0, tag bench-v1.1.0 |
| Release source | f07a2e2ddf1c1c511cb3944659534301c86ff9c9 (merged PR #4) |
| Task suites | scenarios: 149 cases; factual: 831 cases |
The release includes the merged numeric fixes from PR #3, preserving formatted quantities and rejecting partial numeric matches joined to units. PR #4 freezes the corrected scorer, retains previous release hashes and standardizes new tag names. One benchmark tag includes the tooling package; a separate package tag is unnecessary.
Comparison and historical reproduction
Rescore both saved cohorts under 1.1.0 before comparing scores. Requests are unchanged, so saved answers can be reused; new provider calls are unnecessary solely for this correction. Do not relabel earlier scores as 1.1.0.
The historical 1.0.0 catalog points to scoring commit 9c833f833ce748ecfd4abef7c65f00753798a799; its later repository tag source does not match that scorer hash. Preserve that tag and its history; use the catalog's source binding for the original contract. This release's strict catalog verification passes.
Validation
On the merged release commit, make check with the release-tag environment passes 119 tests and two subtests. The source archive, wheel and isolated installation pass 116 tests and two subtests, with three source-only checks skipped. Both current datasets validate. The attached wheel and source archive contain this release’s evaluation code.
Web Search benchmark 1.0.0
Why this release
1.0.0 freezes the first Web Search measurement catalog: task snapshots, answer requirements, provenance and scoring-source hashes. The tag fixes tasks, references, scoring rules and evaluation code together.
Included state
| Component | Version |
|---|---|
| Benchmark | 1.0.0; tag bench-v1.0.0 |
| Release commit | d312653cbdc0396ffefa49ced37f82f7ca764da2 |
| Current scenario suite | scenarios, 149 cases |
| Current factual suite | factual, 831 cases |
This single tag contains the catalog and its Python tools. No second tooling tag is needed. Benchmark release names consistently use bench-v....
Later numeric-answer corrections belong to a successor edition. Compare cohorts only after rescoring both with the same corrected edition. The catalog's individual dataset hashes identify the selected snapshot; case counts alone do not establish comparability.
Historical scorer binding
The catalog in this tag binds scoring commit 9c833f833ce748ecfd4abef7c65f00753798a799 and source hash 1dad63e3c4294ea044abfcd5d0bb3b76b84a35613967629c22b6ebc25cf9cc8a. The later code in the repository tag includes additional scoring changes and does not match that hash.
To reproduce the original contract, combine this tag's catalog with the recorded scorer commit and pass the catalog path explicitly to catalog_release(path). That combination has been tested and passes strict verification; the scorer commit's own earlier catalog is a different snapshot. Use edition 1.1.0 for new evaluations and rescore both cohorts before comparing. The historical tag and scores are preserved.