Skip to content

BenchTrend v0.2.0

Choose a tag to compare

View all tags
@hyunyoungnam hyunyoungnam released this 07 Oct 02:48
· 12 commits to main since this release
v0.2.0
ef26172

First code release of BenchTrend: which benchmarks the field evaluates on, how that changes across conference editions, and which newly introduced benchmarks other authors adopt — counted from the papers' own sentences. Research prototype.

What is in it

  • benchtrend — a terminal conversation over the OpenAI or Anthropic API (your key, hidden, never written to disk) with saved sessions, /sources, and quote and figure verification before an answer is shown.
  • benchtrend mcp — a stdio MCP server with six read-only tools (benchmark_scope, benchmark_usage, benchmark_trend, new_benchmarks, benchmark_adoption, benchmark_evidence); benchtrend mcp --connect claude / --connect codex registers it with the client's own CLI, so that route needs no API key.
  • benchtrend data install — installs a released data snapshot from a file or an HTTPS URL, verifying the bundle manifest and an optional SHA-256, atomically.
  • benchtrend status — data, API keys, and whether Claude Code / Codex have the server registered and connecting.
  • The bellwether command for the earlier browser interface and build-machine exports.

Python 3.10+ (tested on 3.10 and 3.14), standard library only; Windows works natively for the terminal and MCP routes.

Install

uv tool install https://github.com/hyunyoungnam/BenchTrend/archive/refs/tags/v0.2.0.tar.gz
# or, from the wheel attached below:
uv tool install ./benchtrend-0.2.0-py3-none-any.whl

Then the data (tested with this release):

benchtrend data install --url https://github.com/hyunyoungnam/BenchTrend/releases/download/data-20261007/benchtrend-data.tar.gz \
  --sha256 d19dcc2c47cc13738188176b5b87534e7e3eb0f0d11c86731bf41e0a1acca50e

Data release: data-20261007 — snapshot 0dd4b9bbee4b54d2, 28 editions of 10 venues, 58,616 parsed papers, 9,332 benchmarks and datasets. Code and data are versioned separately; see docs/releases.md.

What it does not say

Only stated use is counted ("we evaluate on …"), over papers with parsed arXiv full text — the denominator is printed beside every figure. "New" means first claimed inside this corpus, which starts in 2023. Frequency is never quality. Full guide: docs/terminal.md.

Checksums

SHA256SUMS is attached; verify with sha256sum -c SHA256SUMS.

Assets 5
Loading