Abench v0.2.0

Abench v0.2.0

First public release of Abench — a closed-loop agentic inference benchmark.
It replays recorded agent trajectories (DeepSWE v1.1, LMCache) or a synthetic
workload against any OpenAI-compatible inference server (SGLang, vLLM, ...)
and reports per-turn TTFT / TPOT / E2E, cache behaviour, throughput, and
server-side KV pool metrics.

Install

PyPI (distribution name agentic-bench):

pip install agentic-bench

GitHub Release wheel:

pip install https://github.com/AlibabaResearch/abench/releases/download/v0.2.0/agentic_bench-0.2.0-py3-none-any.whl

From source:

pip install git+https://github.com/AlibabaResearch/abench.git

The PyPI distribution is named agentic-bench; the Python module and the
command remain abench / python3 -m abench.

Do not install this alongside the unrelated PyPI package abench
(ActivitySim) — both install an abench module and command.

Quick start

python3 -m abench data deepswe/mixed
python3 -m abench run --model default --dataset-name deepswe/mixed \
    --max-concurrency 16

Highlights

• Closed-loop replay of real agent sessions with recorded tool waits and
  decode lengths; output length is pinned for cross-engine comparability.
• Backend auto-detection for SGLang, vLLM, and generic OpenAI-compatible
  servers; unmeasurable metrics are reported as null, not 0.
• Per-turn TTFT / TPOT / E2E metrics, ITL percentiles, prefix-cache and KV
  pool observations, and steady-state audit data in JSONL output.
• Minimal runtime dependency (aiohttp); optional extras for zstd,
  tokenizers, and progress UI.
• The repository ships no trajectory data. python3 -m abench data downloads
  Dataset tiers from the public ModelScope mirror
  (Apsara-Stack/ABench), with SHA-256 verification and resumable transfer.

Requirements

Python 3.9+. See README.md and docs/README.md for the full CLI reference.

License

Apache-2.0. See LICENSE and NOTICE for upstream conventions and dataset
attribution.