Compare scores and traces on verifiable reasoning tasks. Test tools, skills and memory. More experiments and analysis of agent behavior, components and resource use can be found in Paper.
AlphaDiana runs agents on benchmarks, scores their answers and saves execution evidence for later analysis.
Positioning | Motivation | Scope | Quick Start | Docs | Citation and acknowledgement | Related Project
A harness controls how a model uses prompts, tools and persistent state.
Changing the harness can change model behavior. AlphaDiana links scores to execution evidence.
A model, its harness and its execution settings jointly shape agent behavior.
- Compare agents: compare scores and resource use across model–harness combinations on the same tasks.
- Inspect failures: trace wrong answers through execution status, tool interactions and saved action patterns.
- Test components: change tools, skills or memory and compare outcomes under recorded tasks and budgets.
Execution produces traces and artifacts. The paper examples show benchmark scores, failure-label incidence, output-token use and tool-removal effects. Each plot identifies the model, benchmark and metric.
Follow setup, execution, measurement and extension in the table; links lead to configuration and runtime guides.
| Layer | Dimension | Options |
|---|---|---|
| CONFIGURE | Models | Qwen3.5-27B, Gemma-4-31B-IT, Kimi-K2.6 Configurable OpenAI-compatible endpoints |
| Harnesses | OpenClaw, OpenCode, ZeroClaw, Pi, DeepSeek Harness SWE-Agent (SWE-bench), Terminus2 (Harbor/TB2), DirectLLM baseline |
|
| Components | Tools: sets, filters, removal Skills: math/general libraries Memory: within-task, cross-sample, cross-task Prompts/context: instructions, native compaction |
|
| Containers | None, Docker, Podman Host/process, ROCK-managed or benchmark-specific environments |
|
| RUN | Benchmarks | AIME 2026, GPQA-Diamond, HLE, IMO-AnswerBench MMMU-Pro, SWE-bench Verified, Terminal-Bench 2, Custom |
| Controls | Task selection, Repeats, Sampling, Token/time/tool budgets Concurrency, Retries, Resume, State isolation |
|
| Interfaces | CLI/YAML and Python API: compute, analyze and export | |
| MEASURE | Measurements | Outcomes: accuracy, Pass@k, Avg@k, status, failure labels |
| Resources: output tokens, wall time, cost per correct answer | ||
| Process: action composition/transitions, token/post-tool entropy | ||
| Components: score and behavior comparisons | ||
| Scoring | Exact/numeric match, Math equivalence, Model judge IMO, SWE-bench and Terminal-Bench verifiers |
|
| Evidence & exports | JSONL, Traces, Tool calls, Usage, Available logprobs/artifacts Markdown, Tables, PDF/SVG/PNG, Source hashes & coverage |
|
| EXTEND | Plugins | Harnesses, Benchmarks, Environments, Scorers, Metrics Saved-run analysis plugins |
Runtime combinations & measurement requirements
- Select supported combinations in the API acceptance matrix and harness guides.
- None means a direct API call or a supported host/process runtime.
- Entropy uses captured probabilities; post-tool curves require tool-feedback attribution.
- Action/failure labels use an explicitly enabled analysis model, which can differ from the execution model.
- Read Pass@k / Avg@k with the repeat count; match scorers, isolation and budgets when comparing runs.
The examples use Qwen3.5-27B as the model and OpenCode 1.3.2 as the harness. Both the CLI and Python API run 3 AIME 2026 tasks, with one sample per task, then analyze the saved results.
You need uv, Python ≥ 3.10, Podman, and an OpenAI-compatible model endpoint.
Create an isolated Python environment and install the execution and analysis dependencies:
git clone --depth 1 https://github.com/tmlr-group/AlphaDiana.git
cd AlphaDiana
uv venv --python 3.10 .venv
source .venv/bin/activate
uv pip install -e '.[analysis,benchmarks,agents]'Replace the URL and key with your provider's values. The model name must match an ID served by that endpoint.
export OPENAI_BASE_URL="https://your-provider.example/v1"
export OPENAI_API_KEY="your-api-key"
export OPENAI_MODEL_NAME="Qwen/Qwen3.5-27B"Run from the repository root with Podman installed:
podman build \
-f alphadiana/harness/opencode/deploy/Containerfile.podman-controller \
-t localhost/alphadiana-opencode-podman:latest \
alphadiana/harness/opencode/deployBoth workflows use this image, which includes OpenCode 1.3.2 and task tools. Podman setup guide
Use agent, data, scorer and optional env with either interface. compute runs tasks; analyzer reads saved artifacts. Exports include tables, PDF/SVG/PNG figures, plotted data, source hashes and evidence coverage.
The CLI uses configs/examples/aime2026_opencode.yaml. It matches the Python example's agent, data and scorer, with the endpoint from step 2 and image from step 3.
Validate the config, compute the scored artifacts, then analyze and export them:
alphadiana validate configs/examples/aime2026_opencode.yaml
alphadiana compute configs/examples/aime2026_opencode.yaml
alphadiana analyzer results/aime_opencode_cli.jsonl --output-dir figures/aime_opencode_cliThe run saves scored attempts to results/aime_opencode_cli.jsonl.
The analyzer reads those artifacts and exports measurements to figures/aime_opencode_cli/.
Check the score, score_status and evidence coverage when reading the results.
Troubleshooting →
Compose an agent, dataset and scorer, run the experiment, then analyze the saved results.
Component composition and analysis workflow. Use the example below.
Run python from the repository root in the same terminal, then execute:
import alphadiana as ad
agent = ad.agent(
"opencode",
version="1.3.2",
controller_mode="podman",
controller_image="localhost/alphadiana-opencode-podman:latest",
)
data = ad.dataset("MathArena/aime_2026", benchmark="aime", max_tasks=3)
scorer = ad.scorer.create("numeric")
runner = ad.runner(
run_id="aime_opencode_api",
num_samples=1,
task_retries=0,
strict_report=True,
)
artifacts = runner.compute(agent=agent, data=data, scorer=scorer)
ad.analyzer(artifacts).export("figures/aime_opencode_api")OpenCode manages its own Podman controller, configured through ad.agent(...).
compute() saves scored attempts in results/aime_opencode_api.jsonl and returns artifact references. The export in figures/aime_opencode_api/ contains the matching tables and figures.
Configuration and API details
Set run controls such as run_id, num_samples and task_retries at the YAML top level or on ad.runner(...). Use agent, data, scorer and optional env for components. Legacy run, benchmark and sandbox remain supported. Give each section one name: data or benchmark; env or sandbox.
- Use
ad.scorer.create(...), an explicit harnessversion=, andbenchmark=for hub datasets. - For supported harnesses, pass a separate
ad.environment(...)toad.runner(env=...).
Reopen either workflow's JSONL artifacts without rerunning the agent. Default analysis reads scores, usage counters and native traces. Entropy uses available probability evidence; coverage identifies missing measurements.
Reopen, compare and select export formats
Reopen either run and export SVG figures through the CLI or Python API:
alphadiana analyzer results/aime_opencode_cli.jsonl \
--output-dir figures/aime_opencode_cli_reopened --formats svgimport alphadiana as ad
ad.analyzer("results/aime_opencode_cli.jsonl").export(
"figures/aime_opencode_cli_reopened", formats=("svg",),
)Use a new export directory. Omit --output-dir to print JSON measurements, matching ad.analyzer(...).compute(). Compare runs using multiple CLI paths or a Python list.
Add action and failure labels with an analysis model
Configure an analysis model that can return structured JSON annotations:
export ANALYSIS_BASE_URL="https://your-provider.example/v1"
export ANALYSIS_API_KEY="your-api-key"
export ANALYSIS_MODEL="your-analysis-model-id"Annotate either saved run with --judge in CLI or judge=True in Python:
alphadiana analyzer results/aime_opencode_cli.jsonl \
--judge --output-dir figures/aime_opencode_cli_annotatedimport alphadiana as ad
ad.analyzer(
"results/aime_opencode_cli.jsonl", judge=True,
).export("figures/aime_opencode_cli_annotated")Use a new export directory. See annotation requirements and caching.
AlphaDiana connects experiment setup, execution and analysis through four stages:
- Configure: select tasks, a model and harness, execution environments, a scorer and budgets.
- Execute & score: the runner manages sessions; the harness runs the agent; the scorer evaluates answers.
- Save evidence: retain scores, execution status, outputs, traces and resource measurements in result records.
- Analyze & export: reopen those records to compare runs, inspect behavior and produce tables and figures.
AlphaDiana system architecture
The runner manages sessions and harnesses; saved evidence connects execution, scoring and analysis.
| I want to… | Start here |
|---|---|
| Run an experiment | Getting Started, Configuration, Presets |
| Choose a harness or benchmark | Harnesses, Benchmarks |
| Analyze saved runs | Python API, Analysis modules |
| Extend the system | Architecture, Analysis modules, Scripts |
CLI reference
| Command | Description |
|---|---|
alphadiana env |
Check configured service health |
alphadiana compute <config.yaml> |
Compute scored artifacts (run remains an alias) |
alphadiana analyzer <results.jsonl>... |
Compute measurements as JSON; add --output-dir to export figures |
alphadiana validate <config.yaml> |
Validate a config locally |
alphadiana report <results_dir> |
Print Markdown reports from saved result JSONL files |
alphadiana batch <config1> <config2> ... |
Run multiple experiments |
alphadiana list-benchmarks |
List the benchmark registry used by the Runner |
Use -o key=value to override configuration values, for example -o max_concurrent=4.
Repository map
alphadiana/
cli/ # command-line entry points
engine/ # execution, configuration, and rollout campaigns
harness/ # agent implementations and runtime skills
benchmarks/ # task loaders and official verifier integrations
scorer/ # answer evaluation and scorer interfaces
results/ # persistence, traces, score status, and sample statistics
analysis/ # measurement plugins, Analyzer composition, and exports
dashboard/ # React frontend and FastAPI backend
scripts/
setup/ # environment and dependency setup
validation/ # preflight, smoke tests, and evaluator audits
tests/ # grouped by analysis, API, engine, harness, and benchmark
configs/ # experiment configurations
docs/ # guides, contracts, and documentation assets
If you find the project helpful, please cite:
@article{zhou2026alphadiana,
title = {{AlphaDiana}: A Measurement System for Agents in Verifiable Reasoning Tasks},
author={Zhanke Zhou and Zongze Li and Weikai Huang and Xuan Li and Chentao Cao and Xiao Feng and Xiangyu Lu and Jinbo Hu and Yi Xie and Nico Pelleriti and Shiyang Liu and Menghan Lu and Max Zimmer and Brando Miranda and Jiangchao Yao and Bo Liu and Sanmi Koyejo and Sebastian Pokutta and Bo Han},
journal = {arXiv preprint arXiv:2610.20675},
year = {2026}
}We thank contributors, collaborators and open-source communities behind OpenClaw, ROCK and other tools.
Please also check out AlphaApollo, our system for deep agentic reasoning (arXiv, GitHub).




