Skip to content

About

AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks

Topics

Resources

Stars

25 stars

Watchers

0 watching

Forks

Latest commit

 

History

1,055 Commits

Folders and files

Repository files navigation

AlphaDiana
AlphaDiana: A Measurement System for Agents
in Verifiable Reasoning Tasks

Compare scores and traces on verifiable reasoning tasks. Test tools, skills and memory. More experiments and analysis of agent behavior, components and resource use can be found in Paper.

AlphaDiana runs agents on benchmarks, scores their answers and saves execution evidence for later analysis.

Positioning | Motivation | Scope | Quick Start | Docs | Citation and acknowledgement | Related Project

Positioning

A harness controls how a model uses prompts, tools and persistent state.

Changing the harness can change model behavior. AlphaDiana links scores to execution evidence.

Position of AlphaDiana

A model, its harness and its execution settings jointly shape agent behavior.

Why AlphaDiana?

  • Compare agents: compare scores and resource use across model–harness combinations on the same tasks.
  • Inspect failures: trace wrong answers through execution status, tool interactions and saved action patterns.
  • Test components: change tools, skills or memory and compare outcomes under recorded tasks and budgets.

AlphaDiana measurement examples: benchmark scores, failure modes, output-token usage, and component ablations.

Execution produces traces and artifacts. The paper examples show benchmark scores, failure-label incidence, output-token use and tool-removal effects. Each plot identifies the model, benchmark and metric.

Scope at a glance

Follow setup, execution, measurement and extension in the table; links lead to configuration and runtime guides.

Layer Dimension Options
CONFIGURE Models Qwen3.5-27B, Gemma-4-31B-IT, Kimi-K2.6
Configurable OpenAI-compatible endpoints
Harnesses OpenClaw, OpenCode, ZeroClaw, Pi, DeepSeek Harness
SWE-Agent (SWE-bench), Terminus2 (Harbor/TB2), DirectLLM baseline
Components Tools: sets, filters, removal
Skills: math/general libraries
Memory: within-task, cross-sample, cross-task
Prompts/context: instructions, native compaction
Containers None, Docker, Podman
Host/process, ROCK-managed or benchmark-specific environments
RUN Benchmarks AIME 2026, GPQA-Diamond, HLE, IMO-AnswerBench
MMMU-Pro, SWE-bench Verified, Terminal-Bench 2, Custom
Controls Task selection, Repeats, Sampling, Token/time/tool budgets
Concurrency, Retries, Resume, State isolation
Interfaces CLI/YAML and Python API: compute, analyze and export
MEASURE Measurements Outcomes: accuracy, Pass@k, Avg@k, status, failure labels
Resources: output tokens, wall time, cost per correct answer
Process: action composition/transitions, token/post-tool entropy
Components: score and behavior comparisons
Scoring Exact/numeric match, Math equivalence, Model judge
IMO, SWE-bench and Terminal-Bench verifiers
Evidence & exports JSONL, Traces, Tool calls, Usage, Available logprobs/artifacts
Markdown, Tables, PDF/SVG/PNG, Source hashes & coverage
EXTEND Plugins Harnesses, Benchmarks, Environments, Scorers, Metrics
Saved-run analysis plugins
Runtime combinations & measurement requirements
  • Select supported combinations in the API acceptance matrix and harness guides.
  • None means a direct API call or a supported host/process runtime.
  • Entropy uses captured probabilities; post-tool curves require tool-feedback attribution.
  • Action/failure labels use an explicitly enabled analysis model, which can differ from the execution model.
  • Read Pass@k / Avg@k with the repeat count; match scorers, isolation and budgets when comparing runs.

Quick Start

The examples use Qwen3.5-27B as the model and OpenCode 1.3.2 as the harness. Both the CLI and Python API run 3 AIME 2026 tasks, with one sample per task, then analyze the saved results.

You need uv, Python ≥ 3.10, Podman, and an OpenAI-compatible model endpoint.

1. Install

Create an isolated Python environment and install the execution and analysis dependencies:

git clone --depth 1 https://github.com/tmlr-group/AlphaDiana.git
cd AlphaDiana
uv venv --python 3.10 .venv
source .venv/bin/activate
uv pip install -e '.[analysis,benchmarks,agents]'

2. Set up the model endpoint

Replace the URL and key with your provider's values. The model name must match an ID served by that endpoint.

export OPENAI_BASE_URL="https://your-provider.example/v1"
export OPENAI_API_KEY="your-api-key"
export OPENAI_MODEL_NAME="Qwen/Qwen3.5-27B"

3. Build the OpenCode controller image

Run from the repository root with Podman installed:

podman build \
  -f alphadiana/harness/opencode/deploy/Containerfile.podman-controller \
  -t localhost/alphadiana-opencode-podman:latest \
  alphadiana/harness/opencode/deploy

Both workflows use this image, which includes OpenCode 1.3.2 and task tools. Podman setup guide

4. Run and analyze with the CLI or Python API

Use agent, data, scorer and optional env with either interface. compute runs tasks; analyzer reads saved artifacts. Exports include tables, PDF/SVG/PNG figures, plotted data, source hashes and evidence coverage.

CLI

The CLI uses configs/examples/aime2026_opencode.yaml. It matches the Python example's agent, data and scorer, with the endpoint from step 2 and image from step 3.

Validate the config, compute the scored artifacts, then analyze and export them:

alphadiana validate configs/examples/aime2026_opencode.yaml
alphadiana compute configs/examples/aime2026_opencode.yaml
alphadiana analyzer results/aime_opencode_cli.jsonl --output-dir figures/aime_opencode_cli

The run saves scored attempts to results/aime_opencode_cli.jsonl. The analyzer reads those artifacts and exports measurements to figures/aime_opencode_cli/. Check the score, score_status and evidence coverage when reading the results. Troubleshooting →

Python API

Compose an agent, dataset and scorer, run the experiment, then analyze the saved results.

AlphaDiana functional interface: agent, dataset, environment, scorer, runner, and analysis composition.

Component composition and analysis workflow. Use the example below.

Run python from the repository root in the same terminal, then execute:

import alphadiana as ad

agent = ad.agent(
    "opencode",
    version="1.3.2",
    controller_mode="podman",
    controller_image="localhost/alphadiana-opencode-podman:latest",
)
data = ad.dataset("MathArena/aime_2026", benchmark="aime", max_tasks=3)
scorer = ad.scorer.create("numeric")
runner = ad.runner(
    run_id="aime_opencode_api",
    num_samples=1,
    task_retries=0,
    strict_report=True,
)

artifacts = runner.compute(agent=agent, data=data, scorer=scorer)

ad.analyzer(artifacts).export("figures/aime_opencode_api")

OpenCode manages its own Podman controller, configured through ad.agent(...).

compute() saves scored attempts in results/aime_opencode_api.jsonl and returns artifact references. The export in figures/aime_opencode_api/ contains the matching tables and figures.

Configuration and API details

Set run controls such as run_id, num_samples and task_retries at the YAML top level or on ad.runner(...). Use agent, data, scorer and optional env for components. Legacy run, benchmark and sandbox remain supported. Give each section one name: data or benchmark; env or sandbox.

  • Use ad.scorer.create(...), an explicit harness version=, and benchmark= for hub datasets.
  • For supported harnesses, pass a separate ad.environment(...) to ad.runner(env=...).

Full API guide: compose, reopen, rescore and extend →

More analysis options

Reopen either workflow's JSONL artifacts without rerunning the agent. Default analysis reads scores, usage counters and native traces. Entropy uses available probability evidence; coverage identifies missing measurements.

Reopen, compare and select export formats

Reopen either run and export SVG figures through the CLI or Python API:

alphadiana analyzer results/aime_opencode_cli.jsonl \
  --output-dir figures/aime_opencode_cli_reopened --formats svg
import alphadiana as ad

ad.analyzer("results/aime_opencode_cli.jsonl").export(
    "figures/aime_opencode_cli_reopened", formats=("svg",),
)

Use a new export directory. Omit --output-dir to print JSON measurements, matching ad.analyzer(...).compute(). Compare runs using multiple CLI paths or a Python list.

Add action and failure labels with an analysis model

Configure an analysis model that can return structured JSON annotations:

export ANALYSIS_BASE_URL="https://your-provider.example/v1"
export ANALYSIS_API_KEY="your-api-key"
export ANALYSIS_MODEL="your-analysis-model-id"

Annotate either saved run with --judge in CLI or judge=True in Python:

alphadiana analyzer results/aime_opencode_cli.jsonl \
  --judge --output-dir figures/aime_opencode_cli_annotated
import alphadiana as ad

ad.analyzer(
    "results/aime_opencode_cli.jsonl", judge=True,
).export("figures/aime_opencode_cli_annotated")

Use a new export directory. See annotation requirements and caching.

Documentation

AlphaDiana connects experiment setup, execution and analysis through four stages:

  1. Configure: select tasks, a model and harness, execution environments, a scorer and budgets.
  2. Execute & score: the runner manages sessions; the harness runs the agent; the scorer evaluates answers.
  3. Save evidence: retain scores, execution status, outputs, traces and resource measurements in result records.
  4. Analyze & export: reopen those records to compare runs, inspect behavior and produce tables and figures.

AlphaDiana system architecture

AlphaDiana system design: configuration, runner, session management, native agent runtimes, inference endpoints, and artifact analysis.

The runner manages sessions and harnesses; saved evidence connects execution, scoring and analysis.

Architecture →

I want to… Start here
Run an experiment Getting Started, Configuration, Presets
Choose a harness or benchmark Harnesses, Benchmarks
Analyze saved runs Python API, Analysis modules
Extend the system Architecture, Analysis modules, Scripts
CLI reference
Command Description
alphadiana env Check configured service health
alphadiana compute <config.yaml> Compute scored artifacts (run remains an alias)
alphadiana analyzer <results.jsonl>... Compute measurements as JSON; add --output-dir to export figures
alphadiana validate <config.yaml> Validate a config locally
alphadiana report <results_dir> Print Markdown reports from saved result JSONL files
alphadiana batch <config1> <config2> ... Run multiple experiments
alphadiana list-benchmarks List the benchmark registry used by the Runner

Use -o key=value to override configuration values, for example -o max_concurrent=4.

Repository map
alphadiana/
  cli/           # command-line entry points
  engine/        # execution, configuration, and rollout campaigns
  harness/       # agent implementations and runtime skills
  benchmarks/    # task loaders and official verifier integrations
  scorer/        # answer evaluation and scorer interfaces
  results/       # persistence, traces, score status, and sample statistics
  analysis/      # measurement plugins, Analyzer composition, and exports
  dashboard/     # React frontend and FastAPI backend
scripts/
  setup/         # environment and dependency setup
  validation/    # preflight, smoke tests, and evaluator audits
tests/           # grouped by analysis, API, engine, harness, and benchmark
configs/         # experiment configurations
docs/            # guides, contracts, and documentation assets

Citation and acknowledgement

If you find the project helpful, please cite:

@article{zhou2026alphadiana,
  title = {{AlphaDiana}: A Measurement System for Agents in Verifiable Reasoning Tasks},
  author={Zhanke Zhou and Zongze Li and Weikai Huang and Xuan Li and Chentao Cao and Xiao Feng and Xiangyu Lu and Jinbo Hu and Yi Xie and Nico Pelleriti and Shiyang Liu and Menghan Lu and Max Zimmer and Brando Miranda and Jiangchao Yao and Bo Liu and Sanmi Koyejo and Sebastian Pokutta and Bo Han},
  journal = {arXiv preprint arXiv:2610.20675},
  year = {2026}
}

We thank contributors, collaborators and open-source communities behind OpenClaw, ROCK and other tools.

Related Project

Please also check out AlphaApollo, our system for deep agentic reasoning (arXiv, GitHub).

About

AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks

Topics

Resources

Stars

25 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages