EvalDock

An end-to-end evaluation framework for heterogeneous agents and plugins.

Website: https://www.evaldock.ai/

• Overview
• V2 directions
• Workbench
• Evaluation workflow
• Scoring
• Supported agents
• Dataset coverage
• Setup and datasets
• Results
• Development
• Adding an agent
• Contributing
• Community

EvalDock tests different agents and compares plugin configurations within the same agent. It selects tasks, runs them, records what happened, and scores the results in a local workbench.

Each agent has its own execution adapter. Datasets, the All Trace format, and the Judge are shared. Every case keeps its tool calls, output files, scores, and scoring explanations for review or later comparison.

The workbench lets you select agents and plugins, configure Planner and Judge API keys, choose tasks, and start or cancel runs.

V2 is being refined on our development server. The architecture of both modules is outlined below; reference source is included, but the modules are not yet enabled in the public version.

• Capability-aware planning: The Planner uses the agent's tools, plugins and capability profile to select a small set of representative tasks covering the target capabilities.
• Separate execution and estimation: The Runtime executes selected tasks and the Judge produces case scores, while an independent estimator infers performance across the target suite's dimensions.
• Calibration and reporting: Historical full-suite results calibrate the model, sampled evaluations correct prediction bias, and reports distinguish measured scores from estimates and their uncertainty.

• Unified evidence input: Probe / Observer collect execution traces, environment changes and deliverables, which All Trace archives as the evidence base for TAG.
• Event–object–state model: Calls and responses become events, files become objects, and observations and snapshots become states, connected by explicit relationships with references to the original evidence.
• Graph-based judging and diagnosis: The Judge examines actions and their linked evidence, attaches detailed assessments and diagnostic findings to specific nodes, and preserves each task's original rubric.

Use the workbench to configure agents, plugins and models and inspect your own runs. See the setup guide.

The diagram uses DSH as an example. The current code supports the eight agent types listed below and runs up to three cases concurrently. The serial execution and score aggregation shown in the diagram are from an earlier design.

1. Inspect and select. Read the agent version, tools, and plugins. The Planner selects datasets from this information, or you can choose cases manually.
2. Run. Prepare a separate workspace and give the agent the task and public inputs. Private grading references stay with the Judge.
3. Collect evidence. The Probe records tool calls and responses. File observation records workspace changes and deliverables. Both are stored in All Trace.
4. Score. The Judge uses the task references, label criteria, and All Trace to score each dimension and produce JSON and HTML reports.

Select cases manually or let the Planner choose 3–10 datasets, with 3–7 cases each (all cases if fewer than three). Automatic sampling targets 20% easy, 40% medium, and 40% hard cases. Selection rules.

The Probe collects evidence on events or task completion, merges duplicates, and limits trace size. Truncation and missing evidence are recorded. Evidence policy.

The current 15 capability labels use a 0–100 scale. Each label has five criteria. The Judge scores against those criteria using the task and recorded evidence.

A dimension with insufficient evidence is marked UNASSESSABLE. Judge request or parsing failures retain their error status. Neither contributes a valid score. A task with evidence of failure can still receive a low score. A score of 100 requires direct evidence that all applicable criteria were met; there is no quota for perfect scores.

EFFECT mode evaluates deliverables. FULL mode also uses execution evidence. Read aggregate scores alongside the assessed dimensions and task coverage. Historical reports retain their original criteria and are not converted to a new scale.

Same-case labels are scored in groups of four, sharing the full evidence once per request. Shared context precedes the rubrics to improve provider cache reuse. Reports record requests, token usage and cache hits without counting a shared request twice. Set EVALDOCK_JUDGE_BATCH_SIZE=1 for separate requests. Judge configuration.

Eight agents currently have adapters. Codex and Claude Code are planned additions.

Agents must be installed and configured separately. See the shared support and validation record for application identities, allowed versions, public validation status and limitations. An implemented adapter, a passing version check and a validated real task are distinct states. Revalidation is required after upgrades.

Logo	Agent	Integration	Required setup
	DSH	Web Profile and runtime Probe	Web Profile, model, plugins
	Pi	CLI JSON events	Executable, model, extensions
	Codex (planned)	Adapter pending	Setup guide will follow the integration
	Claude Code (planned)	Adapter pending	Setup guide will follow the integration
	OpenClaw	CLI and session transcript	Node runtime, executable path, model
	Hermes	CLI stream-json	Python environment, executable path, model
	WorkBuddy	Desktop sessions and events	App login, local debugging connection
	Qwen Work (千问办公)	Desktop sessions and events	App login, local workspace
	Doubao Work (豆包办公)	Desktop sessions and background tasks	App login, local office environment
	LangGraph	graph stream / astream	Python environment, graph entrypoint, tool definitions

Example LangGraph targets include DeepAgents and Agent Service Toolkit. Plugin selection is available for agents that support plugins or extensions.

Adapter notes: shared interface · OpenClaw · Hermes · Qwen / Doubao Work. Replace example installation paths with your own.

This development-catalog snapshot covers 144 entries and 1,658 cases; categories overlap and counts are not additive, differ from the public package below, and do not imply complete runtime validation.

BenchDock: 1,650 task records across 146 subsets. 1,150 include public content; 500 provide sources only. The framework's default datasets/ includes one synthetic demo task for workflow checks, not capability rankings. Public data · BenchDock tools and documentation · Pinned pilot import and separate scoring. External catalog counts must not be added to the bundled count or treated as loaded, executed or scored task counts; see the dataset guide. Public availability does not imply verified execution or official upstream scores.

Choose the source setup to run EvalDock today. Docker packaging is still in preparation.

After installation:

1. Install and configure the agent you want to test.
2. Open System Settings in the workbench and set the model, endpoint, and API key for the Planner and Judge. The agent uses its own credentials.
3. Prepare a dataset and its required tools or services, then start an evaluation from the agent console.

The Docker plan keeps the workbench and scoring services in a container, with a Runner on the agent's machine for execution and evidence collection. Desktop agents run on their native OS. Startup commands and volume paths will accompany the Compose release.

The basic template contains one task: read numbers from JSON, calculate their sum, and write the result to a file. Use it to check the setup and evaluation flow, or as a starting point for your own cases.

The default datasets/ uses this demo; without external imports, it is the only task available for selection. Follow the setup guide to configure the Agent, runtime dependencies and required model services.

Public task packages are available from BenchDock. Use the integration guide for an explicit three-task pilot. Workbench import automation and full-collection runtime coverage are not provided. Existing result artifacts remain local and may include sensitive evaluator material.

datasets/
├── catalog.md           # Dataset index
└── <dataset>/
    └── <case>/
        ├── question.json
        ├── input/       # Public inputs for the agent
        └── private/     # Grading references for the Judge

Prepare any tools, databases, or external services required by a dataset separately. Public tasks without authorized grading references remain UNASSESSABLE: no Judge call is made and no zero score is assigned. Scores on adapted tasks are not official results for the upstream benchmarks.

Open reports from the workbench to inspect scores, traces, output files, and Judge records.

Re-score an archived Case without running the Agent again: pnpm rescore --report /path/to/case/report.json. Each run creates a separate score revision; the original report stays unchanged. Usage.

For shared-adapter runs, plan.json records the selected queue; run.json, per-case sessions, reports and traces establish the actual execution evidence. See the evidence guide. Execution completion does not establish answer correctness or completed scoring.

The planned Docker release will persist results in a host-mounted directory. The mount path will be documented with the Compose release. The var/ paths below apply to the current source version.

var/
├── evaluation-results/  # Saved results
├── batch-runtime/       # Workspaces and intermediate data
├── workbench/           # Job state and caches
└── logs/                # Runtime logs

These commands are for building and testing the source. They are not Docker installation steps.

pnpm install --frozen-lockfile
pnpm run verify

pnpm --silent cli -- help

pnpm --silent cli -- run \
  --target config/targets/real-dsh.json \
  --datasets datasets \
  --labels labels \
  --trace trace/dsh-runtime.json \
  --environment environments/macos.json \
  --config config/macos-vm.json \
  --test-profile STANDARD

adapters/       # Agent adapters and shared execution code
src/            # Planning, Trace, Judge, reports, and storage
workbench/      # Workbench pages and control service
labels/         # Capability labels and scoring criteria
planning/       # Selection policies, prompts, and difficulty annotations
config/         # Agent and runtime configuration
trace/          # Probe configuration
observer-lab/   # Environment observation
environments/   # Task environments and supporting services
examples/       # Basic case template
tests/          # Contract and regression tests
scripts/        # Import, migration, and maintenance tools

An adapter supplies three main parts: static information, task execution, and a Probe.

• Static information describes the version, tools, plugins, and runtime limits for the Planner.
• Execution prepares sessions, submits tasks, and handles cancellation and cleanup for the current run.
• The Probe converts native events into the shared Trace format and handles duplicates, long text, and observation gaps.

Register the target and workbench controller, then reuse the existing datasets, Judge, and result pages. Adapters return evidence and execution status; the Judge assigns scores.

DSH retains its original execution implementation. Most other adapters use adapters/shared/. See the adapter interface.

Contributions to adapters, tasks, and scoring criteria are welcome. For bug reports, include the agent version, steps to reproduce, and logs or traces with credentials and other sensitive data removed.

The repository does not yet have a project-wide license. Third-party agents, datasets, and brand assets remain subject to their respective licenses.

For questions and feedback, use GitHub Issues.

Website · Report an issue · Release notes