Traccia v1.0 — AI Agent Execution Benchmark
Repository: https://github.com/d87skg/traccia
Dataset: https://huggingface.co/datasets/vasdvae/traccia-benchmark
Tool: pip install traccia-eval
Abstract
Current AI agent benchmarks evaluate only final outputs. We propose Execution-Centric Evaluation — scoring agents on their full execution trajectory using a 4‑dimensional residual vector (temporal, semantic, structural, recovery). The Traccia framework includes structured execution traces (.evidence), a deterministic scoring function, a benchmark dataset of 27 traces across 6 runtimes, and a public leaderboard.
Quick Start
pip install traccia-eval traccia-guard
traccia loop run --iterations 10
traccia-eval dataset.jsonl
Citation
bibtex
@software{traccia2026,
author = {Traccia Project},
title = {Traccia: An Execution-Centric Benchmark for AI Agents with Residual Evaluation},
year = {2026},
url = {https://github.com/d87skg/traccia}
}