RepoTrials v0.1.0 — Private coding-agent evals from your Git history

RepoTrials v0.1.0 is the first public release.

RepoTrials turns bugs a team already fixed into private, repeatable coding-agent evaluations. Instead of asking which agent wins on a public leaderboard, it helps answer a narrower question: which model, scaffold, prompt, and budget work on this repository?

It is inspired by SWE-bench's behavioral grading, but it is not a drop-in reimplementation of SWE-bench.

What ships in v0.1.0

• Local-first mining of real fixes from Git history
• Python/pytest-oriented task discovery
• Deterministic BASE, RED, and GOLD validation
• Hidden test and reference-patch material kept outside the agent workspace
• Frozen task contracts, content digests, and exact submission-path policies
• Any coding agent that can be invoked as a command
• Repeated attempts and empirical task-level pass@k
• Strict cohort checks before comparing two run groups
• JSON records and local HTML reports
• Harbor v0.20.0 task-schema 1.3 export with a separate no-network verifier
• Zero third-party runtime dependencies in the RepoTrials core

RepoTrials itself does not upload your repository, task set, or results.

Try the complete demo

The demo creates a real two-commit Git repository, mines and validates a task, runs a no-op and a deterministic fixing agent, compares them, writes a report, and creates a Harbor export. It requires no model API key.

git clone --depth 1 --branch v0.1.0 https://github.com/PozziTiv4ik/Repo-Trials.git
cd Repo-Trials
python -m pip install -e .[dev]
python scripts/demo.py

Expected comparison:

noop-agent   0/1 resolved
fix-agent    1/1 resolved
delta       +100 percentage points

Important boundaries

v0.1.0 is intentionally conservative:

• The supported profile is currently Git repositories with Python tests.
• Automatic reproducibility is not proof that a task is fair; trusted task sets still need human review.
• Submissions may modify only implementation paths touched by the historical fix, so some valid alternative designs are rejected.
• git archive reconstruction does not cover every submodule, Git LFS, symlink, or export-ignore dependent repository.
• Public repository history may already be present in model training data.
• RepoTrials is not a hardened sandbox. Run untrusted tests and agents inside an operator-provided isolation boundary.
• Harbor images and downstream runtimes still require qualification in their actual environment.
• Commands and schemas may change before 1.0.

Read the methodology and threat model before using results for consequential decisions.

Feedback wanted

The most valuable early feedback is a repository where mining breaks, a reproducible-but-unfair task, a rough edge in agent integration, or a real Harbor conformance result.

Use Discussions for questions and evaluation design, and Issues for reproducible bugs or scoped improvements.