CodeTraceLab

English | 中文

CodeTraceLab is a data infrastructure toolkit for coding-agent training and evaluation. It turns real repository history into verified, reproducible, difficulty-aware task data.

The core pipeline starts from merged pull requests, reconstructs patch provenance, builds verification inputs, runs before/after tests, extracts FAIL_TO_PASS signals, and exports metadata for SFT, RL, evaluation, and rollout analysis.

What It Does

CodeTraceLab helps teams build coding-agent data from real software history without losing the runtime evidence that makes the data trustworthy.

It is designed to:

• collect merged pull requests and reconstruct the exact pre-fix repository state
• split implementation patches from test patches and check whether they apply cleanly
• build repo-aware verification inputs for Django, SymPy, SQLFluff, and future repositories
• run local Docker or E2B before/after verification to extract FAIL_TO_PASS and PASS_TO_PASS
• repair missing stable PASS_TO_PASS tests without changing the main Stage3 contract
• track runs, artifacts, decontamination status, difficulty, and task category in SQLite
• export unified task JSONL for SFT, RL, evaluation, and rollout analysis

Why CodeTraceLab

High-quality coding-agent data needs more than a patch and an issue title. It needs to answer:

Can the pre-fix repository state be reconstructed?
Can the test patch be applied cleanly?
Can the failing test be observed before the fix?
Can the same test pass after the fix?
Can failures be diagnosed and repaired at scale?
Can the final sample be exported without leaking private gold signals?

CodeTraceLab is built around those checks.

System Flow

Layer	Purpose	Main Outputs
PR snapshot	Capture merged PR metadata, commits, files, and patches.	raw PR records
Patch assets	Split solution_patch and test_patch, check apply safety, and classify candidate quality.	patch manifest
Repo profile	Convert repo-specific tests and install rules into standard verification inputs.	Stage3 input JSONL
Verification	Run before/after tests in Docker or E2B and parse logs.	verified F2P/P2P records
Stage3.5 repair	Add stable PASS_TO_PASS commands for verified cases that are missing regression guards.	repaired P2P records
Data quality	Apply decontamination, prompt cleaning, difficulty labels, and task categories.	delivery-ready task JSONL
Storage/index	Record runs, artifacts, statuses, and queryable metadata in SQLite.	reproducible run index

The runtime contract is intentionally simple: a good task should have a reconstructable base_commit, an applicable test_patch, non-empty FAIL_TO_PASS, non-empty PASS_TO_PASS, and an export schema that does not leak gold repair signals.

Core Artifacts

patch manifest
  -> Stage3 input
  -> Stage3 verified / rejects
  -> Stage3.5 P2P repair result
  -> decontaminated unified task JSONL
  -> data cards and delivery summaries

JSONL keeps each stage easy to inspect and rerun. SQLite adds an index over runs, artifacts, verification status, decontamination level, and delivery state so larger experiments can be resumed and audited.

Package Layout

codetracelab/
  data/          Dataset import/export, difficulty annotation, schema helpers
  evaluation/    Candidate-patch evaluation runner
  llm/           OpenAI-compatible client helpers
  profiles/      Repo adapters for Django, SymPy, SQLFluff, and future repos
  recipes/       Version-aware install recipe selectors
  stages/        PR snapshots, patch assets, Stage input builders, stage runners
  storage/       SQLite state/index helpers
  structure/     Optional structural features, including CST node changes
  validators/    LLM output and recipe validation
  verification/  Docker verifier, E2B helpers, agent runtime, log parsers

scripts/          Backward-compatible CLI wrappers
lib/              Backward-compatible import wrappers
examples/         Small public fixtures
docs/             Public documentation and design notes
tests/            Unit tests and refactor guards

Local experiment artifacts such as data/, repos/, worktrees/, logs/, doc/, hf-datasets/, and SWE-agent/ are intentionally ignored by Git.

For a fuller map, see Repository Layout.

System Docs

• docs/CURRENT_SYSTEM_ARCHITECTURE_AND_REPRO.md — current system architecture diagram + practical reproduction guide
• docs/ENGINEERING_OVERVIEW.md — detailed upstream pipeline walkthrough
• docs/data_cards/README.md — dataset distribution snapshots and card generation notes

Installation

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[dev]"

Optional extras:

python -m pip install -e ".[stage1]"    # Ray-backed Stage1 scraping
python -m pip install -e ".[datasets]"  # parquet / SWE-Lego helpers
python -m pip install -e ".[cst]"       # CST structure feature extraction
python -m pip install -e ".[e2b]"       # E2B sandbox experiments

Copy the environment template before using GitHub, LLM, or E2B APIs:

cp .env.example .env

Smoke Test

The lightweight refactor guard is:

python scripts/smoke_test.py

It avoids Docker, network APIs, and long-running jobs. It checks:

pytest tests
core CLI --help imports
all codetracelab package files compile
key legacy wrappers compile

Minimal Examples

Small public fixtures live in examples:

examples/minimal_patch_manifest.jsonl
examples/minimal_stage3_input.jsonl

They document artifact shapes and are useful for parser/schema development. They are not intended to be final training data.

Build generic Stage3 input through a repo profile:

python scripts/build_repo_stage3_input.py \
  --profile django \
  --input examples/minimal_patch_manifest.jsonl \
  --output /tmp/codetracelab_stage3_input.jsonl \
  --git-provider worktree \
  --worktree /path/to/django/worktree \
  --limit 1

Run Stage3 verification on an existing Stage3 input JSONL:

python scripts/stage3_verify.py \
  --input examples/minimal_stage3_input.jsonl \
  --output /tmp/verified.jsonl \
  --rejects /tmp/rejects.jsonl \
  --max-items 1 \
  --max-workers 1

Stage3 verification requires Docker and a real repository-compatible input row.

Useful Commands

# GitHub PR snapshots
python scripts/stage0_pr_snapshot.py --help

# Build Stage1 patch assets from snapshots
python scripts/stage1_build_assets.py --help

# Build repo-profile based Stage3 input
python scripts/build_repo_stage3_input.py --help

# Verify before/after tests in Docker
python scripts/stage3_verify.py --help

# Run verification in resumable batches
python scripts/run_stage3_batches.py --help

# Import SWE-Lego metadata into SQLite
python scripts/import_swe_lego.py --help

After pip install -e ., package entrypoints are also available:

codetracelab-stage1-scrape --help
codetracelab-stage2-recipe-gen --help
codetracelab-stage3-verify --help
codetracelab-stage4-score --help
codetracelab-run-stage3-batches --help
codetracelab-build-repo-stage3-input --help
codetracelab-build-django-stage3-input --help
codetracelab-build-sympy-stage3-input --help
codetracelab-eval --help
codetracelab-export-difficulty-dataset --help
codetracelab-export-swe-bench-verified-difficulty --help
codetracelab-compute-cst-node --help
codetracelab-abstract-cst-node-changes --help
codetracelab-e2b-sympy-batch --help

Repo Profiles

Repo profiles keep repository-specific behavior out of the core pipeline:

test file / fixture detection
target extraction
test command construction
install recipe selection
environment-risk classification
recipe fingerprinting

See Add A New Repo Profile to adapt a new repository.

Compatibility

New code should import from codetracelab.*.

Legacy imports remain supported during the refactor:

from lib.stage3_input import stage3_row_from_manifest
from lib.verifier import compute_fail_to_pass
from lib.repo_profiles import get_profile

The test suite keeps exercising these old paths so previous experiments remain reproducible.

Documentation

• Quickstart
• Architecture
• Repository Layout
• Add A New Repo Profile
• Refactor Plan
• Contributing

License

MIT. See LICENSE.