regin

Regin reforged Gram from its shards. This one reforges codebases from failing tests.

---

A terminal coding agent — plan, edit, run tests, read the failure, iterate — and the benchmark loop that proves whether changing it made it better.

Status: Phases 1–4 implemented, unscored. The harness runs end to end; no sweep has been paid for yet, so there is no number here. When there is one it goes in the table below, or the project has failed on its own terms. See docs/PLAN.md.

Every Phase 3 orchestration feature ships off by default — the shipped default has to stay the measured baseline until a sweep says otherwise. See Experiments.

Why another coding agent

There are two hundred of them. The difference here is the second half: every orchestration and prompt change is A/B'd against a held-out benchmark, so claims look like

changed the planner prompt, resolve-rate 31% → 38%, median cost $0.14/task

rather than "it feels smarter now". Scoring lives in the sibling repo evalgate; regin emits JSONL run records it consumes. Neither imports the other's internals.

Results

agent	config	resolve-rate	n	median $/task	p95 latency	mean turns
not yet measured	—	—	—	—	—	—

A resolve-rate here only ever comes from evalgate verdicts. regin bench score refuses to label an ungraded completion rate as a resolve-rate, because that is the easiest way to publish a number that is not true.

Install

pip install -e .                 # the agent
pip install -e ".[bench]"        # plus the Docker sandbox for sweeps
pip install -e ".[dev]"          # plus ruff, pyright, pytest

Set a key in .env (see .env.example):

ANTHROPIC_API_KEY=sk-ant-...

Use

regin fix "make the failing test in tests/test_parser.py pass"
regin fix "..." --yes            # auto-approve tools, for unattended runs
regin trace last --limit 5       # cost, turns, latency per recent run

Every tool call passes a permission gate. read_file, glob, and grep are allowed by default; write_file, edit_file, and bash ask. Configure in regin.toml.

Benchmark

# one-time: fetch a task set and validate it before spending anything
pip install datasets
python scripts/fetch_tasks.py lite
python scripts/check_tasks.py tasks/swe-bench-lite.jsonl

regin bench run   --tasks tasks/swe-bench-lite.jsonl --agent regin --limit 5 --max-cost 2
regin bench score runs/bench/*-records.jsonl --verdicts verdicts.json --markdown results.md

--max-cost is a hard ceiling enforced before each instance is dispatched, not a warning after the bill arrives. See docs/BENCHMARK.md for the WSL2 + Docker setup and the full sweep procedure.

Experiments

Phase 3 added four orchestration features. Each is a hypothesis, not a fact, so each ships disabled and is enabled one at a time in regin.toml:

Flag	What it does	The hypothesis
plan_first	One planning turn before any edit	Less flailing on multi-file tasks
outline_tool	outline tool: signatures, not whole files	Fewer input tokens per run
subagent_tool	search_agent: read-only search in its own context	Fewer context overflows
retry_attempts	Bounded backoff on transient failures (on)	Corrects a measurement error, not behaviour

Turn one on, re-run the sweep, compare. The scorecard groups by config_hash, so both configurations appear as separate rows in one table.

DSPy prompt optimisation (plan item 4.4) is deliberately not built. Optimising against a metric needs a measured baseline to optimise against; building it before Phase 2 has been scored would produce a claim nobody can falsify.

Other commands

regin fix "..." --resume last    # continue a run that hit the turn or budget limit
regin session list               # saved sessions, and which are resumable

Resume refuses to continue a session recorded under a different config_hash — blending two configurations would make the result attributable to neither.

Design commitments

• No agent framework. LangChain, LangGraph, CrewAI, AutoGen, and the OpenAI Agents SDK are all banned as foundations — they own the control flow that is this project's deliverable. claude-agent-sdk appears only as a benchmark competitor.
• Anchored search/replace edits, not model-generated unified diffs — far higher apply rate, and a failed match is a clean, recoverable error the model can react to.
• Cost from provider usage fields, including cache-read and cache-write token classes. Never estimated, because cost is a headline metric.
• loop.py stays under 300 lines. If it grows, compaction or permissions have leaked in.
• Every path goes through safe_path(); every tool call goes through the permission layer, including in tests.
• No real network in unit tests. The provider boundary is stubbed.
• Every run carries a config_hash, so a result is attributable to an exact configuration.

How it works

regin fix "..."
   │
   ├── agent.py ........ assembles provider + tools + permissions (shared with the bench runner)
   ├── loop.py ......... the agent loop: complete → execute tools → feed results back
   │     ├── permissions.py ... allow / deny / ask, --yes for unattended runs
   │     ├── context/ ......... summarise-and-continue compaction
   │     └── tools/ ........... read, write, edit, glob, grep, bash
   │            └── sandbox/ .. path jail + local or Docker execution
   └── trace.py ........ SQLite history + JSONL run records → evalgate

The benchmark half (bench/) drives the same agent across a task set with bounded concurrency and a hard spend ceiling; scorers/swebench.py then grades the patches inside evalgate.

Stack

Python 3.11+ · Typer · Pydantic v2 · Rich · Anthropic SDK behind a thin provider protocol · ripgrep for search · SQLite + JSONL traces · Docker for benchmark sandboxes · pyright strict · pytest

Development

ruff check src tests scorers
pyright                                    # strict
pytest --cov=src/regin --cov-report=term-missing

The type gate and the test suite both have to be green before anything merges. See CONTRIBUTING.md.

Naming

regin — the Norse smith who reforged the shattered sword Gram from its shards.

License

MIT.