Registry-as-ground-truth verification & autonomous research framework for agent-driven development.

Coding agents don't remember. Every new session starts blind: what was already proven, what failed and why, which dead ends were closed for good — gone, buried in chat scrollback and stale plan documents. So agents re-try known failures, re-explore settled questions, and report "done" for work that no gate ever verified. Pawl fixes this by making progress itself a machine-readable artifact: every meaningful change is a claim with acceptance gates frozen before implementation, a CSV registry — not plans, not dashboards, not memory of a conversation — records what is actually implemented, and failures are written down as honest negatives that name the missing capability and permanently block the dead end. Like the mechanism it is named after — the small catch that lets a ratchet wheel turn forward but never slip back — Pawl makes verified progress the only direction the record can move. The origin project ran 590 claims through it in 18 days with 4–5 parallel agent sessions: 404 verified, 167 honest negatives, zero corrupted records (see the case study).

Works for any kind of software project — web app, library, ML project, CLI tool. Pure Python standard library, zero dependencies: adopt it by copying one folder and writing one small config file.

A pawl is the small catch in a ratchet mechanism — the part that engages the teeth of a ratchet wheel and lets it turn forward one click at a time, but blocks any backward slip. That is the whole metaphor: agent sessions should advance only on verified progress, and known dead ends should stay closed instead of being re-explored every time someone opens a fresh chat.

The name is also a nod to karpathy/autoresearch (March 2026), which uses the same ratchet idea for a single training file: keep improvements, discard regressions, one scalar, one budget. Pawl generalizes that loop to a whole repository — many concurrent claims, per-claim edit scopes, a persistent dependency-linked registry, and honest negatives promoted into the record instead of silently reset away.

Pawl is maintained and open-sourced by Agentic Commerce Lab — Shopware's initiative for open protocols and developer tooling at the intersection of AI agents and commerce (agentic-commerce.dev). The Lab's repos — UCP, AXP, WebMCP, sales-agent harnesses, and related bridges — are built by parallel agent sessions where a false "done" is expensive. Pawl is the verification discipline those sessions run on: not a commerce protocol itself, but the registry-and-gates layer that keeps agent work honest in any codebase.

It was first battle-tested on Hylaean (the origin project below) and then extracted as a standalone, project-agnostic framework so other teams — inside Shopware, across the agentic-commerce ecosystem, or anywhere agents ship code — can adopt the same ratchet without copying a whole research repo.

1. No green verification → the feature is not implemented. A merged PR, a passing unit test, or a convincing demo is not "done" — a green claim gate is.
2. Plans are proposals; the registry is ground truth. Plans propose claims; only registered, gated claims count.
3. Measurable success criteria come before implementation. Gates are frozen a priori — a gate defined after seeing the results is a rationalization.
4. Honest negatives are first-class results. A clean Failed with the missing capability named is progress: it blocks the dead end for every future session.
5. Failed triggers a downstream sweep. Every Planned claim that depended on a Failed one is immediately re-hung, re-scoped, or closed.

Concretely, Pawl gives agents (a) a queryable memory of every attempt and outcome, (b) gates they cannot argue with, and (c) a loop for running experiments unattended without corrupting the record.

pawl/
├── README.md                 # this file
├── AGENTS.template.md        # agent entry point template (copy to repo root)
├── docs/                     # methodology
│   ├── verification.md       # registry, claims, lifecycle, folder layout
│   ├── autoresearch.md       # the autonomous research loop
│   ├── big-picture-guide.md  # how to write the agent-facing project doc
│   ├── plans.md              # plans as proposals; plan->claim linkage
│   ├── tracks-and-fronts.md  # research fronts and success criteria
│   ├── honest-negatives.md   # Failed as a first-class result
│   ├── metric-integrity.md   # honest measurement discipline
│   └── adoption.md           # step-by-step adoption guide
├── templates/                # big-picture, plan, claim README, contract,
│                             # tracks.json, verify.json, CHANGELOG
├── rules/                    # portable .mdc rules -> your .cursor/rules/
├── skills/                   # portable skills    -> your .cursor/skills/
├── tools/                    # the runnable layer (stdlib only)
│   ├── init_project.py       # bootstrap verify.json + registry into a repo
│   ├── register.py           # add a Planned claim (duplicate check, schema)
│   ├── init_folder.py        # create the claim folder from the registry row
│   ├── run.py                # official runner: gate -> Verified | Failed
│   ├── scan.py               # audit, search, Planned triage
│   ├── memory.py             # BM25 index over claims + plans (fronts, query, show)
│   └── lib/                  # config, ledger schema, locking
├── autoresearch/             # autonomous loop: program template + bootstrap
└── examples/demo-project/    # a complete worked example (real runs)

# 0. Copy the pawl folder into your repo (or keep it as a sibling checkout).
#    TOOLS below = path to pawl/tools.

# 1. Bootstrap your project (creates verify.json, registry.csv, tracks.json):
cd ~/my-project
python3 $TOOLS/init_project.py my-project --plans-dir docs/plans

# 2. Register your first claim — a falsifiable sentence with thresholds:
python3 $TOOLS/register.py VERIFY.CORE.CSV_PARSER.01 \
  "CSV parser handles quoted fields and CRLF; 12 fixture files round-trip \
byte-identical; parse of 100k rows <= 2 s on the fixed bench"

# 3. Create its folder (README + run.py skeleton):
python3 $TOOLS/init_folder.py VERIFY.CORE.CSV_PARSER.01

# 4. Freeze the gates in the generated run.py, THEN implement the feature.

# 5. Official pass — writes Verified/Failed back to the registry:
python3 $TOOLS/run.py VERIFY.CORE.CSV_PARSER.01

# 6. Query your project's memory:
python3 $TOOLS/memory.py fronts
python3 $TOOLS/memory.py query "csv parser"
python3 $TOOLS/memory.py show VERIFY.CORE.CSV_PARSER.01

A fully executed example lives in examples/demo-project/ — including one Verified claim and one honest-negative Failed claim.

scan -> register.py (CSV row only) -> init_folder.py -> freeze gates
     -> implement -> tools/run.py -> Verified | Failed

No folder without a registry row. Folder name is always YYYY-MM-DD-NN-CLAIM_ID. scan.py --audit enforces consistency.

registry.csv, one row per claim:

The autoresearch/ layer turns the registry into an autonomous experiment loop: an agent picks a target claim, edits only that claim's declared scope, measures against a fixed harness it may not edit, commits scoped keeps or reverts discards on main, and promotes only through the official runner. The human programs the research organization (rules, harness, program); the agent executes it. See docs/autoresearch.md.

Diagram sources: docs/assets/src/ — regeneration: docs/assets/README.md.

1. Write the big picture — the agent-facing project doc: mission, one-sentence law, layers, anti-sprawl rules (templates/big-picture.template.md, guide: docs/big-picture-guide.md).
2. Define tracks — group your claim families into research fronts with success criteria (templates/tracks.template.json, docs/tracks-and-fronts.md).
3. Register the first claims — init_project.py, then register.py + init_folder.py for the 2–5 changes you actually care about next.
4. Copy rules and skills — rules/*.mdc → .cursor/rules/, skills/* → .cursor/skills/, AGENTS.template.md → AGENTS.md; replace the <PLACEHOLDERS>.
5. Start the loop — manual first (agent implements against frozen gates), then autonomous (autoresearch/program.template.md).

Full guide: docs/adoption.md.

Pawl records experimental memory — what was tried, what the gates said, what is closed for good. Agent sessions lose a second kind of memory between runs: structural memory — how the code hangs together and what breaks if you change it. The setup Pawl was extracted from pairs it with two codebase-memory systems, both equally project-agnostic and worth adopting alongside it:

• graphify — a committed knowledge-graph snapshot of the repo (graphify-out/: graph.json, a generated GRAPH_REPORT.md, optional wiki/HTML export). A CLI answers structural questions offline and the artifact travels with the repo: graphify query "<question>" (scoped subgraph), graphify path "<A>" "<B>" (dependency path between two symbols), graphify explain "<concept>", and graphify update . after edits to keep the snapshot current. Because it is committed, reviewers and CI see the same architecture picture the agent saw.
• codebase-memory-mcp — a live local MCP server (a Go binary with a persistent SQLite graph built from tree-sitter ASTs: qualified symbols, call edges, HTTP edges) that auto-syncs in the background as files change. Agents query it instead of grepping: search_graph (find symbols by name or concept), trace_path (who calls X / what does X call), get_architecture (packages, hotspots, wiring), and detect_changes (blast radius of the current git diff — what is affected before you run anything).

The division of labor with Pawl is clean, and the combination is what makes agent sessions stop repeating themselves:

Structural memory keeps agents from re-exploring the codebase every session; experimental memory keeps them from re-running settled experiments. A practical convention from the origin setup: make the code graph the mandatory first step for structural questions (a rule in .cursor/rules/), exactly like Pawl's memory query is mandatory before registering a claim.

There is a wave of autonomous-loop and agentic-SWE tooling (as of mid-2026). Pawl occupies a specific, mostly empty slot: the bookkeeping and integrity layer for agent-driven development in an existing repo — not an agent, not a harness, not a tracker.

Prose, in a nutshell:

• vs. karpathy/autoresearch (released March 2026; the closest relative and an explicit inspiration — even for this project's name): Karpathy's loop is a beautifully minimal ratchet — one file (train.py), one scalar (val_bpb), a fixed 5-minute budget, keep on improvement or reset on equal-or-worse. Pawl generalizes the ratchet to a whole repository: many concurrent claims instead of one file, per-claim edit scopes instead of "everything in train.py is fair game", and a persistent registry with dependency edges instead of a local results.tsv. The biggest philosophical difference: negatives are promoted, not just logged. In the Karpathy loop a failed idea is reset away, surviving only as a line in the local experiment log; here it becomes a Failed claim that names the missing capability and permanently blocks the dead end. Shared core conviction: the human programs the research org (his program.md ↔ Pawl's rules + program template).
• vs. Sakana AI Scientist: that system automates science-shaped output — idea generation, experiments inside provided code templates, paper drafting, and an automated LLM peer review. Pawl automates no science at all — it is the ledger and gate discipline that keeps whatever agent you use honest while it works on your codebase. (Independent evaluation of the AI Scientist also found it needs heavy supervision — a large share of its experiments fail on execution — which is precisely the failure mode a frozen external gate is built to catch.)
• vs. SWE-agent / OpenHands: those are the agents — SWE-agent (Princeton) established the SWE-bench-style issue-resolution loop, OpenHands (formerly OpenDevin) is the leading open-source agent platform. They decide how to edit code; Pawl is the ground-truth layer around any agent: what counts as done, what was already tried, what failed and why. They compose: an OpenHands or Cursor session is a fine executor for claims registered here.
• vs. MLflow / W&B: experiment trackers record what happened; they have no notion of a falsifiable claim, a frozen gate, or a verdict that a runner writes back to a ground-truth ledger. Tracking answers "what were the numbers?" — a registry answers "is it implemented, and how do we know?"

This section names the origin project as provenance; everything else in the framework is project-agnostic.

Pawl was extracted from Hylaean (the "Field AI" research repo) — an AI-research codebase where essentially all implementation work is done by parallel agent sessions, and where a wrong "it works" is far more expensive than a slow one. The numbers below are counted live from that repo's registry (2026-07-13), not estimated:

• 590 claims registered in 18 days (2026-06-26 → 2026-07-13), spanning 57 claim families curated into 10 tracks.
• Of the 588 active claims: 404 Verified, 167 Failed, 17 Planned — a 28% honest-negative rate, every Failed row naming its missing capability.
• 45 plan documents linked to claims by the memory index (~770 plan→claim edges) — plans proposed the work; only the registry says what landed.
• 4–5 parallel agent sessions on one host, coordinated through the registry lock and per-claim run locks — zero registry corruption and no duplicated official runs over the whole period.

What the discipline bought, concretely:

• Dead ends closed once, permanently. A single census wave measured eight candidate "breakthrough" directions against pre-frozen kill rules and closed all eight before any of them was built — honest negatives that saved months. A separately tempting 88-task shortcut was measured to zero in five independent attempts and is terminal-negative: reopening is locked behind two named missing primitives.
• Regressions caught before merge. A verified baseline lock freezes every solved benchmark task by ID; any change that regresses one solve or introduces one false result fails verification before it lands (re-locked stepwise from 94 to 145 tasks as capability grew).
• Honest public accounting. The project publishes its status log — including the negatives — straight from the registry (hylaean.ai); its front page reports an unsolved headline benchmark (0/120 eval) next to its verified wins, because the registry, not the marketing, is the source of truth.
• Memory that survives sessions. New agent sessions start with memory.py fronts / query instead of re-reading hundreds of folders — the duplicate check before registering is what keeps 590 claims from becoming 900 redundant ones.

• CSV + JSON + Markdown, stdlib only. Diffs are reviewable, merges are resolvable, and there is nothing to install. That is deliberate: the framework must be more durable than any toolchain fashion.
• Registry locking is portable. Lock-file with stale-PID takeover + atomic replace — safe for parallel agent sessions on one machine without OS-specific APIs.
• The memory index is generated and gitignored. It rebuilds in well under a second for hundreds of claims; committing it would only create merge noise.
• _template/ folder over generators where possible. Projects customize the claim skeleton by editing files, not code.
• The config file stays verify.json. It describes function, not brand — an adopting repo's root should explain itself to someone who has never heard of Pawl.

Maintained by Agentic Commerce Lab (Shopware). Extracted and generalized from the origin project described in the case study above. The autonomous-loop design is inspired in part by karpathy/autoresearch, extended with per-claim scopes, a persistent dependency-linked registry, and honest negatives as first-class results.