mla-bench

A reproducible benchmark harness for governed memory in coding agents — the setup behind the results at research.meetless.ai.

It answers one question per capability, with numbers instead of vibes: does an agent with governed memory (accepted/rejected decisions, provenance, live conflict detection) beat the things a competent team actually uses — a CLAUDE.md, a vector store, a pile of notes — on the tasks where memory should matter, without costing more where it shouldn't?

We publish this harness for one reason: you shouldn't trust a vendor benchmark you can't inspect. The isolation rigor is the point — most of this repo is the machinery that stops the test from cheating, in either direction.

The honest headline

Run fairly, most categories tie or lose. For raw fact recall, a CLAUDE.md or a vector store hands back text and wins on cost — that is not a defect, it is the thesis: we are a decision store, not a document store. Only a few claims survive an opponent that is allowed to know things, and those are the only ones we stand behind:

capability	the claim	why a text store can't close it
trust-containment	ungoverned memory ships the rejected decision	a store returns text; it can't record which decision won
provenance	a file answers "who approved this?" 0/5	a markdown file has nowhere to put a reviewer
contested-canonical	on a rename practice contradicts, raw reasoning ships the superseded term 16/17	text records what was said, not which decision still stands

We also publish the losses and retractions (a benchmark whose prompt told the opponent it couldn't win; a "thesis" we set out to prove and disproved). A suite that ships its null results is worth more than one that only ships wins.

The arms

arm	what it is	role
vanilla	Claude Code, no memory	the floor
static	Claude Code + a CLAUDE.md holding the org's rules/snapshot	what most teams do now
ungoverned	the same corpus with perfect recall, trust labels stripped	a stronger-than-real RAG (no ranking, no cutoff, no misses)
mla	governed memory (accept/reject verdicts, provenance)	the thing under test

The interesting wins are against static and ungoverned, never vanilla — beating an agent with no memory proves nothing.

Isolation contract (what makes a run trustworthy)

The answer to a governed-knowledge task exists in exactly two places: the fixture's database and the launcher's process memory. It exists in no file, git object, env var, or network endpoint a non-mla arm can reach.

Enforced by: a neutral, unrelated substrate repo (no product code to teach the agent the thesis); expected answers injected via env and deleted from the subprocess environment; sandbox-exec denials on the local store; and — this is the part worth reading — a harness that aborts rather than measures on any of ~18 contamination modes we caught benchmarking ourselves. See the write-ups' methodology pages for the full contamination log; roughly half of those leaks flattered us and half flattered the opponent.

Run it

node selftest.mjs                      # unit-test the pure parsers (no cost)
node run.mjs --dry-run                 # print the plan, spend nothing
node run.mjs --tasks <task> --arms vanilla,static,ungoverned --trials 5
node analyze.mjs                       # runs/ -> results/summary.json + a console table

Add a task: drop tasks/<id>/{task.md, meta.json, oracle.sh}. Oracles run against the finished worktree and must be deterministic — a flaky oracle poisons the benchmark. Fixtures live in fixtures/ and generate fresh, random answers per run, so no model can have trained on them and no published result can contaminate a future one.

The honest caveat about the mla arm

The baseline arms (vanilla, static, ungoverned) run standalone — you can clone this, inspect every line, and reproduce their numbers with just Node + Claude Code. The mla arm needs a running Meetless stack to seed and serve governed memory, so reproducing its numbers requires the product. What you can verify without it is the whole setup, the isolation, and the separation — the shape of the result, which is what a benchmark is for.

Layout

run.mjs                 the harness (arms, worktree isolation, sandbox, stream-JSON metrics)
preflight.sh            refuses to run against a lying stack
fixtures/seed.mjs       fixtures-as-code: provision a workspace, materialize governed state
tasks/<id>/             one benchmark each: task.md + meta.json + oracle.sh
substrate/              neutral substrate (an order-fulfilment service)
substrate-tracker/      neutral substrate (a work-item tracker) for terminology tasks
arms/                   the ungoverned-memory MCP (perfect recall, no verdict)
metrics.mjs stats.mjs   parsing + nonparametric stats (median/IQR, Cliff's δ)
site-src/               the published write-ups (paired results + methodology pages)

Reproducibility & honesty, as features

• Fixtures are generated per run — your values differ from ours; what should reproduce is the separation, not the literal numbers.
• Runs are graded by a deterministic oracle, never by a model.
• We read the traces, not just the verdicts — a green checkmark hides the mechanism.
• Raw run transcripts are not published; they embed the generated answer and a real credential.

---

Built by Meetless. Results: research.meetless.ai. Issues and reproductions welcome.